Comment on page

Time your data with freshness

Learn how to check the punctuality of your information flows and act according to the severity of the delay

​🕐
 Track your data operation like Amazon delivery
Tailer Studio allows you to display all Job information relating to execution time and period. It is also possible to announce the next execution. Let's see if your data operations are well-timed.
​⚠
 Only data operations in production with at least 2 regular executions can have a freshness status.
​👀
 View your data freshness
Log in to Tailer Studio.
If necessary, select an account in the drop-down menu at the top of the screen.
In the left navigation panel, in the Data quality section, select the Freshness Monitoring
All your jobs are listed and can be sorted :
​ℹ
 Information lists
Information
Description
Configuration Type
Sorting option
Type of data operation.

ex: tables-to-tables, storage-to-tables, vm-launcher...
Job ID
ID of the job linked to the execution of a configuration.

A job ID is the concatenation of data operation, ID of the configuration and optionally the environnement and the file used or produced depending of the data operation.
This ID is unique and linked to the page of the configuration related.
Last Execution
Date time (UTC+0) of the last execution of your data operation.
Next Execution
Date time (UTC+0) of the next execution of your data operation.

This is one of our main objectives. It estimates from previous executions the next time it will come.

⚠
The more regular the executions, the more precise it will be. Conversely, if there are few executions, or some that come irregularly, the calculation will be less accurate.
Confidence
It's an indicator about the arrival confidence interval. The value is correlated with the potential number of hours of delay for the data operation : the higher the number is, the more likely the operation is to be late.

Here are a few examples to help you understand :
[0] : The operation is well-timed overall.
[1-3] : Often, the operation may be delayed because it is at the end of the workflow or waiting to receive an inconsistent file.
[5-10] : For a daily operation, you may have an issue. For weekly or monthly information, you may lake of well-timed executions or waiting to receive an inconsistent key file.
[10 +] : the operation is often late or inconsistent. It might be interesting to see if there's a problem behind it.

Frequency
Sorting option
Type of frequency detected with cron schedule expressions or with frequency calculation.

ex: Daily, Weekly, Monthly Yearly and Unfollow.

Yearly is about a data operation with very few regular executions with long waiting times
Unfollow is about data operation with less than 2 regular executions in production.
Status
Sorting option
Freshness status of the data operation for the current day.

It estimates from previous executions whether the operation has already been executed today, is pending or should not be activated.
Executed : The operation is already successful today.
Awaited : Thanks to the deduction of the next execution, this indicates that the flow must be executed today.
Not today : Thanks to the deduction of the next execution, this indicates that the flow does not have to be executed today. Check the date time to see the futur execution date.
Learning & Archived : Either the data operation in production do not have enough execution or was archived to not been displayed.
Timing
Sorting option
Freshness timing of the data operation for the current day.

This is one of our main objectives. It estimates from the calculus of the next execution with the confidence interval how far data is behind the times due.
On time : The data operation is well-timed today.
Paired with Executed statut, it is perfect !
Paired with Awaited statut, the data operation is on time and should be executed later in the day.
Late : Arrival time exceeds predicted time.
Paired with Executed statut, the data operation was executed with a little delay. Check that the operations behind it accept latency.
Paired with Awaited statut, the data operation is late but should have been executed already. There may be an issue with receiving files or scheduling execution. This is the first step before we get to the extreme delay problem.
​⚠
Very Late : The due execution overlap with the next predicted execution.

For example, a daily data operation has more than 25 hours late.

Tailer has an alert system and it's possible to send you a message via Slack or via email. You can also send a message to [email protected] in that case.
NA & Learning : Either the data operation in production do not have enough execution or was archived to not been displayed.
​✍
 Use case with a file received on-time or lately

Information	Description
Configuration Type Sorting option	Type of data operation. ex: tables-to-tables, storage-to-tables, vm-launcher...
Job ID	ID of the job linked to the execution of a configuration. A job ID is the concatenation of data operation, ID of the configuration and optionally the environnement and the file used or produced depending of the data operation. This ID is unique and linked to the page of the configuration related.
Last Execution	Date time (UTC+0) of the last execution of your data operation.
Next Execution	Date time (UTC+0) of the next execution of your data operation. This is one of our main objectives. It estimates from previous executions the next time it will come. ⚠ The more regular the executions, the more precise it will be. Conversely, if there are few executions, or some that come irregularly, the calculation will be less accurate.
Confidence	It's an indicator about the arrival confidence interval. The value is correlated with the potential number of hours of delay for the data operation : the higher the number is, the more likely the operation is to be late. Here are a few examples to help you understand : [0] : The operation is well-timed overall. [1-3] : Often, the operation may be delayed because it is at the end of the workflow or waiting to receive an inconsistent file. [5-10] : For a daily operation, you may have an issue. For weekly or monthly information, you may lake of well-timed executions or waiting to receive an inconsistent key file. [10 +] : the operation is often late or inconsistent. It might be interesting to see if there's a problem behind it.
Frequency Sorting option	Type of frequency detected with cron schedule expressions or with frequency calculation. ex: Daily, Weekly, Monthly Yearly and Unfollow. Yearly is about a data operation with very few regular executions with long waiting times Unfollow is about data operation with less than 2 regular executions in production.
Status Sorting option	Freshness status of the data operation for the current day. It estimates from previous executions whether the operation has already been executed today, is pending or should not be activated. Executed : The operation is already successful today. Awaited : Thanks to the deduction of the next execution, this indicates that the flow must be executed today. Not today : Thanks to the deduction of the next execution, this indicates that the flow does not have to be executed today. Check the date time to see the futur execution date. Learning & Archived : Either the data operation in production do not have enough execution or was archived to not been displayed.
Timing Sorting option	Freshness timing of the data operation for the current day. This is one of our main objectives. It estimates from the calculus of the next execution with the confidence interval how far data is behind the times due. On time : The data operation is well-timed today. Paired with Executed statut, it is perfect ! Paired with Awaited statut, the data operation is on time and should be executed later in the day. Late : Arrival time exceeds predicted time. Paired with Executed statut, the data operation was executed with a little delay. Check that the operations behind it accept latency. Paired with Awaited statut, the data operation is late but should have been executed already. There may be an issue with receiving files or scheduling execution. This is the first step before we get to the extreme delay problem. ⚠ Very Late : The due execution overlap with the next predicted execution. For example, a daily data operation has more than 25 hours late. Tailer has an alert system and it's possible to send you a message via Slack or via email. You can also send a message to [email protected] in that case. NA & Learning : Either the data operation in production do not have enough execution or was archived to not been displayed.

Tailer Studio - Previous

View your data catalog

Next - Tailer API

Overview

Last modified 3d ago

Time your data with freshness

​🕐 Track your data operation like Amazon delivery

​👀 View your data freshness

​ℹ Information lists

​✍ Use case with a file received on-time or lately

🕐
Track your data operation like Amazon delivery

👀
View your data freshness

ℹ
Information lists

✍
Use case with a file received on-time or lately