Time your data with freshness
Learn how to check the punctuality of your information flows and act according to the severity of the delay
Last updated
Learn how to check the punctuality of your information flows and act according to the severity of the delay
Last updated
Tailer Studio allows you to display all Job information relating to execution time and period. It is also possible to announce the next execution. Let's see if your data operations are well-timed.
Only data operations in production with at least 2 regular executions can have a freshness status.
Log in to Tailer Studio.
If necessary, select an account in the drop-down menu at the top of the screen.
In the left navigation panel, in the Data quality section, select the Freshness Monitoring
All your jobs are listed and can be sorted :
Configuration Type
Sorting option
Type of data operation.
ex: tables-to-tables, storage-to-tables, vm-launcher...
Job ID
ID of the job linked to the execution of a configuration. A job ID is the concatenation of data operation, ID of the configuration and optionally the environnement and the file used or produced depending of the data operation. This ID is unique and linked to the page of the configuration related.
Last Execution
Date time (UTC+0) of the last execution of your data operation.
Next Execution
Confidence
It's an indicator about the arrival confidence interval. The value is correlated with the potential number of hours of delay for the data operation : the higher the number is, the more likely the operation is to be late. Here are a few examples to help you understand :
[0] : The operation is well-timed overall.
[1-3] : Often, the operation may be delayed because it is at the end of the workflow or waiting to receive an inconsistent file.
[5-10] : For a daily operation, you may have an issue. For weekly or monthly information, you may lake of well-timed executions or waiting to receive an inconsistent key file.
[10 +] : the operation is often late or inconsistent. It might be interesting to see if there's a problem behind it.
Frequency Sorting option
Type of frequency detected with cron schedule expressions or with frequency calculation. ex: Daily, Weekly, Monthly Yearly and Unfollow. Yearly is about a data operation with very few regular executions with long waiting times Unfollow is about data operation with less than 2 regular executions in production.
Status
Sorting option
Freshness status of the data operation for the current day. It estimates from previous executions whether the operation has already been executed today, is pending or should not be activated.
Executed : The operation is already successful today.
Awaited : Thanks to the deduction of the next execution, this indicates that the flow must be executed today.
Not today : Thanks to the deduction of the next execution, this indicates that the flow does not have to be executed today. Check the date time to see the futur execution date.
Learning & Archived : Either the data operation in production do not have enough execution or was archived to not been displayed.
Timing
Sorting option
Freshness timing of the data operation for the current day. This is one of our main objectives. It estimates from the calculus of the next execution with the confidence interval how far data is behind the times due.
On time : The data operation is well-timed today.
Paired with Executed statut, it is perfect !
Paired with Awaited statut, the data operation is on time and should be executed later in the day.
Late : Arrival time exceeds predicted time.
Paired with Executed statut, the data operation was executed with a little delay. Check that the operations behind it accept latency.
Paired with Awaited statut, the data operation is late but should have been executed already. There may be an issue with receiving files or scheduling execution. This is the first step before we get to the extreme delay problem.
NA & Learning : Either the data operation in production do not have enough execution or was archived to not been displayed.
Date time (UTC+0) of the next execution of your data operation. This is one of our main objectives. It estimates from previous executions the next time it will come. The more regular the executions, the more precise it will be. Conversely, if there are few executions, or some that come irregularly, the calculation will be less accurate.
Very Late : The due execution overlap with the next predicted execution. For example, a daily data operation has more than 25 hours late. Tailer has an alert system and it's possible to send you a message via Slack or via email. You can also send a message to support@tailer.ai in that case.
