Monitoring and alerting parameters

Learn how to add monitoring and alerting information to your data operation configurations.
The monitoring and alerting parameters are defined in a JSON object added at the root level of the JSON data operation configuration. It contains the following sections:
  • Monitoring parameters: General information about the criticality of the data operation.
  • Alerting parameters: One or several alert messages and channels, containing information about how to alert the right people throw which system or application.
Currently, you can only send on alert message, and only by email. But we are thinking about adding more alert systems like Pagerduty, Datadog, or throw a generic web-hook. Feel free to suggest to us your preferred alerting platform.

Example

Here is an example in a TTT configuration file:
{
"configuration_type": "table-to-table",
"configuration_id": "000001_append_some_data",
"short_description": "Append some data to a partitioned table",
"account": "000099",
"environment": "DEV",
"activated": true,
"archived": false,
"start_date": "2023, 1, 23",
"schedule_interval": "*/5 * * * *",
"max_active_runs": 1,
"task_concurrency": 3,
"default_gcp_project_id": "my-project",
"default_bq_dataset": "my_dataset",
"default_write_disposition": "WRITE_TRUNCATE",
"direct_execution": true,
"task_dependencies": [
"create_my_data_table >> merge_table_with_last_data"
],
"workflow": [
{
"task_type": "create_gbq_table",
"id": "create_my_data_table",
"short_description": "Create the destination table with partitioning on date and clustering",
"bq_table": "my_data",
"ddl_file": "my_data.json",
"force_delete": false
},
{
"task_type": "run_gbq_script",
"id": "merge_table_with_last_data",
"sql_file": "merge_table_with_last_data.sql"
}
],
"monitoring": {
"impact": 2,
"urgency": 2,
"alert_enabled": true,
"alert_status": ["FAILED","NO_MATCH"],
"alert_environment": ["PROD","DEV"],
"alert_info": "Put here information about the alert",
"alert": {
"email" : {
"email_from": "[email protected]",
"email_reply_to": "[email protected]",
"email_subject": "Data Operation Alert : @configuration_id has just failed",
"email_body_type": "txt",
"email_body": "Type : @configuration_type\nID : @configuration_id\nImpact : @impact\nEnvironnement : @environnement"
}
}
}
}

🌐
Global monitoring parameters

General parameters about the monitoring.
Parameter
Description
impact
type: integer
optional
'Impact' is an ITIL measure of the extent of the Incident and of the potential damage caused by the Incident before it can be resolved. Learn more
If not specified, the default value will be 2.
urgency
type: integer
optional
'Urgency' is a measure of how quickly a resolution of the Incident is required. Learn more
If not specified, the default value will be 2.
alert_enabled
type: boolean
mandatory
Flag used to enable/disable the execution of the alerting (i.e. send an alert to a recipient when the run failed)
alert_status (beta) type: array
optional
Specifies the Run Status that will trigger the alert.
Possible values: FAILED, SUCCESS, NO_MATCH, CHECKED. Default value: "FAILED"
alert_environment
type: array
mandatory
Specifies the environments that will trigger the alert.
Possible values: PROD, PREPROD, STAGING, DEV.
alert_info
type: string
optional
Short information describing the alert. You can refer it as a variable in your triggering message (as email) with the @alert_info parameter.
alert
type: array of maps
optional
List of alert messages the data operation will trigger if it fails.
Check the section below for detailed information on their parameters.

⚠️
Alert parameters

An alert will be able to trigger different types of messages. Currently, only an email alert can be sent.
For each alert message, parameters will differ depending on the message type.

📨
Email alert

An email alert will send an email with specific parameters each time the data operation fails in the specified environments.
By default, Tailer provides an email template with detailed information. Thus, you don't have to fill in all the parameters such as subject, body, etc. But you can also personalize all the parameters to set precisely how the alert email should look like. For that, you can use the alert variables described below.
Parameter
Description
email_from
type: string
optional
Surcharge the "email from" attribute.
Default value: "[email protected]"
email_to
type: string
mandatory
List of email recipients
You can specify more than one recipient by separating the email addresses with a ;
email_reply_to
type: string
optional
Surcharge the "email reply to" attribute.
Default value: "[email protected]"
email_subject
type: string
optional
Subject of the triggered alert email.
Default value: "TAILER RUN ALERT: @jobid FAILED at @execution_date"
You can personalize the subject with the alert variables described below.
email_body_type
type: string
optional
Format type of the email body.
Default value: "html"
Other possible value: "txt"
email_body
type: string
optional
Body of the alert email.
Default value: "See attached html file below"
You can personalize the body with the alert variables described below. Be careful to use the right body format according to the email_body_type parameter.
Tailer Alert Failed Email Template.html
42KB
Binary
Default Alert Failed Email Template.html

🧩
Alert message variables

Alert messages can be personalized with alert message variables. Those variables are contextualized during the data operation run.
Variable
Description
@url_to_tailer_studio_run_id
Tailer Studio URL for the run
@url_to_tailer_studio_conf_id
Tailer Studio URL for the run current configuration ID
@account
Account of the data operation's run
@environment
Environment of the data operation's run
@status
Status of the data operation's run
@configuration_id
Configuration ID of the data operation
@configuration_type
Configuration type of the data operation's run
@execution_date
Execution date of the data operation's run
@short_description
Short description of the data operation
@job_id
Job ID of the data operation
@run_id
Run ID of the data operation's run
@updated_by
Email address of the user who last updated the configuration
@update_date
Date tile of the last configuration update
@impact
Monitoring impact level parameter
@urgency
Monitoring urgency level parameter
@alert_info
Alert info set in the monitoring parameter