Tables to Tables configuration file

This is the description of the JSON configuration file of a Tables to Tables data operation.

The configuration file is in JSON format. It contains the following sections:

  • Global parameters: General information about the data operation.

  • Workflow task parameters: Information about the different tasks of the workflow.

Here is an example of TTT configuration file:

{
  "$schema": "http://jsonschema.tailer.ai/schema/table-to-table-veditor",
  "configuration_type" : "table-to-table",
  "configuration_id" : "000099_test_sql_dag_v1",
  "short_description" : "Short description of the job",
  "doc_md": "readme.md",
  "account" : "000099",
  "environment" : "PROD",
  "activated": true,
  "archived": false,
  "start_date" : "2023, 1, 23",
  "schedule_interval" : "None",
  "default_gcp_project_id" : "my-tailer-project",
  "default_bq_dataset" : "my_dataset",
  "default_write_disposition" : "WRITE_TRUNCATE",
  "task_dependencies" : [
    "create_collection_plan_table >> customer_value_step1 >> pda_dmp_event",
    "pda_dmp_event >> pda_customers"
  ],
  "workflow" : [
    {
      "id" : "customer_value_step1",
      "gcp_project_id" : "Project_A",
      "bq_dataset" : "Dataset_Z", 
      "table_name" : "customer_value",
      "write_disposition" : "WRITE_TRUNCATE",
      "sql_file" : "customer_value_step1.sql"
    },
    {
      "id" : "pda_dmp_event",
      "gcp_project_id" : "Project_A",
      "bq_dataset" : "Dataset_Y",
      "table_name" : "dmp_event",
      "write_disposition" : "WRITE_TRUNCATE",
      "sql_file" : "pda_dmp_event.sql"
    },
    {
      "id" : "pda_customers",
      "gcp_project_id" : "Project_A",
      "bq_dataset" : "Dataset_X", 
      "table_name" : "customers",
      "write_disposition" : "WRITE_TRUNCATE",
      "sql_file" : "pda_customers.sql"
    },
    {
      "id": "create_collection_plan_table",
      "short_description": "Create fd-io-dlk-pimkie.dlk_pim_pda.collection_plan",
      "task_type": "create_gbq_table",
      "bq_table": "collection_plan",
      "force_delete": true,
      "ddl_file" : "000020_Load_PSA_to_PDA_collection_plan_DDL.json"
    }
  ]
}

General information about the data operation.

A Tables to Tables workflow can include four types of tasks:

  • create_gbq_table: This type of task allows you to create the skeleton of a table based on a DDL file. You could compare it to the baking pan of your table.

  • sql: This type of task allows you to fill that baking pan using a SQL query or a SQL script from a file.

  • copy_gbq_table: This type of task allows you to duplicate a table named X into another table named Y.

  • expectation: This type of task allows you to perform data quality tests. See expectations for more details.

  • delete_gbq_table: This type of task allows you to delete a table or a partition of a table.

For each workflow sub-object, parameters will differ depending on the task type.

Refer to this page to know how to create the DDL and SQL files corresponding to these tasks.

SQL task parameters

Table Creation task parameters

Table copy task parameters

Table Deletion task parameters

Last updated