$schema
type: string
optional
| The url of the json-schema that contains the properties that your configuration must verify. Most Code Editor can use that to validate your configuration, display help boxes and enlighten issues. | |configuration\_type
type: string
mandatory
|Type of data operation.
For an STT data operation, the value is always "storage-to-tables".
configuration\_id
type: string
mandatory
|ID of the data operation.
You can pick any name you want, but is has to be unique for this data operation type.
Note that in case of conflict, the newly deployed data operation will overwrite the previous one. To guarantee its uniqueness, the best practice is to name your data operation by concatenating:
version
type: string
mandatory
environment
type: string
mandatory
|Deployment context.
Values: PROD, PREPROD, STAGING, DEV.
account
type: string
mandatory
| Your account ID is a 6-digit number assigned to you by your Tailer Platform administrator. | |activated
type: boolean
optional
|Flag used to enable/disable the execution of the data operation.
If not specified, the default value will be "true".
archived
type: boolean
optional
|Flag used to enable/disable the visibility of the data operation's configuration and runs in Tailer Studio.
If not specified, the default value will be "false".
max\_active\_runs
type: integer
optional
|This parameter limits the number of concurrent runs for this data operation.
If not set, the default value is 50.
| |short\_description
type: string
optional
| Short description of the context of the configuration. | |doc\_md
type: string
optional
| Path to a file containing a detailed description. The file must be in Markdown format. | ## :arrow\_down: Source parameters (GCS) The destination section contains all information related to the data source provider. ```json "source": { "type": "gcs", "gcp_project_id": "my-project", "gcs_source_bucket": "mirror-fd-io-exc-demo-wbd--n-in", "gcs_source_prefix": "testjul", "gcs_archive_prefix": "archive", "gcp_credentials_secret": { "cipher_aes": "223xxx", "tag": "8ddxxx", "ciphertext": "4c7xxx", "enc_session_key": "830xxx" } } ``` | Parameter | Description | | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | |type
type: string
mandatory
|Source type.
The only supported source type for now is "gcs".
| |gcp\_project\_id
type: string
mandatory
|Specify the Google Cloud Platform project where to deploy the data operation and its associated cloud functions.
If not set, the user will be prompted to choose a project.
| |gcs\_source\_bucket
type: string
mandatory
|Name of the source bucket.
⚠️ See restrictions below
gcs\_source\_prefix
type: string
mandatory
|Path where the files will be found, e.g. "some/sub/dir".
⚠️ See restrictions below
gcs\_archive\_prefix
type: string optional
|Path where the source files will be archived.
If present and populated, the STT data operation will archive the source files in the location specified, in the GCS source bucket.
If not present or empty, there will be no archiving.
| |gcp\_credentials\_secret
type: dict
mandatory
|Encrypted credentials needed to read/move data from the source bucket.
You should have generated credentials when setting up GCP. To learn how to encrypt them, refer to this page.
| ## :arrow\_up: Destination parameters (BigQuery) The destination section contains all the information related to the data destinations. The **destinations** parameter is an array containing maps. Each map can contain a type of destination and many actual "tables" as ultimate destination. Example: ```json "destinations": [ { "type": "bigquery", "gcp_project_id": "my-project", "gbq_dataset": "dlk_demo_wbd_psa", "source_format": "CSV", "create_disposition": "CREATE_IF_NEEDED", "write_disposition": "WRITE_TRUNCATE", "skip_leading_rows": 1, "field_delimiter": "|", "add_tailer_metadata": true, "gcp_credentials_secret": { "cipher_aes": "223008256918c292ff3ec1axxx", "tag": "8dd3db4c71bfb963d475d1bbd1xxx", "ciphertext": "4c74df268d5a7541f8264c2e7a282fxxx", "enc_session_key": "830606cbc4f9401a29c7977d364398xxx" }, "tables": [ { "table_name": "sales_details_test", "short_description": "Daily detailed Sales with tickets and tickets lines", "filename_template": "input_{{FD_DATE}}-{{FD_TIME}}-ORS-ventes.csv", "ddl_mode": "file", "ddl_file": "ddl/sales_details.json", "doc_md": "ddl/sales_details.md", "add_tailer_metadata": true, "skip_leading_rows": 0, "write_disposition": "WRITE_APPEND" }, { "table_name": "stores", "short_description": "Full Stores referential", "filename_template": "input_{{FD_DATE}}-{{FD_TIME}}-ORS-magasins.csv", "ddl_mode": "file", "ddl_file": "ddl/stores.json", "doc_md": "ddl/stores.md", "add_tailer_metadata": false } ] } ] ``` ### Global destination parameters | Parameter | Description | | ------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | |type
type: string
mandatory
|Type of destination.
The only supported destination type for now is "bigquery".
| |gcp\_project\_id
type: string
optional
|Default GCP Project ID.
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
| |gbq\_dataset
type: string
optional
|Default BigQuery Dataset.
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
| |gcp\_credentials\_secret
type: object
optional
|Encrypted credentials needed to interact with Storage and BigQuery.
You should have generated credentials when setting up GCP. To learn how to encrypt them, refer to this page.
| |source\_format
type: string
optional
|Default source format for input files.
Possible values (case sensitive):
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
| |create\_disposition
type: string
optional
|Specifies behavior for creating tables (see Google BigQuery documentation).
Possible values:
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
| |write\_disposition
type: string
optional
|Action that occurs if the destination table already exists (see Google BigQuery documentation).
Possible values:
skip\_leading\_rows
type: integer
optional
|Number of rows to skip when reading data, CSV only.
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
Default value: 1
field\_delimiter
type: string
optional
|Separator for fields in a CSV file, e.g. ";".
Note: For Tab separator, set to "\t".
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
Default value:
quote\_character
type: string
optional
|Character used to quote data sections, CSV only (see Google BigQuery documentation).
Note: For quote and double quotes, set to "'" and """ respectively.
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
Default value: ""
null\_marker
type: string
optional
|Represents a null value, CSV only (see Google BigQuery documentation).
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
Default value: ""
bq\_load\_job\_ignore\_unknown\_values
type: boolean
optional
|Ignore extra values not represented in the table schema (see Google BigQuery documentation).
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
Default value: false
bq\_load\_job\_max\_bad\_records
type: integer
optional
|Number of invalid rows to ignore (see Google BigQuery documentation).
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
Default value: 0
bq\_load\_job\_schema\_update\_options
type: array
optional
|Specifies updates to the destination table schema to allow as a side effect of the load job (see Google BigQuery documentation).
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
Default value: \[]
bq\_load\_job\_allow\_quoted\_newlines
type: boolean
optional
|Allows quoted data containing newline characters, CSV only (see Google BigQuery documentation).
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
Default value: false
bq\_load\_job\_allow\_jagged\_rows
type : boolean
optional
|Allows missing trailing optional columns, CSV only (see Google BigQuery documentation).
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
Default value: false
add\_tailer\_metadata
type : boolean
optional
|\[NEW]
Allows automatic metadata feature that add specific columns during the ingestion process related to the input source.
The added columns are:tlr\_ingestion\_timestamp\_utc(TIMESTAMP) tlr\_input\_file\_source\_type(STRING)tlr\_input\_file\_name(STRING) tlr\_input\_file\_full\_resource\_name(STRING)
This parameter can be set for each table sub-object, and will be overridden by that value if it is different.
Default value : false
table\_name
type: string
mandatory
| Name of the destination BigQuery table. | |short\_description
type: string
optional
| Short description of the destination BigQuery table. | |filename\_template
type: string
mandatory
|Template for the files to be processed.
The following placeholders are currently supported:
Information:
Example 1
This template:
"stores\_{{FD\_DATE}}{{FD\_TIME}}.txt"
will allow you to process this type of files:
"stores\_20201116\_124213.txt"
Example 2
This template:
"{{FD\_DATE}}{{FD\_BLOB\_5}}fixedvalue{{FD\_BLOB\_11}}.gz"
will allow you to process this type of files:
"20201116\_12397\_fixedvalue\_12312378934.gz"
Example 3
If table\_name is set to: "table\_{{FD\_TABLE\_NAME}}"
and filename\_template to: "{{FD\_DATE}}\_{{FD\_TIME}}fixedvalue{{FD\_TABLE\_NAME}}.csv"
A file named "20201116\_124523\_fixedvalue\_stores.csv" will be loaded into a table named: "table\_stores\_20191205"
A file named "20190212\_063412\_fixedvalue\_visits.csv" will be loaded into a table named: "table\_visits\_20190212"
| |ddl\_mode
type: string
optional
|This parameter allows you to specify how the schema of the table will be obtained.
Possible values:
Default value: file
| |ddl\_file
type: string
mandatory if ddl\_mode is set to "file"
| Path to the [DDL file](/data-pipeline-operations/load-data-with-storage-to-tables/storage-to-tables-ddl-files.md) where the destination schema is described. | |doc\_md
type: string
optional
add\_tailer\_metadata
type : boolean
optional
|\[NEW]
Allows automatic metadata feature that add specific columns during the ingestion process related to the input source.
The added columns are:tlr\_ingestion\_timestamp\_utc(TIMESTAMP) tlr\_input\_file\_source\_type(STRING)tlr\_input\_file\_name(STRING) tlr\_input\_file\_full\_resource\_name(STRING)
Default value : false