Storage to Storage configuration file

This is the description of the JSON configuration file of a Storage to Storage data operation.

The configuration file is in JSON format. It contains the following sections:

  • Global parameters: General information about the data operation.

  • Source parameters: One source block, containing information about the data source.

  • Destination parameters: One or several destination blocks, containing information about the data destinations.

👁️‍🗨️ Example

Here is an example of STS configuration file for a GCS to SFTP transfer:

{
  "$schema": "http://jsonschema.tailer.ai/schema/storage-to-storage-veditor",
  "configuration_type": "storage-to-storage",
  "configuration_id": "copy-my-files-gcs-to-sftp",
  "doc_md": "readme.md",
  "environment": "PROD",
  "account": "000099",
  "version": "3",
  "activated": true,
  "archived": false,
  "filename_templates": [
    {
      "filename_template": "{{FD_DATE}}_sales_file.txt",
      "file_description": "This is a description for sales_file.txt."
    },
    {
      "filename_template": "{{FD_DATE}}_products.txt",
      "file_description": "This is a description for proucts.txt."
    }
  ],
  
  "source": {
    "type": "gcs",
    "gcp_project_id": "my_gcp_project",
    "gcs_source_bucket" : "my-input-bucket",
    "gcs_source_prefix" : "input-folder",
    "archive_prefix": "archive-folder",
    "gcp_credentials_secret": {
      "cipher_aes": "b42724dcbbf0aba89a0f106d1c4",
      "tag": "5c8816ea0a7aded7c6f2df61f5b9",
      "ciphertext": "fd096e",
      "enc_session_key": "8f6f7c"
    }
  },
  
  "destinations": [
    {
      "type": "sftp",
      "generate_top_file": "REPLACE_EXTENSION",
      "sftp_host": "sftp.domain.com",
      "sftp_port": 22,
      "sftp_userid": "john_doe",
      "sftp_password_secret": {
        "cipher_aes": "3926f71cd00d10b07d0fee4e",
        "tag": "1f5c066351d91041343a2ab37aebe",
        "ciphertext": "921776fd04228fe8aaa42af04",
        "enc_session_key": "2fb0ad2b0df9771"
      },
      "sftp_destination_dir": "/",
      "sftp_destination_dir_create": false
    }
  ]
}

🌐 Global parameters

Parameter
Description

$schema type: string optional

The url of the json-schema that contains the properties that your configuration must verify. Most Code Editor can use that to validate your configuration, display help boxes and enlighten issues.

configuration_type

type: string

mandatory

Type of data operation.

For an STS data operation, the value is always "storage-to-storage".

configuration_id

type: string

mandatory

ID of the data operation.

You can pick any name you want, but is has to be unique for this data operation type.

Note that in case of conflict, the newly deployed data operation will overwrite the previous one. To guarantee its uniqueness, the best practice is to name your data operation by concatenating:

  • your account ID,

  • the source bucket name,

  • and the source directory name.

environment

type: string

mandatory

Deployment context.

Values: PROD, PREPROD, STAGING, DEV.

account

type: string

mandatory

Your account ID is a 6-digit number assigned to you by your Tailer Platform administrator.

version

type: string

optional

Version of the configuration in order to use new features.

filename_templates

type: string

mandatory

List of filename templates that will be processed.

You can set the value to "*" for all files to be copied. However, this is not recommended, as unnecessary or sensitive files might be included by mistake. Besides, the date value specified in filename_template will be used to sort files in the archive folder. If no date value is specified, all files will be stored together under one folder named /ALL.

The best practice is to specify one or more filename templates with the filename_template and file_description parameters as described in the next paragraphe.

activated

type: boolean

optional

Flag used to enable/disable the execution of the data operation.

If not specified, the default value will be "true".

archived

type: boolean

optional

Flag used to enable/disable the visibility of the data operation's configuration and runs in Tailer Studio.

If not specified, the default value will be "false".

max_active_runs

type: integer

optional

This parameter limits the number of concurrent runs for this data operation.

If not set, the default value is 5.

empty_file_policy

type: string

optional

This parameter will tell Tailer how to behave when an empty file (0 bytes) is read.

  • "NONE" the file is ignored and left in place.

  • "PROCESS" the file will be processed normally.

  • "TRIGGER_FAILED_STATUS" a Tailer run will be traced and set to FAILED.

  • "TRIGGER_SUCCESS_STATUS" a Tailer run will be traced and set to SUCCESS.

The default value is: "NONE"

short_description

type: string

optional

Short description of the Data Operation

doc_md

type: string

optional

Path to a file containing a detailed description. The file must be in Markdown format.

"Filename Templates" sub-object parameters

The "filename_templates" object contains the definition of expected source files to copy to the destinations.

Parameter
Description

filename_template

type: string

mandatory

Template for the files to be processed. The following placeholders are currently supported:

  • "FD_DATE" looks for an 8-digit date (e.g. "20191015").

  • "FD_DATE_YEAR_4" looks for 4-digit year (e.g "2021").

  • "FD_DATE_YEAR_2" looks for 2-digit year (e.g "21").

  • "FD_DATE_MONTH" looks for 2-digit month (e.g "05").

  • "FD_DATE_DAY" looks for 2-digit day (e.g "12").

  • "FD_TIME" looks for a 6-digit time (e.g. "124213").

  • "FD_BLOB_XYZ", where XYZ is a non-zero positive integer, looks for a string of characters of XYZ length.

Information:

  • if "FD_DATE" is specified, it will have priority upon "FD_DATE_YEAR_X".

  • if "FD_DATE_YEAR_4" or "FD_DATE_YEAR_2" is specified, the final date will be concatenated with "FD_DATE_MONTH" and "FD_DATE_DAY".

  • if "FD_DATE_YEAR_2" is specified, it will be prefixed by "20".

  • if "FD_DATE_YEAR_4" or "FD_DATE_YEAR_2" is specified only "FD_DATE_MONTH" and "FD_DATE_DAY" will be set to "01".

Example 1

This template:

"stores_{{FD_DATE}}{{FD_TIME}}.txt"

will allow you to process this type of files:

"stores_20201116_124213.txt"

Example 2

This template:

"{{FD_DATE}}{{FD_BLOB_5}}fixedvalue_{{FD_BLOB_11}}.gz"

will allow you to process this type of files:

"20201116_12397_fixedvalue_12312378934.gz"

file_description

type: string

optional

Short description of the files that will match the filename template.

⬇️ Source parameters

There can only be one source block, as STS data operations can only process one source at a time.

Google Cloud Storage source

Example:

{
  "source": {
    "type": "gcs",
    "gcp_project_id": "my_gcp_project",
    "gcs_source_bucket" : "my_bucket",
    "gcs_source_prefix" : "INPUT_SOMEDIR",
    "archive_prefix": "archive",
    "gcp_credentials_secret": {
      "cipher_aes": "b42724dcbbf6c3310aba89a0f106d1c4",
      "tag": "5c8816ea0a7aded9cb47c6f2df61f5b9",
      "ciphertext": "fdf09c6e",
      "enc_session_key": "8f63f7c"
    }
  }
}

Parameter

Description

type

type: string

mandatory

Type of source.

In this case : "gcs".

gcp_project_id

type: string

mandatory

Set the project where deploy the source configuration and associated cloud functions

If not set, the user will be prompted to choose a profile where deploy the configuration

gcs_source_bucket

type: string

mandatory

Name of the source bucket.

gcs_source_prefix

type: string

mandatory

Path where the files will be found, e.g. "some/sub/dir".

archive_prefix

type: string

optional

Path where the source files will be archived.

If present and populated, the STS data operation will archive the source files in the location specified, in the GCS source bucket.

If not present or empty, there will be no archiving.

gcp_credentials_secret

type: dict

mandatory

Encrypted credentials needed to read/move data from the source bucket.

You should have generated credentials when setting up GCP. To learn how to encrypt them, refer to this page.

Amazon S3 source

Example:

{
  "source": {
    "type": "s3",
    "s3_source_bucket": "my_s3_bucket",
    "s3_source_prefix": "input/my_source/",
    "archive_prefix": "archive"
    "aws_access_key": "3VJ3F6JJQBA2",
    "aws_access_key_secret": {
      "cipher_aes": "e6f5a68d4de8af89e83ea93e42facbed",
      "tag": "20e174e34c5d0c537be77d85ed8dda33",
      "ciphertext": "60a98b884110aab84",
      "enc_session_key": "9c4648e"
    }
  }
}

Parameter

Description

type

type: string

mandatory

Type of source.

In this case : "s3".

s3_source_bucket

type: string

mandatory

Name of the source S3 bucket.

s3_source_prefix

type: string

mandatory

Path where the files will be found, e.g. "some/sub/dir".

archive_prefix

type: string

optional

Path where the source files will be archived.

If present and populated, the STS data operation will archive the source files in the location specified, in the GCS source bucket.

If not present or empty, there will be no archiving.

aws_access_key

type: string

mandatory

Amazon S3 access key ID.

aws_access_key_secret

type: dict

mandatory

Encrypted Amazon S3 access private key.

This is needed to read/move data from the source bucket. To learn how to encrypt the private key value, refer to this page.

Azure source

Example:

{
    "source": {
        "type": "azure",
        "azure_source_storage": "my_azure_storage",
        "azure_source_prefix": "input/my_source/",
        "archive_prefix": "archive",
        "azure_connection_string_secret": {
            "cipher_aes": "f1c4xxxx",
            "tag": "0052fxxxx",
            "ciphertext": "7e1a3xxxx",
            "enc_session_key": "2dc2xxxx"
        }
    }
}

Parameter

Description

type

type: string

mandatory

Type of source.

In this case : "azure"

azure_source_storage

type: string

mandatory

Name of the source Azure storage.

azure_source_prefix

type: string

mandatory

Path where the files will be found, e.g. "some/sub/dir".

archive_prefix

type: string

optional

Path where the source files will be archived.

If present and populated, the STS data operation will archive the source files in the location specified, in the GCS source bucket.

If not present or empty, there will be no archiving.

azure_connection_string_secret

type: dict

mandatory

Encrypted Azure access private key.

This is needed to read/move data from the source bucket. To learn how to encrypt the private key value, refer to this page.

SFTP source

Example:

{
  "source": {
    "type": "sftp",
    "sftp_source_directory": "/",
    "sftp_source_filename": "20190621_test_file.txt",
    "archive_prefix": "archive",
    "sftp_host": "sftp.domain.com",
    "sftp_port": 22,
    "sftp_userid": "john_doe",
    "sftp_authentication_method": "USERNAME_PASSWORD",
    "sftp_password_secret": {
      "cipher_aes": "3926f71cd00f8d2b812d10b07d0fee4e",
      "tag": "1f5c066351db5f91041343a2ab37aebe",
      "ciphertext": "921776fd0caa71a04228fe8aaa42af04",
      "enc_session_key": "2fb2f8d271"
    }
  }
}

Parameter

Description

type

type: string

mandatory

Type of source.

In this case : "sftp".

sftp_source_directory

type: string

mandatory

Sub-path to switch to before downloading the file.

sftp_source_filename

type: string

mandatory

File to retrieve.

archive_prefix

type: string

optional

Path where the source files will be archived.

If present and populated, the STS data operation will archive the source files in the location specified, in the GCS source bucket.

If not present or empty, there will be no archiving.

sftp_host

type: string

mandatory

SFTP host, e.g. "sftp.something.com".

sftp_port

type: integer

mandatory

SFTP port, e.g. "22".

sftp_userid

type: string

mandatory

SFTP user ID, e.g. "john_doe".

sftp_authentication_method

type: string

optional

Authentication method used to connect to the SFTP server.

The following methods are supported:

  • USERNAME_PASSWORD

  • PRIVATE_KEY

Default : USERNAME_PASSWORD

sftp_password_secret

type: dict

optional

Encrypted SFTP password for the user ID.

This is needed to read/move data from the source SFTP.

This attribute MUST be set if sftp_authentication_method is set to USERNAME_PASSWORD

To learn how to encrypt the password, refer to this page.

sftp_private_key_secret type: dict

optional

Encrypted SFTP private key.

This attribute MUST be set if sftp_authentication_method is set to PRIVATE_KEY.

To learn how to encrypt the password, refer to this page.

sftp_private_key_passphrase_secret type: dict

optional

Encrypted SFTP private key passphrase if provided

This attribute MUST be set if sftp_authentication_method is set to PRIVATE_KEY.

To learn how to encrypt the password, refer to this page.

⬆️ Destination parameters

These parameters allow you specify a list of destinations. You can add as many "destination" sub-objects as you want, they will all be processed.

Google Cloud Storage destination

Example:

{
  "destinations": [
    {
      "type": "gcs",
      "gcs_destination_bucket": "my_dest_bucket",
      "gcs_destination_prefix": "DEV/output",
      "gcp_credentials_secret": {
        "cipher_aes": "b42724dcbbf6c3310aba89a0f106d1c4",
        "tag": "5c8816ea0a7aded9cb47c6f2df61f5b9",
        "ciphertext": "fdf09c6e",
        "enc_session_key": "8f634f7f7c"
      }
    }
  ]
}
Parameter
Description

type

type: string

mandatory

Type of destination.

In this case : "gcs".

gcs_destination_bucket

type: string

mandatory

Google Cloud Storage destination bucket.

gcs_destination_prefix

type: string

mandatory

Google Cloud Storage destination path, e.g. "/subdir/subdir_2" to send the files to "gs://BUCKET/subdir/subdir_2/source_file.ext"

gcp_credentials_secret

type: dict

mandatory

Encrypted credentials needed to read/write/move data from the destination bucket.

You should have generated credentials when setting up GCP. To learn how to encrypt them, refer to this page.

Amazon S3 destination

Example:

{
  "destinations": [
    {
      "type": "s3",
      "s3_bucket" : "my_dest_bucket",
      "s3_destination_prefix": "PROD/output",
      "aws_access_key": "J3F6JLUVJQ",
      "aws_access_key_secret": {
        "cipher_aes": "e6f5a68dxxxx",
        "tag": "20e1xxxx",
        "ciphertext": "60a84xxxx",
        "enc_session_key": "9c4619048exxxx"
      }
    }
  ]
}
Parameter
Description

type

type: string

mandatory

Type of destination.

In this case : "s3".

s3_bucket

type: string

mandatory

Amazon S3 bucket name.

s3_destination_prefix

type: string

mandatory

Amazon S3 destination path, e.g. "subdir_A/subdir_B" to send the files to "s3://bucket/subdir_A/subdir_B/source_file.ext".

aws_access_key

type: string

mandatory

Amazon S3 access key ID.

aws_access_key_secret

type: dict

mandatory

Encrypted Amazon S3 access private key.

This is needed to read/write/move data from the destination bucket.

To learn how to encrypt the private key value, refer to this page.

Azure destination

Example:

{
  "destinations": [
    {
      "type": "azure",
      "azure_destination_prefix": "my_azure_bucket/output",
      "azure_connection_string_secret": {
        "cipher_aes": "3926fxxxx",
        "tag": "1f5cxxxx",
        "ciphertext": "9217xxxx",
        "enc_session_key": "2fb0xxxx"
      }
    }
  ]
}
Parameter
Description

type

type: string

mandatory

Type of destination.

In this case : "azure".

azure_destination_prefix

type: string

mandatory

Complete Azure destination path, i.e. storage name and subdirectory if needed.

azure_connection_string_secret

type: dict

mandatory

Encrypted Azure access private key.

This is needed to read/move data from the source bucket. To learn how to encrypt the private key value, refer to this page.

SFTP destination

Example:

{
  "destinations": [
    {
      "type": "sftp",
      "generate_top_file": "REPLACE_EXTENSION",
      "sftp_destination_dir": "/",
      "sftp_destination_dir_create": false,
      "sftp_host": "sftp.domain.com",
      "sftp_port": 22,
      "sftp_userid": "john_doe",
      "sftp_password_secret": {
        "cipher_aes": "3926f71cd00f8d2b812d10b07d0fee4e",
        "tag": "1f5c066351db5f91041343a2ab37aebe",
        "ciphertext": "921776fd0caa71a04228fe8aaa42af04",
        "enc_session_key": "2fb0adf8d271"
      }
    }
  ]
}
Parameter
Description

type

type: string

mandatory

Type of destination.

In this case : "sftp".

generate_top_file

type: string

optional

This flag, if set, will generate a TOP file along with the file copied.

Possible values are:

  • "REPLACE_EXTENSION": if the source file is "20190708_data.txt", the TOP file will be "20190708_data.top".

  • "ADD_EXTENSION": if the source file is "20190708_data.txt", the TOP file will be "20190708_data.txt.top".

sftp_destination_dir

type: string

mandatory

Path to switch to before uploading the file.

sftp_destination_dir_create

type: string

mandatory

Will try to create the subdir specified in sftp_destination_dir on the SFTP filesystem before switching to it and copying files.

sftp_host

type: string

mandatory

SFTP host, e.g. "sftp.something.com".

sftp_port

type: string

mandatory

SFTP port, e.g. "22".

sftp_userid

type: string

mandatory

SFTP user ID, e.g. "john_doe".

sftp_authentication_method

type: string

optional

Authentication method used to connect to the SFTP server.

The following methods are supported:

  • USERNAME_PASSWORD

  • PRIVATE_KEY

Default : USERNAME_PASSWORD

sftp_password_secret

type: dict

optional

Encrypted SFTP password for the user ID.

This is needed to read/move data from the source SFTP.

This attribute MUST be set if sftp_authentication_method is set to USERNAME_PASSWORD.

To learn how to encrypt the password, refer to this page.

sftp_private_key_secret type: dict

optional

Encrypted SFTP private key.

This attribute MUST be set if sftp_authentication_method is set to PRIVATE_KEY.

To learn how to encrypt the password, refer to this page.

sftp_private_key_passphrase_secret type: dict

optional

Encrypted SFTP private key passphrase if provided.

This attribute MUST be set if sftp_authentication_method is set to PRIVATE_KEY.

To learn how to encrypt the password, refer to this page.

PreviousMove files with Storage to StorageNextLoad data with Storage to Tables

Last updated