# Configuration file for data decryption

The configuration file is in JSON format. It contains the following sections:

* Global parameters: General information about the data operation.
* Tasks parameters: One or several task blocks, containing information about the specific data operation.
* Credential parameters: Information about the credentials for the buckets and the PGP public key.

## :eye\_in\_speech\_bubble: Example

Here is an example of File Utilities configuration file for data decryption:

```json
{
    "configuration_type": "file-utilities",
    "configuration_id": "000010-file-utilities_demo",
    "environment": "DEV",
    "account": "000099",
    "activated": true,
    "archived": false,
    "version": "2",
    "doc_md": "readme.md",
    "gcp_project_id": "my-project",
    "gcs_bucket": "my-bucket",
    "gcs_path": "input",
    "gcs_destination_suffix": "input_decrypt",
    "launch_mode": "gcs",
    "filename_templates": [{
            "filename_template": "SCORES_{{FD_DATE}}-{{FD_BLOB_36}}.{{FD_BLOB_13}}.csv.gz.gpg",
            "file_description": "Scores"
        },
        {
            "filename_template": "ADR_{{FD_DATE}}-{{FD_BLOB_36}}.csv.gpg",
            "file_description": "Addresses"
        }
    ],
    "task_dependencies": [
        "pgp_decrypt"
    ],
    "tasks": [{
        "task_id": "pgp_decrypt",
        "task_type": "pgp",
        "pgp_mode": "decrypt",
        "private_key.pgp": {
            "passphrase": {
                "cipher_aes": "af3...",
                "tag": "fl9...",
                "ciphertext": "1e2...",
                "enc_session_key": "qvt..."
            },
            "recipient": "me@my-domain.com",
            "content": {
                "cipher_aes": "hk5...",
                "tag": "dfh...",
                "ciphertext": "cj5...",
                "enc_session_key": "2fk..."
            }
        }
    }],
    "credentials": {
        "gcp-credentials.json": {
            "content": {
                "cipher_aes": "gf5...", 
                "tag": "cvh...", 
                "ciphertext": "4et...", 
                "enc_session_key": "g5d..."
            }
        }
    }
}
```

## :globe\_with\_meridians: Global parameters

| Parameter                                                                               | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| --------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| <p><strong>configuration\_type</strong></p><p>type: string</p><p>mandatory</p>          | <p>Type of data operation.</p><p>For an File Utilities data operation, the value is always "file-utilities"</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| <p><strong>configuration\_id</strong></p><p>type: string</p><p>mandatory</p>            | <p>ID of the data operation.</p><p>You can pick any name you want, but is has to be <strong>unique</strong> for this data operation type.</p><p>Note that in case of conflict, the newly deployed data operation will overwrite the previous one.<br>To guarantee its uniqueness, the best practice is to include in your data operation name:</p><ul><li>your account ID</li><li>the source bucket</li><li>the source path</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| <p><strong>environment</strong></p><p>type: string</p><p>mandatory</p>                  | <p>Deployment context.</p><p>Values: PROD, PREPROD, STAGING, DEV.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| <p><strong>account</strong></p><p>type: string</p><p>mandatory</p>                      | Your account ID is a 6-digit number assigned to you by your Tailer Platform administrator.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| <p><strong>activated</strong></p><p>type: boolean</p><p>optional</p>                    | <p>Flag used to enable/disable the execution of the data operation.</p><p><em>Default value: true</em></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| <p><strong>archived</strong></p><p>type: boolean</p><p>optional</p>                     | <p>Flag used to enable/disable the visibility of the data operation's configuration and runs in Tailer Studio.</p><p><em>Default value: false</em></p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |
| <p><strong>doc\_md</strong></p><p>type: string</p><p>optional</p>                       | Path to a file containing a detailed description of the data operation. The file must be in Markdown format.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| <p><strong>version</strong></p><p>type: string</p><p>optional</p>                       | Enter 2 is you are using context & environment and 1 if not.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
| <p><strong>gcp\_project\_id</strong></p><p>type: string</p><p>mandatory</p>             | <p>Set the project where deploy the configuration and the associated cloud functions.</p><p>If not set, the user will be prompted to choose a project id.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
| <p><strong>gcs\_bucket</strong></p><p>type: string</p><p>mandatory</p>                  | Name of the bucket.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
| <p><strong>gcs\_path</strong></p><p>type: string</p><p>mandatory</p>                    | Path where the files will be found, e.g. "some/sub/dir".                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
| <p><strong>gcs\_destination\_suffix</strong></p><p>type: string</p><p>mandatory</p>     | Google Cloud Storage destination path, e.g. "/subdir/subdir\_2" to send the files to "gs\://BUCKET/subdir/subdir\_2/source\_file.ext"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| <p><strong>launch\_mode</strong></p><p>type: string</p><p>mandatory</p>                 | Choice of triggering system. Choose "gcs" to trigger the operation on file creation on a bucket. Futur modes will be implemented.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
| <p><strong>filename\_templates</strong></p><p>type: string</p><p>mandatory</p>          | <p>List of filename templates that will be processed.</p><p>You can set the value to "\*" for all files to be copied. However, this is <strong>not recommended</strong>, as unnecessary or sensitive files might be included by mistake. Besides, the date value specified in <strong>filename\_template</strong> will be used to sort files in the archive folder. If no date value is specified, all files will be stored together under one folder named <strong>/ALL</strong>.</p><p>The best practice is to specify one or more filename templates with the <strong>filename\_template</strong> and <strong>file\_description</strong> parameters as described in the next paragraphe.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| <p><strong>task\_dependencies</strong></p><p>type: array of strings</p><p>mandatory</p> | <p>The <strong>task\_dependencies</strong> parameter allows you to create dependencies between the different tasks specified in the <strong>workflow</strong> parameter (see below). It will define in which order the workflow tasks will run, some of them running concurrently, others sequentially.</p><p><strong>Syntax</strong></p><ul><li>The double chevron <code>>></code> means that the first task needs to be completed before the next one can start.</li><li>The comma <code>,</code> means that the tasks will run concurrently.</li><li>The square brackets <code>\[</code> and <code>]</code> allow you to define a set of tasks that will run together.</li></ul><p>For detailed information about the syntax, refer to the <a href="https://airflow.apache.org/concepts.html#bitshift-composition">Airflow documentation</a>.</p><p><strong>Example 1</strong></p><p>We have the following tasks that we want to run sequentially: taskA (create\_gbq\_table), taskB (sql) and taskC (copy\_gbq\_table).<br>The <strong>task\_dependencies</strong> parameter will be as follows: <code>"task\_dependencies": \[" taskA >> taskB >> taskC "],</code></p><p><strong>Example 2</strong></p><p>We have the following tasks that we want to run concurrently: taskA, taskB and taskC.</p><p>The <strong>task\_dependencies</strong> parameter will be as follows: <code>"task\_dependencies": \[" taskA, taskB, taskC "],</code><br></p><p><strong>Example 3</strong></p><p>We have the following 9 tasks we want to order: taskA, taskD, taskG (create\_gbq\_table), taskB, taskE, taskH (sql), taskC, taskF, taskI (copy\_gbq\_table).<br>The <strong>task\_dependencies</strong> parameter will be as follows:<code>"task\_dependencies": \[" \[taskA, taskD, taskG] >> \[taskB, taskE, taskH] >> \[taskC, taskF, taskI] "],</code></p><p><strong>Example 4</strong></p><p>In the example above, we want taskH to run before taskE so we can use its result for taskE.</p><p>The <strong>task\_dependencies</strong> parameter will be as follows:</p><p><code>"task\_dependencies": \[" \[taskA, taskD, taskG] >> taskH >> \[taskB, taskE] >> \[taskC, taskF, taskI] "],</code></p> |
| <p><strong>credentials</strong></p><p>type:array</p><p>mandatory</p>                    | <p>Encrypted credentials needed to read/move data from the source bucket.</p><p>You should have generated credentials when <a href="../../../getting-started/set-up-google-cloud-platform">setting up GCP</a>. To learn how to encrypt them, refer to <a href="../../../getting-started/encrypt-your-credentials">this page</a>.</p>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |

## :desktop: PGP decrypt task parameters

Information related to the Google Cloud Compute Engine VM where the script will be executed.

| Parameter                                                                 | Description                                                                                                                                                                                                                                                                                                                                                                   |
| ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| <p><strong>task\_id</strong></p><p>type: string</p><p>mandatory</p>       | ID of the task. It must be unique within the data operation.                                                                                                                                                                                                                                                                                                                  |
| <p><strong>task\_type</strong></p><p>type: string</p><p>mandatory</p>     | The value has to be set to "pgp" for this task type.                                                                                                                                                                                                                                                                                                                          |
| <p><strong>pgp\_mode</strong></p><p>type: string</p><p>mandatory</p>      | <p>PGP mode.</p><p>For data decryption, the value is always "decrypt".</p>                                                                                                                                                                                                                                                                                                    |
| <p><strong>private\_key.pgp</strong></p><p>type:array</p><p>mandatory</p> | <p>Encrypted <strong>private</strong> key. This array contains three entities:</p><p>- the passphrase "password" of the private key<br>- the recipient "username" of the private key<br>- the content "schema" credentials after passing it through <a href="https://docs.tailer.ai/getting-started/encrypt-your-credentials#encrypt-the-credentials">tailer encrypt</a>.</p> |