Tailer Documentation
  • What is Tailer Platform?
  • Getting Started
    • Prepare your local environment for Tailer
    • Install Tailer SDK
    • Set up Google Cloud Platform
    • Encrypt your credentials
  • [Tutorial] Create a first data pipeline
    • Introduction
    • Prepare the demonstration environment
    • Copy files from one bucket to another
    • Load files into BigQuery tables
    • Prepare data
    • Build predictions
    • Export data
    • Congratulations!
    • [Video] Automatic Script
      • SQL script file
      • DDL script file
      • Tables to Tables script file
      • Launch configuration and furthermore
  • Data Pipeline Operations
    • Overview
    • Set constants with Context
      • Context configuration file
    • Move files with Storage to Storage
      • Storage to Storage configuration file
    • Load data with Storage to Tables
      • Storage to Tables configuration file
      • Storage to Tables DDL files
    • Stream incoming data with API To Storage
      • API To Storage configuration file
      • API To Storage usage examples
    • Transform data with Tables to Tables
      • Tables to Tables configuration file
      • Table to Table SQL and DDL files
    • Export data with Tables to Storage
      • [V3] Table to Storage configuration file
      • Table to Storage SQL file
      • [V1-V2: deprecated] Table to Storage configuration file
    • Orchestrate processings with Workflow
      • [V2] Workflow configuration file
      • [V1: deprecated] Workflow configuration file
    • Convert XML to CSV
      • Convert XML to CSV configuration file
    • Use advanced features with VM Launcher
      • Process code with VM Launcher
        • VM Launcher configuration file for code processing
      • Encrypt/Decrypt data with VM Launcher
        • VM Launcher configuration file for data encryption
        • VM Launcher configuration file for data decryption
    • Monitoring and Alerting
      • Monitoring and alerting parameters
    • Asserting Data quality with Expectations
      • List of Expectations
    • Modify files with File Utilities
      • Encrypt/Decrypt data with File Utilities
        • Configuration file for data encryption
        • Configuration file for data decryption
    • Transfer data with GBQ to Firestore
      • Table to Storage: configuration file
      • Table to Storage: SQL file
      • VM Launcher: configuration file
      • File-to-firestore python file
  • Tailer Studio
    • Overview
    • Check data operations' details
    • Monitor data operations' status
    • Execute data operations
    • Reset Workflow data operations
    • Archive data operations
    • Add notes to data operations and runs
    • View your data catalog
    • Time your data with freshness
  • Tailer API
    • Overview
    • Getting started
    • API features
  • Release Notes
    • Tailer SDK Stable Releases
    • Tailer Beta Releases
      • Beta features
      • Beta configuration
      • Tailer SDK API
    • Tailer Status
Powered by GitBook
On this page
  • Example
  • 🌐 Global parameters
  • 💼 Working directory parameters
  • 🖥 VM parameters
  • 🔐 Credential parameters

Was this helpful?

Edit on GitHub
  1. Data Pipeline Operations
  2. Use advanced features with VM Launcher
  3. Encrypt/Decrypt data with VM Launcher

VM Launcher configuration file for data decryption

This is the description of the JSON configuration file for a VM Launcher data decryption data operation.

PreviousVM Launcher configuration file for data encryptionNextMonitoring and Alerting

Last updated 2 years ago

Was this helpful?

The configuration file is in JSON format. It contains the following sections:

  • Global parameters: General information about the data operation.

  • Working directory parameters: Information about the input/output directories.

  • Credential parameters: Information about the credentials for the input/output buckets and the PGP private key.

Example

Here is an example of VM Launcher configuration file for data decryption:

{
    "configuration_type": "vm-launcher",
    "configuration_id": "pgp-decrypt",
    "environment": "DEV",
    "account": "000099",
    "activated": true,
    "archive": false,
    "pgp_mode": "DECRYPT",
    "gcp_project_id": "my-project",
    "gcs_source_bucket": "my-bucket",
    "gcs_source_prefix": "input",
    "destination_gcs_bucket": "my-bucket",
    "destination_gcs_path": "input_decrypted",
    "vm_delete": true,
    "credentials": {
        "input-credentials.json": {
            "content": {
                "cipher_aes": "",
                "tag": "",
                "ciphertext": "",
                "enc_session_key": ""
            }
        },
        "output-credentials.json": {
            "content": {
                "cipher_aes": "",
                "tag": "",
                "ciphertext": "",
                "enc_session_key": ""
            }
            
        },
        "private_key.pgp": {
            "passphrase": {
                "cipher_aes": "",
                "tag": "",
                "ciphertext": "",
                "enc_session_key": ""
            },
            "recipient": "my.recipient@domain.com",
            "content": {
                "cipher_aes": "",
                "tag": "",
                "ciphertext": "",
                "enc_session_key": ""
            }
        }
    }
}

🌐 Global parameters

Parameter
Description

configuration_type

type: string

mandatory

Type of data operation.

For an STS data operation, the value is always "storage-to-storage".

configuration_id

type: string

mandatory

ID of the data operation.

You can pick any name you want, but is has to be unique for this data operation type.

Note that in case of conflict, the newly deployed data operation will overwrite the previous one. To guarantee its uniqueness, the best practice is to name your data operation by concatenating:

  • your account ID,

  • "pgp-decrypt",

  • and a description of the data to decrypt.

environment

type: string

mandatory

Deployment context.

Values: PROD, PREPROD, STAGING, DEV.

account

type: string

mandatory

Your account ID is a 6-digit number assigned to you by your Tailer Platform administrator.

activated

type: boolean

optional

Flag used to enable/disable the execution of the data operation.

If not specified, the default value will be "true".

archived

type: boolean

optional

Flag used to enable/disable the visibility of the data operation's configuration and runs in Tailer Studio.

If not specified, the default value will be "false".

pgp_mode

type: string

optional

PGP mode.

For data decryption, the value is always "DECRYPT".

💼 Working directory parameters

Information about the script location and instructions to execute it.

Parameter
Description

gcp_project_id

type: string

mandatory

Google Cloud Platform project ID for the bucket containing the files to decrypt.

gcs_source_bucket

type: string

mandatory

Name of the GCS bucket containing the files to decrypt.

gcs_source_prefix

type: string

mandatory

Path in the GCS bucket containing the files to decrypt, e.g. "some/sub/dir".

destination_gcs_bucket

type: dict

mandatory

Name of the GCS bucket containing the decrypted files.

destination_gcs_path

type: array

mandatory

Path in the GCS bucket containing the decrypted files, e.g. "some/sub/dir".

🖥 VM parameters

Information related to the Google Cloud Compute Engine VM where the script will be executed.

Parameter
Description

vm_delete

type: string

mandatory

If set to "true", this parameter will force the deletion of the VM at the end of the data operation. Running Compute Engine VMs will incur extra costs, so it is recommended to leave this parameter on "true".

Default value: true

vm_core_number

type: string

mandatory

Virtual CPU (vCPU) count. It is recommended to leave the default parameter, as this should allow sufficient performance to run a standard script.

Default value: 2

vm_memory_amount

type: string

mandatory

System memory size (in GB).

It is recommended to leave the default parameter, as this should allow sufficient performance to run a standard script.

Default value: 4

vm_disk_size

type: string

mandatory

Persistent disk size (in GB).

It is recommended to leave the default parameter, as this should provide enough space to store the data to process.

Default value: 20

🔐 Credential parameters

Parameter
Description

credentials

type:array

mandatory

Array containing three entities: input credentials for the input bucket, output credentials for the output bucket, and the private PGP key.

👁️‍🗨️