A Storage to Tables (STT) data pipeline operation allows you to load data files from a Google Cloud Storage (GCS) bucket into one or several BigQuery databases.
CSV and any delimited flat files
New line delimited JSON files
These two file types can be compressed using gzip
Every time a new file matching the specified rule appears in a given directory of a Google Cloud Storage bucket:
it will be removed from the source directory,
if options have been set accordingly, the file will be copied to an archive directory located in the same storage, inside a folder named with the date contained in the filename,
the file data will be loaded into a BigQuery table matching its filename template for each database specified.
Access your tailer folder (created during installation).
Create a working folder as you want, and create a JSON file for your data operation inside.
Prepare your JSON configuration file. Refer to this page to learn about all the parameters.
Prepare a DDL file for each database table. Refer to this page to learn about all the parameters.
Access your working folder by running the following command:
cd "[path to your working folder]"
To deploy the data operation, run the following command:
tailer deploy your-file.json
Log in to Tailer Studio to check the status and details of your data operation.
Access your output table(s), and archive folder, if any, to check the result of the data operation.