Set up Google Cloud Platform
Some features of Tailer Platform rely on Google Cloud Platform's resources, or allow you to manipulate them. Some steps need to be performed as preparation.
Create a dedicated Google Cloud Platform project to use with Tailer Platform. Billing needs to be enabled.
To create a new project:
- 1.
- 2.In the organization drop-down list at the top of the page, select the organization in which you want to create a project.
- 3.Click Create Project.
- 4.In the New Project window that appears, enter a project name and select a billing account as applicable.
- 5.If you want to add the project to a folder, enter the folder name in the Location box.
- 6.When you're finished entering new project details, click Create.
The following Google Cloud APIs need to be enabled for the project you have created:
- Cloud Functions
- Identity and Access Management (IAM)
- Cloud Resource Manager
To enable an API:
- 1.
- 2.Click the Enable APIs and Services button.
- 3.Click the API you want to enable. If you need help finding the API, use the search field.
- 4.In the page that displays information about the API, click Enable.
When you enable the Cloud Functions API, a service account is automatically created for your project. It should be named as follows:
You need to grant roles to this service account so that it has permission to complete specific actions on some resources in your Cloud Platform project.
To grant roles to the service account:
- 1.
- 2.
- 3.Add the Project > Editor and Service Account > Service Account Token Creator roles.
- 4.Click Save to apply the roles to the service account.
The App Engine default account will need to access the following elements of Tailer Platform:
- Composer (Airflow): to trigger DAGs
- Firestore: to retrieve data operations
- Source Repositories: to retrieve Cloud Functions source code
The GCP project hosting the Composer and Firestore instances should already have groups with the appropriate permissions. You have to add the App Engine default account to these groups.
This generic service account will be used among other things for:
- Deploying configurations on behalf of an authorized user
- Moving files from one bucket to another
You will need a dedicated service account for each GCP project you will use Tailer with (for example if they contain a source or destination bucket used in a transfer operation). We recommend you only create one service account per project to avoid right administration becoming too complex.
To create the service account:
- 1.
- 2.Click Select a project, choose your project, and click Open.
- 3.Click Create Service Account.
- 4.You can use YOUR-PROJECT-ID as a name for the service account.
- 5.Click Save.
You should now have a new service account named as follows:
To generate JSON credentials:
- 1.In the Service Accounts page of the Cloud Console, find the row of the [email protected] service account that you've just created.
- 2.In that row, click the Morebutton, and then click Create key.
- 3.Select JSON as Key type and click Create.
When you create a key, your new public/private key pair is generated and downloaded to your machine. It serves as the only copy of the private key. You are responsible for storing the private key securely.
These credentials will need to be encrypted, so you can use them later in a data operation JSON configuration file.