DBT Cloud

This guide provides step-by-step instructions for setting up the sample jobs located in the DBT Cloud example folder.

Prerequisites

Before continuing with this guide, ensure you have completed the following prerequisites:

  • Install the Snoweaver CLI.

  • If you have just installed Snoweaver, ensure you have launched the admin console to initialize the Admin API.

Preparation

  1. Donwlaod or Clone the Quickstart Repository to your local machine.

  2. Navigate to the directory examples/dbtcloud

  3. Open the setup.sh file in a text editor to update the following variables, or alternatively, set their values using environment variables:

    • CONN Specify the Snowsql connection for the Snowflake account, ensuring it points to database: SNOWEAVER and schema: LEARN_DEV. Verify that the role associated with this connection has the required privileges to execute the commands in setup.sql. Ideally, use the AccountAdmin role or an equivalent role in a sandbox environment.

    • USER The Snowflake user defined in the Snowsql connection.

    • DBT_TOKEN Your DBT service account token, which provides the necessary permissions to trigger jobs within specific projects. For more information, refer to the DBT Cloud API service tokens documentation.

    • DBT_URL The URL for the DBT Cloud instance. For the multi-tenant deployment environment, this will be cloud.getdbt.com.

    • DBT_ACCOUNT_ID Your DBT Account ID, which can be located under Settings > Account > Account Information, or in the URL when you open a project. For example: cloud.getdbt.com/deploy/<account_id>/projects/.

Set up the sample project

  1. On your client machine, open a command-line terminal (Linux, macOS) or Git Bash (Windows) and navigate to the same directory.

  2. Execute the following command to set up the example project:

    bash setup.sh

    Note

    If the LEARN_DEV project already exists, all its resources will be deleted, with the exception of the DATA stage and the RESULTS table.

    To retain your data, you can remove the -d flag from the ‘sw import all -c ${CONN} -d’’ command in the setup.sh script.

    Alternatively, use the Snoweaver CLI to create a backup of the project’s contents.

  3. After the execution completes, you can access the sample jobs in the LEARN_DEV project.

Sample jobs

dbt_run_job

This job initiates a DBT Cloud job using the specified project name and job name. The process involves the following steps:

  • In the prehook, it invokes dbt_get_project_id and dbt_get_job_id with the provided project name and job name to retrieve the corresponding job ID.

  • The job is then triggered on DBT Cloud.

  • In the posthook, the check_job_status macro is called to monitor the job’s run status through dbt_check_job, with the status data being captured as the output.

  • The job run information is then saved in the RESULTS table.

dbt_get_project_id

This job retrieves the project ID based on the provided name and raises an exception if no matching project is found.

dbt_get_job_id

This job retrieves the DBT Cloud job ID based on a specified job name. If no matching job is found, it raises an exception.

dbt_check_job_status

This job retrieves the current status of a job run by using the job run ID obtained from the response object variable.

dbt_lib

This library provides two macros:

get_id

This macro searches for a match in the project or job data response based on the specified name. If no match is found, it raises an exception.

check_job_status

This macro monitors the status of a specified job run under the following conditions:

  • Waits for a specified interval (in seconds) and continues polling until the maximum iteration limit is reached.

  • If the upload job is not completed within the allowed iterations, an exception is raised.

  • Returns the response payload and terminates the polling loop when the job transitions to status code 10.

  • Calls raise_exception and terminates the polling loop when the job transitions to status code 20 or 30.