Data Interface

Snoweaver provisions each project with a table named RESULTS and two internal stages, CODE and DATA, serving as the data interface. These can be accessed by users or other data pipelines.

DATA

File Downloads

Jobs that download files from remote endpoints will place them in this stage, organized into folders named after the jobs. Files downloaded during a test call will be placed in a folder named <job_name>_test_call.

File Uploads

Jobs that upload files to remote endpoints can source their files from this stage. The Developer and Operator roles in the project have write access to this stage, allowing them to unload data into files from other databases if needed.

CODE

Snoweaver uses the CODE stage to import and export project resources as YAML files. The Developer role has access to this stage for uploading and downloading these configuration files.

RESULTS

The RESULTS table stores the outcomes of a Snoweaver job when the Save Job Results option is enabled. Entries within this table can be categorized using the scope label.

Considerations

Storage costs

Data stored in the RESULTS table or project stages incurs storage costs. To manage these costs effectively:

  • Perform regular cleanup: If you don’t intend to keep data for extended periods, implement periodic cleanup processes.

  • Replicate to raw layer: Consider adding a step in your pipeline to replicate new data or files to your raw data layer. This ensures data preservation in case Snoweaver is uninstalled (accidentally or otherwise).

  • Set retention policies: Implement data retention policies aligned with your business needs and compliance requirements.

Data retention

The retention time for the RESULTS table is set to 1 day, with a maximum data extension time of 14 days.