Skip to content

Database sync to bigquery

Database Sync to BigQuery

We will use Airbyte to sync data from a PostgreSQL database to BigQuery. The BigQuery instance is part of a GCP project managed by our client -Tailorframe.

Setup Details

  1. PostgreSQL User: We created a PostgreSQL user with read permissions to access the necessary tables.
  2. GCP Project Permissions: I have permissions to create a BigQuery dataset and a service account. Client grand permissions to that service account with with read and write permissions to BigQuery.

Sync Process

  • The connection will sync every 3 hours "incremental" to ensure data is up-to-date.

external_BigQuery project name “root-talon-455111-q5” 1. Created new dataset 1. Dataset ID:  tailorframe_admin_stable 2. Disable table expiration 3. Data location : europe-north1 (Finland) 4. Key Created: Apr 2, 2025, 2:49:41 PM UTC+2

  1. Created service account with permissions:
    1. BigQuery User 
    2. BigQuery Data Editor
    3. airbyte-tailorframe-stable@root-talon-455111-q5.iam.gserviceaccount.com
    4. Key Creation date: Apr 2, 2025

Tables synced: * device_7a1l0fr2 * photo_7a1l0fr2 * photo_code_7a1l0fr2 * team_code_7a1l0fr2 * team_photo_7a1l0fr2

Conclusion: We are syncing data from db:“tailorframe_admin_stable”, using Airbyte and sending data to BigQuery managed by Tailorframe.

First data sync started: 04.03.2025