Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub topics: cloud-composer

Repositories

wintermi/cloud-composer-trigger

Examples on how to Trigger a Google Cloud Composer DAG

Language: Go - Size: 79.1 KB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 1 - Forks: 0

Giray18/Data-Engineering-on-GCP

A repo containing auto-triggered Airflow ETL activities for datasets located on GCP storage that flattens and creates analytical views on Big Query

Language: Python - Size: 694 KB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 1 - Forks: 0

GoogleCloudPlatform/public-datasets-pipelines

Cloud-native, data onboarding architecture for Google Cloud Datasets

Language: Python - Size: 7.12 MB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 136 - Forks: 61

doitintl/doit-composer-airflow-training

Getting started with Apache Airflow on Cloud Composer

Language: Python - Size: 4.24 MB - Last synced: about 1 month ago - Pushed: about 2 years ago - Stars: 29 - Forks: 4

kamikennn/dbt-bigquery-cloud-composer-tutorial

DBT execution from Cloud Composer. BigQuery is used for main DWH and Compute Engine is built for dbt execution.

Language: Shell - Size: 514 KB - Last synced: 3 months ago - Pushed: 4 months ago - Stars: 1 - Forks: 0

cristian-rincon/action-composer-sync

This is a simple action that helps you to fetch your Apache Airflow DAGs to Google Cloud Composer

Language: Shell - Size: 3.91 KB - Last synced: 2 months ago - Pushed: about 2 years ago - Stars: 6 - Forks: 0

janaom/gcp-data-engineering-etl-with-composer-dataflow

This project leverages GCS, Composer, Dataflow, BigQuery, and Looker on Google Cloud Platform (GCP) to build a robust data engineering solution for processing, storing, and reporting daily transaction data in the online food delivery industry.

Language: Python - Size: 290 KB - Last synced: 6 months ago - Pushed: 6 months ago - Stars: 0 - Forks: 0

sungchun12/airflow-toolkit

Any Airflow project day 1, you can spin up a local desktop Kubernetes Airflow environment AND one in Google Cloud Composer with tested data pipelines(DAGs) :desktop_computer: >> [ :rocket:, :ship: ]

Language: HCL - Size: 12.3 MB - Last synced: 7 months ago - Pushed: 8 months ago - Stars: 104 - Forks: 30

TristanBilot/airflow-rbac-roles-cli

A tool to create Airflow RBAC roles with dag-level permissions from cli.

Language: Python - Size: 10.7 KB - Last synced: 7 months ago - Pushed: 9 months ago - Stars: 12 - Forks: 7

yuya-hanzawa/bq-cloud-composer

Practice GCP Cloud composer

Language: Python - Size: 37.1 KB - Last synced: 8 months ago - Pushed: almost 2 years ago - Stars: 0 - Forks: 0

Franklin-Muhuni/Toronto-Shelter-ETL-Pipeline

In this project, I built an end-to-end data pipeline that processes and analyzes daily occupancy and capacity data from Toronto's shelter and overnight service programs.

Language: Python - Size: 14.6 KB - Last synced: 9 months ago - Pushed: 9 months ago - Stars: 0 - Forks: 0

zaimy/trinity

A tool to synchronize workflows (DAGs) between Codebase, Cloud Storage and Airflow metadata.

Language: Go - Size: 46.9 KB - Last synced: about 1 month ago - Pushed: almost 3 years ago - Stars: 4 - Forks: 0

tosun-si/teams-league-airflow-elt

The goal of this article is showing a real world use case for ELT batch pipeline, with Cloud Storage, BigQuery, Apache Airflow and Cloud Composer : The Extract part is managed in Cloud Storage The Load part is managed from Cloud Storage to BigQuery The Transform part is managed by a BigQuery SQL query Everything is orchestrated by Airflow

Language: Python - Size: 311 KB - Last synced: 10 months ago - Pushed: 10 months ago - Stars: 0 - Forks: 0