GitHub | Ecosyste.ms: Repos

stellar/stellar-etl-airflow

Airflow DAGs for the Stellar ETL project

Language: Python - Size: 2.97 MB - Last synced: 17 minutes ago - Pushed: 32 minutes ago - Stars: 31 - Forks: 14

fioremartiro/PortfolioProjectsData

Professional portfolio of data science and analytics projects - 2024.

Language: TSQL - Size: 0 Bytes - Last synced: about 17 hours ago - Pushed: about 18 hours ago - Stars: 0 - Forks: 0

harish876/forge

Opiniated Framework to write ETL Pipelines which controlled by a central config store.

Language: Python - Size: 36.1 KB - Last synced: about 20 hours ago - Pushed: about 21 hours ago - Stars: 0 - Forks: 0

dataforgelabs/dataforge-core

DataForge helps data teams write functional transformation pipelines by leveraging software engineering principles

Language: PLpgSQL - Size: 277 KB - Last synced: about 21 hours ago - Pushed: 2 days ago - Stars: 18 - Forks: 0

globalbioticinteractions/globalbioticinteractions

Global Biotic Interactions provides access to existing species interaction datasets

Language: Java - Size: 150 MB - Last synced: 3 days ago - Pushed: 17 days ago - Stars: 118 - Forks: 17

TriplyDB/Documentation

Documentation for the TriplyDB and TriplyETL products

Language: HTML - Size: 11.8 MB - Last synced: 2 days ago - Pushed: 2 days ago - Stars: 6 - Forks: 4

data-solution-automation-engine/DIRECT

DIRECT, the Data Integration Run-time Execution Control Tool, is a data logistics control framework that can be used to monitor, log, audit and control data integration / ETL processes.

Language: TSQL - Size: 12.9 MB - Last synced: 1 day ago - Pushed: 2 days ago - Stars: 25 - Forks: 8

halestudio/hale

(Spatial) data harmonisation with hale studio (formerly HUMBOLDT Alignment Editor)

Language: Java - Size: 348 MB - Last synced: 3 days ago - Pushed: 3 days ago - Stars: 138 - Forks: 44

cloudquery/cloudquery

The open source high performance ELT framework powered by Apache Arrow

Language: Go - Size: 145 MB - Last synced: 23 days ago - Pushed: 23 days ago - Stars: 5,575 - Forks: 484

apache/seatunnel-web

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

Language: Java - Size: 16.8 MB - Last synced: about 16 hours ago - Pushed: 2 days ago - Stars: 431 - Forks: 196

Cinchoo/ChoETL

ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

Language: C# - Size: 39.6 MB - Last synced: 4 days ago - Pushed: 15 days ago - Stars: 738 - Forks: 132

usc-isi-i2/kgtk

Knowledge Graph Toolkit

Language: Jupyter Notebook - Size: 105 MB - Last synced: 6 days ago - Pushed: 7 months ago - Stars: 341 - Forks: 56

sparsecode/DaFlow

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.

Language: Scala - Size: 490 KB - Last synced: 9 days ago - Pushed: almost 3 years ago - Stars: 26 - Forks: 13

nh43de/DataPowerTools

Bridging the gap between IEnumerable and IDataReader for dealing with unstructured and loosely-structured data, plus fast ETL + SQL Bulk Copy.

Language: C# - Size: 11.1 MB - Last synced: 8 days ago - Pushed: 9 days ago - Stars: 8 - Forks: 2

kyaiooiayk/ETL-and-ML-Pipelines-Notes

Notes, tutorials, code snippets and templates focused on ETL pipelines for Machine Learning

Language: Jupyter Notebook - Size: 3.71 MB - Last synced: 11 days ago - Pushed: 11 days ago - Stars: 0 - Forks: 0

walleXD/ts-dag

Collection of pkgs to build pipelines in JS/TS

Language: TypeScript - Size: 186 KB - Last synced: 4 days ago - Pushed: 5 days ago - Stars: 0 - Forks: 0

geopython/stetl

Stetl, Streaming ETL, is a lightweight geospatial processing and ETL framework written in Python.

Language: Python - Size: 6.95 MB - Last synced: 9 days ago - Pushed: 13 days ago - Stars: 82 - Forks: 34

elastic/logstash

Logstash - transport and process your logs, events, or other data

Language: Java - Size: 121 MB - Last synced: 17 days ago - Pushed: 17 days ago - Stars: 14,000 - Forks: 3,467

DAGWorks-Inc/hamilton

Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.

Language: Jupyter Notebook - Size: 49.9 MB - Last synced: 17 days ago - Pushed: 18 days ago - Stars: 1,312 - Forks: 80

cefriel/chimera

Composable Semantic Transformation Pipelines

Language: Java - Size: 6.26 MB - Last synced: 17 days ago - Pushed: 17 days ago - Stars: 24 - Forks: 2

bizrockman/AI-Enhanced-DataHive

AI Enhanced DataHive embarks on a mission to become a centralized hub for data of various kinds, offering templates for collectors to aggregate data centrally for further processing in other applications. This initiative arises from the repeated cycles of developing crawlers, extractors, and collectors across numerous projects.

Language: Python - Size: 126 KB - Last synced: 18 days ago - Pushed: 18 days ago - Stars: 0 - Forks: 0

data-dot-all/dataall

A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.

Language: Python - Size: 80.3 MB - Last synced: 21 days ago - Pushed: 21 days ago - Stars: 209 - Forks: 75

duoan/OpenKettleWebUI

一款基于kettle的数据处理web调度控制平台，支持文档资源库和数据库资源库，通过web平台控制kettle数据转换，可作为中间件集成到现有系统中

Language: Java - Size: 110 MB - Last synced: 16 days ago - Pushed: about 7 years ago - Stars: 145 - Forks: 98

singer-io/getting-started

This repository is a getting started guide to Singer.

Language: Makefile - Size: 197 KB - Last synced: 19 days ago - Pushed: about 1 year ago - Stars: 1,226 - Forks: 145

PastorGL/datacooker-etl

Data transformation framework for ETL processing with SQL-like syntax and GIS extensions, based on Apache Spark

Language: Java - Size: 10.4 MB - Last synced: 28 days ago - Pushed: 28 days ago - Stars: 11 - Forks: 0

quintoandar/butterfree

A tool for building feature stores.

Language: Python - Size: 1.01 MB - Last synced: 29 days ago - Pushed: 29 days ago - Stars: 268 - Forks: 35

stellar/stellar-etl

Stellar ETL will enable real-time analytics on the Stellar network

Language: Go - Size: 2.13 MB - Last synced: 27 days ago - Pushed: 28 days ago - Stars: 28 - Forks: 12

frankframework/frankframework

The Frank!Framework is an easy-to-use, stateless integration framework which allows (transactional) messages to be modified and exchanged between different systems.

Language: Java - Size: 97.2 MB - Last synced: 26 days ago - Pushed: 27 days ago - Stars: 110 - Forks: 76

marsupialtail/quokka

Making data lake work for time series

Language: Python - Size: 38.5 MB - Last synced: 24 days ago - Pushed: 7 months ago - Stars: 1,081 - Forks: 57

redis-field-engineering/redis-connect-dist

Real-Time Event Streaming & Change Data Capture

Language: Shell - Size: 37.2 MB - Last synced: 7 days ago - Pushed: 8 days ago - Stars: 40 - Forks: 11

noflo/noflo

Flow-based programming for JavaScript

Language: JavaScript - Size: 3.07 MB - Last synced: 26 days ago - Pushed: about 1 month ago - Stars: 3,466 - Forks: 258

bailuWX/big-data-etl

etl日志收集分析工具

Language: Java - Size: 138 KB - Last synced: 26 days ago - Pushed: 27 days ago - Stars: 0 - Forks: 0

stitchfix/hamilton 📦

A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN MOVED TO www.github.com/dagworks-inc/hamilton

Language: Python - Size: 7.72 MB - Last synced: 20 days ago - Pushed: 10 months ago - Stars: 874 - Forks: 39

Nextdoor/bender 📦

Bender - Serverless ETL Framework

Language: Java - Size: 1.48 MB - Last synced: 26 days ago - Pushed: 5 months ago - Stars: 184 - Forks: 21

apache/seatunnel-datasource-sdk

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

Language: Java - Size: 54.7 KB - Last synced: about 15 hours ago - Pushed: 11 months ago - Stars: 6 - Forks: 4

KenSuenobu/objectified

Open Source Next Generation Time-Series Database, built on object storage

Language: TypeScript - Size: 234 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 1 - Forks: 0

giacomolat/The-Future-of-Sustainability-and-Training

The future of sustainability and training: involvement and performance of companies and strategic suppliers

Language: SQL - Size: 30 MB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

vim89/datapipelines-essentials-python

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Language: Python - Size: 1.76 MB - Last synced: 24 days ago - Pushed: about 1 year ago - Stars: 53 - Forks: 34

YotpoLtd/metorikku

A simplified, lightweight ETL Framework based on Apache Spark

Language: Scala - Size: 4.2 MB - Last synced: 25 days ago - Pushed: 4 months ago - Stars: 576 - Forks: 150

nhl/link-move

A model-driven dynamically-configurable framework to acquire data from external sources and save it to your database.

Language: Java - Size: 1.74 MB - Last synced: 3 days ago - Pushed: 3 months ago - Stars: 34 - Forks: 15

speedbits/LimitlessETL

A Python and Spark based ETL framework. While it operates within speed limits that is framework and standards, but offers boundless possibilities.

Language: Python - Size: 15.4 MB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

mikeAdamss/tidychef

Python framework for transforming tabulated data with visual relationships into tidy data

Language: Jupyter Notebook - Size: 16.3 MB - Last synced: 11 days ago - Pushed: 10 months ago - Stars: 1 - Forks: 0

BitwiseInc/Hydrograph

A visual ETL development and debugging tool for big data

Language: Java - Size: 33.5 MB - Last synced: about 1 month ago - Pushed: over 1 year ago - Stars: 152 - Forks: 109

piyushsingariya/Shift

Shift is a high performance better alternative to Airbyte, Singer, Meltano

Language: Go - Size: 13.7 MB - Last synced: about 1 month ago - Pushed: 7 months ago - Stars: 8 - Forks: 3

OpenChaos/ogi Fork of gojekfarm/ogi

utility to enable flexible ETL scenarios, supports golang plug-in for built-in consumer|transformer|producer options

Language: Go - Size: 164 KB - Last synced: about 2 months ago - Pushed: over 3 years ago - Stars: 0 - Forks: 0

Renien/ETL-Starter-Kit

:file_folder: Extract, Transform, Load (ETL) :construction_worker: refers to a process in database usage and especially in data warehousing. This repository contains a starter kit featuring ETL related work.

Language: Scala - Size: 1.18 MB - Last synced: about 2 months ago - Pushed: about 7 years ago - Stars: 21 - Forks: 9

flow-php/flow

Flow PHP - strongly typed data processing framework

Language: PHP - Size: 19.3 MB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 279 - Forks: 22

JohnnyBravo75/DataBridge.NET

Configurable data bridge for permanent ETL jobs

Language: C# - Size: 11.1 MB - Last synced: about 2 months ago - Pushed: over 1 year ago - Stars: 20 - Forks: 10

usc-isi-i2/dig-etl-engine

Download DIG to run on your laptop or server.

Size: 7.51 MB - Last synced: about 2 months ago - Pushed: over 5 years ago - Stars: 99 - Forks: 39

seanharr11/etlalchemy

Extract, Transform, Load: Any SQL Database in 4 lines of Code.

Language: Python - Size: 20.8 MB - Last synced: 22 days ago - Pushed: almost 5 years ago - Stars: 553 - Forks: 83

giucris/yasp

Yet Another SPark Framework

Language: Scala - Size: 228 KB - Last synced: 19 days ago - Pushed: over 1 year ago - Stars: 10 - Forks: 1

ceumicrodata/mETL

mito ETL tool

Language: Python - Size: 7.43 MB - Last synced: 17 days ago - Pushed: almost 3 years ago - Stars: 164 - Forks: 40

san089/goodreads_etl_pipeline

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Language: Python - Size: 1.31 MB - Last synced: 2 months ago - Pushed: about 4 years ago - Stars: 1,209 - Forks: 205

prithvijitguha/flowrunner

Flowrunner is a lightweight package to organize and represent Data Engineering/Science workflows

Language: Python - Size: 312 KB - Last synced: 10 days ago - Pushed: 11 months ago - Stars: 8 - Forks: 0

dalenewman/Transformalize

Configurable Extract, Transform, and Load

Language: C# - Size: 170 MB - Last synced: 2 days ago - Pushed: 3 months ago - Stars: 155 - Forks: 33

speedbits/SpeedFluxETL

A Scala and Spark based ETL framework. The name SpeedFlux implies movement of data and the agility with which it flows.

Size: 6.84 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

ywan3223/Analysis-of-Customer-Orders

Analysis of customer orders in retail application from Oracle database

Language: TSQL - Size: 10.7 MB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

bala-1409/SQL-Projects

The repository contains Structured Query Language (SQL) Scripts. The Multiple SQL scripts for various projects which includes data cleaning, data pre-processing, data processing, data transformation and insights gaining through Query Language.

Language: TSQL - Size: 723 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

vh-d/Rflow

Rflow is a general-purpose workflow management framework for R

Language: R - Size: 568 KB - Last synced: about 2 months ago - Pushed: about 2 years ago - Stars: 4 - Forks: 1

cpzt/parade-manage

A manage module for parade

Language: Python - Size: 59.6 KB - Last synced: 4 days ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

socialpoint-labs/sqlbucket 📦

Lightweight library to write, orchestrate and test your SQL ETL. Writing ETL with data integrity in mind.

Language: Python - Size: 463 KB - Last synced: 8 days ago - Pushed: 4 months ago - Stars: 71 - Forks: 7

aaduhalde/Data_integration_clean_code

Size: 1.95 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

patterns-app/patterns-devkit

Data pipelines from re-usable components

Language: Python - Size: 1.75 MB - Last synced: 28 days ago - Pushed: about 1 year ago - Stars: 106 - Forks: 5

pratiksha2712/ETL-Pipeline

An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform.

Language: Python - Size: 1.47 MB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

SharpData/SharpETL

Write ETL using your favorite SQL dialects

Language: Scala - Size: 3.37 MB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 36 - Forks: 5

harry-s-grewal/mls-real-estate-scraper-for-realtor.ca

Python MLS and Real-Estate Data Scraper for the Realtor.ca Website

Language: Python - Size: 10.7 KB - Last synced: 3 months ago - Pushed: 8 months ago - Stars: 29 - Forks: 10

fatih-ml/python-sql-etl-process

Explore streamlined Python-SQL ETL workflows in this repository. 🚀 Learn efficient data extraction, transformation, and loading techniques using Pandas and SQL. 🐍 Ideal for mastering ETL processes in diverse contexts. 🌐💡

Language: Jupyter Notebook - Size: 974 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

extralo/loom

Weaving together different threads (services like image/audio converse, ETL services, etc.) to enable the World Wide Flow

Language: JavaScript - Size: 11.7 MB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 0 - Forks: 0

restarone/violet_rails

an app engine for your business. Seamlessly implement business logic with a powerful API. Out of the box CMS, blog, forum and email functionality. Developer friendly & easily extendable for your next SaaS/XaaS project. Built with Rails 6, Devise, Sidekiq & PostgreSQL

Language: Ruby - Size: 32.7 MB - Last synced: 3 months ago - Pushed: 7 months ago - Stars: 95 - Forks: 42

rtimbro185/syr_mads_ist722_data_warehouse

Syracuse University, Masters of Applied Data Science - IST 722 Data Warehouse

Language: TSQL - Size: 50.6 MB - Last synced: 5 months ago - Pushed: about 4 years ago - Stars: 3 - Forks: 4

hritiksauw199/Visualizing-Real-Time-Covid-Data-using-SQL

Visualized UK's Covid Data by implementing an ETL framework using sqlite3, python and seaborn.

Language: Jupyter Notebook - Size: 17 MB - Last synced: 5 months ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

FizzcodeSoftware/EtLast

ETL (Extract, Transform and load) library for .Net

Language: C# - Size: 4.65 MB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 7 - Forks: 5

FernandoCL1991/etl-analysis2

Project uses Pandas to create multiple DataFrames from CSV files containing social media data on presidential candidates, cleaned those DataFrames, then used SQL to create a relational database to join everything together.

Language: Rich Text Format - Size: 1.24 MB - Last synced: 5 months ago - Pushed: almost 3 years ago - Stars: 0 - Forks: 0

jpvanegasc/entropic

A simple data processing framework for a quick, no-frills setup of a local data pipeline.

Language: Python - Size: 102 KB - Last synced: about 1 month ago - Pushed: 5 months ago - Stars: 0 - Forks: 0

JamesHanZhang/table-data-format-transform-app

excel, markdown, csv, sql 数据源批量/单独格式互相转换

Language: Python - Size: 321 MB - Last synced: 6 months ago - Pushed: 6 months ago - Stars: 1 - Forks: 0

ebrard/talend-examples

Some technical implementations of data integration with Talend (e.g. bridge table, incremental loads)

Size: 269 KB - Last synced: 6 months ago - Pushed: over 6 years ago - Stars: 2 - Forks: 1

FredHutch/HDC-DataEngineering-SQLSync

Table synchronization utility for MS-SQL

Language: PLpgSQL - Size: 57.6 KB - Last synced: 24 days ago - Pushed: over 4 years ago - Stars: 5 - Forks: 1

StrictlySkyler/harbormaster Fork of luzlab/harbormaster-apache 📦

A framework for microservices

Language: JavaScript - Size: 1.82 MB - Last synced: 26 days ago - Pushed: 6 months ago - Stars: 3 - Forks: 5

Chavis00/csv-db-loader

Python package that enables customized loading of data from a CSV file into a MySQL database

Language: Python - Size: 13.7 KB - Last synced: 7 months ago - Pushed: 7 months ago - Stars: 0 - Forks: 0

SanjinKurelic/AntennaDistribution

Antenna Distribution is a project that shows how to run business analysis tools on a set of a data.

Language: TSQL - Size: 23.5 MB - Last synced: 7 months ago - Pushed: about 2 years ago - Stars: 0 - Forks: 0

MassStreetAnalytics/etl-framework

A framework for moving data into a data warehouse.

Language: Jupyter Notebook - Size: 17.2 MB - Last synced: about 1 month ago - Pushed: over 2 years ago - Stars: 51 - Forks: 21

davinci-std/inbulk

Declarative ETL Framework

Language: Python - Size: 35.2 KB - Last synced: 6 months ago - Pushed: over 2 years ago - Stars: 5 - Forks: 0

shridhar1504/SQL-Projects

The repository contains Structured Query Language (SQL) Scripts. The Multiple SQL scripts for various projects which includes data cleaning, data pre-processing, data processing, data transformation and insights gaining through Query Language.

Language: TSQL - Size: 687 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

2298-Software/Mambo

A simple in-memory, configuration driven, data processing pipeline for Apache Spark.

Language: Scala - Size: 103 KB - Last synced: 7 months ago - Pushed: over 1 year ago - Stars: 5 - Forks: 3

smmiri/etl-visuals

Codes for data flow between models, data post-process, and visualization

Language: Jupyter Notebook - Size: 3.81 MB - Last synced: 9 months ago - Pushed: 9 months ago - Stars: 0 - Forks: 0

etlbox/etlbox 📦

A lightweight ETL (extract, transform, load) library and data integration toolbox for .NET.

Language: C# - Size: 15.5 MB - Last synced: 7 months ago - Pushed: almost 2 years ago - Stars: 233 - Forks: 75

spoke-data/spoke

The universal data connector

Language: Svelte - Size: 3.94 MB - Last synced: 9 months ago - Pushed: 9 months ago - Stars: 3 - Forks: 0

The-Tech-Idea/Beep.winform

Application for Managing your Different DataSources . Still in Alpha.please be patient

Language: PowerShell - Size: 200 MB - Last synced: 14 days ago - Pushed: 6 months ago - Stars: 0 - Forks: 0

gabriellouzada/dnc

Desafios concluídos na Formação em Dados pela escola Dinâmica (DNC). | Challenges done in the Data Training Course at Dinâmica school (DNC).

Language: Jupyter Notebook - Size: 11.6 MB - Last synced: 9 months ago - Pushed: 9 months ago - Stars: 0 - Forks: 0

ldaniels528/transgress

A distributed processing/orchestration server and ETL for NodeJS

Language: Scala - Size: 685 KB - Last synced: 9 months ago - Pushed: about 7 years ago - Stars: 1 - Forks: 0

kklimexk/spark-playground

Repository for playing with spark

Language: Scala - Size: 778 KB - Last synced: 9 months ago - Pushed: over 3 years ago - Stars: 0 - Forks: 0

geniusfox/etl_with_ruby

基于ActiveRecord构建的简单ETL框架。

Language: Ruby - Size: 19.5 KB - Last synced: 9 months ago - Pushed: over 2 years ago - Stars: 4 - Forks: 0

eSolutionsGrup/datatweak

Data Tweak is a simplified, lightweight ETL framework based on Apache Spark.

Language: Scala - Size: 132 KB - Last synced: 9 months ago - Pushed: over 3 years ago - Stars: 0 - Forks: 0

hotienvu/spipe

spark library to construct ETL pipeline with monads

Language: Scala - Size: 3.91 KB - Last synced: 9 months ago - Pushed: about 1 year ago - Stars: 1 - Forks: 0

tharwaninitin/etlflow

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases, SFTP servers, On-Prem Systems and more.

Language: Scala - Size: 13.4 MB - Last synced: 10 months ago - Pushed: 10 months ago - Stars: 42 - Forks: 12

ikama-innovation/Bauta

Bauta is a batch automation framework based on Spring Batch and Spring Boot. It adds a user interface in which you can start/stop/restart jobs, monitor step/job status. Also provides some add-ons to jobs and steps, such as report/log generation, job parameter handling etc.

Language: Java - Size: 3.88 MB - Last synced: 6 months ago - Pushed: about 1 year ago - Stars: 4 - Forks: 2

GitHub topics: etl-framework