GitHub topics: data-integration

Repositories

mariliafernandez/nutritionall

Consulta a tabelas nutricionais integradas, cálculo de macronutrientes e insulina por refeição

Language: Python - Size: 764 KB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 1 - Forks: 0

bruin-data/ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

Language: Python - Size: 167 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 2,951 - Forks: 79

SysBioChalmers/GECKO

Toolbox for including enzyme constraints on a genome-scale model.

Language: MATLAB - Size: 107 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 70 - Forks: 52

apache/seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.

Language: Java - Size: 42.4 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 8,476 - Forks: 1,964

jitsucom/jitsu

Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

Language: TypeScript - Size: 43 MB - Last synced at: 2 days ago - Pushed at: 4 days ago - Stars: 4,290 - Forks: 312

apache/flink-cdc

Flink CDC is a streaming data integration tool

Language: Java - Size: 40.9 MB - Last synced at: 1 day ago - Pushed at: 11 days ago - Stars: 6,051 - Forks: 2,010

cloudquery/plugin-sdk

CloudQuery Go SDK for source and destination plugins

Language: Go - Size: 18.2 MB - Last synced at: 2 days ago - Pushed at: 3 days ago - Stars: 25 - Forks: 25

elolab/Coralysis

A multi-level integration algorithm for sensitive integration, reference-mapping, and cell state identification in single-cell data.

Language: R - Size: 25.4 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 1 - Forks: 0

SDM-TIB/SDM-RDFizer

An Efficient RML-Compliant Engine for Knowledge Graph Construction

Language: Python - Size: 21 MB - Last synced at: 3 days ago - Pushed at: 3 days ago - Stars: 118 - Forks: 25

apache/seatunnel-web

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

Language: Java - Size: 17.4 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 677 - Forks: 302

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Language: Go - Size: 38.3 MB - Last synced at: 3 days ago - Pushed at: 4 days ago - Stars: 2,721 - Forks: 577

ConduitIO/conduit-site

Language: TypeScript - Size: 11.2 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 1 - Forks: 7

infinyon/fluvio

🦀 event stream processing for developers to stream and process data in motion to power responsive data intensive applications.

Language: Rust - Size: 34.1 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 4,916 - Forks: 514

cloudquery/cloudquery

The developer first cloud governance platform

Language: Go - Size: 172 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 6,083 - Forks: 528

morph-kgc/morph-kgc

Powerful RDF Knowledge Graph Generation with RML Mappings

Language: Python - Size: 32.8 MB - Last synced at: 3 days ago - Pushed at: 5 days ago - Stars: 209 - Forks: 39

rudderlabs/rudder-server

Privacy and Security focused Segment-alternative, in Golang and React

Language: Go - Size: 308 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 4,176 - Forks: 330

artie-labs/transfer

Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift, Databricks) in real-time.

Language: Go - Size: 3.85 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 651 - Forks: 33

mage-ai/mage-ai

🧙 Build, run, and manage data pipelines for integrating and transforming data.

Language: Python - Size: 233 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 8,300 - Forks: 841

apache/hudi

Upserts, Deletes And Incremental Processing on Big Data.

Language: Java - Size: 1.74 GB - Last synced at: 5 days ago - Pushed at: 6 days ago - Stars: 5,756 - Forks: 2,396

DSIN-INSA-Strasbourg/Hermes

Change Data Capture (CDC) tool from any source(s) to any target

Language: Python - Size: 569 KB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 3 - Forks: 1

dagster-io/dagster

An orchestration platform for the development, production, and observation of data assets.

Language: Python - Size: 1.26 GB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 13,081 - Forks: 1,668

airbytehq/airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

Language: Python - Size: 666 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 18,044 - Forks: 4,503

apache/airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Language: Python - Size: 374 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 39,930 - Forks: 14,972

Avaiga/taipy

Turns Data and AI algorithms into production-ready web applications in no time.

Language: Python - Size: 150 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 18,048 - Forks: 1,881

starlake-ai/starlake

Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.

Language: Scala - Size: 170 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 107 - Forks: 23

ConduitIO/conduit

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

Language: Go - Size: 12.6 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 490 - Forks: 51

apache/hop

Hop Orchestration Platform

Language: Java - Size: 197 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 1,130 - Forks: 367

lightfeed/lightfeed

Lightfeed SDK to search and filter web data

Language: Python - Size: 112 KB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 3 - Forks: 1

quadbio/cellmapper

k-NN-based mapping of cells across representations to transfer labels, embeddings, and expression values.

Language: Python - Size: 27.2 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 1 - Forks: 0

cyberconcepts/py-scopes

Implementation of the strange 'scopes' paradigm in Python

Language: Python - Size: 124 KB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 0 - Forks: 0

sandra-liedtke/cortex_helper

The repo contains a few scripts which might be helpful to batch-update records in Cortex XSOAR.

Language: Python - Size: 32.2 KB - Last synced at: 9 days ago - Pushed at: 9 days ago - Stars: 0 - Forks: 0

immunogenomics/harmony

Fast, sensitive and accurate integration of single-cell data with Harmony

Language: R - Size: 52.9 MB - Last synced at: 5 days ago - Pushed at: 6 months ago - Stars: 568 - Forks: 102

VIDA-NYU/bdi-viz

Language: Python - Size: 734 KB - Last synced at: 5 days ago - Pushed at: 10 days ago - Stars: 2 - Forks: 0

siyul-park/uniflow

A high-performance, extremely flexible, and easily extensible universal workflow engine.

Language: Go - Size: 2.94 MB - Last synced at: 6 days ago - Pushed at: 7 days ago - Stars: 51 - Forks: 5

Seddryck/Streamistry

Streamistry is a lightweight library designed to support pipeline, streaming, and ETL development for data engineering and integration. Its versatility makes it an excellent tool for building robust, scalable data workflows and optimizing data processing tasks. With features such as accumulators, windows, and sinks, it efficiently handles streaming

Language: C# - Size: 2.22 MB - Last synced at: 2 days ago - Pushed at: 11 days ago - Stars: 5 - Forks: 0

alankrantas/loc-documentation

End-user documentation for FST Network's Logic Operating Centre (LOC), a serverless SaaS data product platform

Language: MDX - Size: 19.6 MB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 0 - Forks: 0

BioDataFuse/pyBiodatafuse

Python package for biodatafuse project.

Language: Python - Size: 34.7 MB - Last synced at: 11 days ago - Pushed at: 11 days ago - Stars: 6 - Forks: 10

DTStack/chunjun

A data integration framework

Language: Java - Size: 126 MB - Last synced at: 12 days ago - Pushed at: 2 months ago - Stars: 4,046 - Forks: 1,699

kopeckylukas/py-mamsi

The Multi-assay Mass Spectrometry Integration Project

Language: Python - Size: 1.58 MB - Last synced at: 13 days ago - Pushed at: 13 days ago - Stars: 4 - Forks: 0

CogStack/CogStack-NiFi

Building data processing pipelines for documents processing with NLP using Apache NiFi and related services

Language: Python - Size: 74.4 MB - Last synced at: 15 days ago - Pushed at: 15 days ago - Stars: 50 - Forks: 20

seandavi/awesome-single-cell

Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.

Size: 1.43 MB - Last synced at: 11 days ago - Pushed at: 19 days ago - Stars: 3,358 - Forks: 1,016

gabledata/recap

Work with your web service, database, and streaming schemas in a single format.

Language: Python - Size: 1.42 MB - Last synced at: 13 days ago - Pushed at: 16 days ago - Stars: 344 - Forks: 26

chofste/ETL

Language: Python - Size: 2.54 MB - Last synced at: 5 days ago - Pushed at: 5 days ago - Stars: 4 - Forks: 0

CloudFormations/CF.Cumulus

A cloud data platform product to accelerate time to insights. Our open-source framework is designed for the real world. Stripping away the complexity, giving you the power to build, scale, and manage your dataflows with ease, accelerating data delivery.

Language: TSQL - Size: 10.5 MB - Last synced at: 1 day ago - Pushed at: 1 day ago - Stars: 19 - Forks: 11

ronpinkas/dbBridge

dbBridge is an 'SQL Migration Tool' - enabling import of SQL Databases from any supported Dialect (MsSql, MySql, Oracle, PostgreSQL, Sqlite) to any of these supported dialects with just three lines of PHP code.

Language: PHP - Size: 81.1 KB - Last synced at: 11 days ago - Pushed at: over 1 year ago - Stars: 6 - Forks: 1

brooks-code/fuzzy-carnival

dailyword: discover a new word, once a day, straight from your terminal.

Language: Shell - Size: 1000 Bytes - Last synced at: 19 days ago - Pushed at: 19 days ago - Stars: 0 - Forks: 0

krisrs1128/multimedia

multimedia is an R package for multimodal mediation analysis of microbiome data. It has been designed to help integration of relative abundance, survey, and metabolomic data through causal mediation analysis.

Language: R - Size: 23.6 MB - Last synced at: 19 days ago - Pushed at: 19 days ago - Stars: 2 - Forks: 0

aelmah/Data-integration

Multilingual Bibliographic Data Integration for PMB : Converting French and Arabic library records into PMB-ready data using PHP, ensuring consistency and multilingual support.

Language: Jupyter Notebook - Size: 3.37 MB - Last synced at: 13 days ago - Pushed at: 19 days ago - Stars: 0 - Forks: 0

opensanctions/nomenklatura

Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources

Language: Python - Size: 2.54 MB - Last synced at: 16 days ago - Pushed at: 17 days ago - Stars: 209 - Forks: 38

saeyslab/nichenetr

NicheNet: predict active ligand-target links between interacting cells

Language: R - Size: 152 MB - Last synced at: 16 days ago - Pushed at: 17 days ago - Stars: 534 - Forks: 124

leesf/hudi-resources

汇总Apache Hudi相关资料

Size: 23.7 MB - Last synced at: 20 days ago - Pushed at: 20 days ago - Stars: 550 - Forks: 160

VIDA-NYU/bdi-kit

A Python toolkit for biomedical data integration and harmonization

Language: Python - Size: 45.3 MB - Last synced at: 19 days ago - Pushed at: 19 days ago - Stars: 10 - Forks: 3

DerwenAI/ERKG

Demonstrate integration of Senzing and Neo4j to construct an Entity Resolved Knowledge Graph

Size: 13.9 MB - Last synced at: 8 days ago - Pushed at: 9 months ago - Stars: 32 - Forks: 6

linkml/linkml-model

Link Modeling Language (LinkML) model

Language: Python - Size: 12.7 MB - Last synced at: 22 days ago - Pushed at: 23 days ago - Stars: 48 - Forks: 20

CommonCoreOntology/CommonCoreOntologies

The Common Core Ontology Repository holds the current released version of the Common Core Ontology suite.

Language: Makefile - Size: 16.5 MB - Last synced at: 19 days ago - Pushed at: 19 days ago - Stars: 230 - Forks: 61

otrecoding/OptimalTransportDataIntegration.jl

Data integration using optimal transport theory

Language: Julia - Size: 504 KB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 0 - Forks: 0

neuroforgede/nfcompose

Build REST APIs/Integrations in minutes instead of hours - NF Compose is a (data) integration platform that allows developers to define REST APIs in seconds instead of hours. Generated REST APIs are backed by postgres and support automatic consumer webhook notifications on data changes out of the box.

Language: Python - Size: 2.57 MB - Last synced at: 7 days ago - Pushed at: 23 days ago - Stars: 39 - Forks: 3

HSU-HPC/BERT

Batch-Effect Reduction Trees

Language: R - Size: 162 KB - Last synced at: 14 days ago - Pushed at: 3 months ago - Stars: 3 - Forks: 0

zazuko/barnard59

An intuitive and flexible RDF pipeline solution designed to simplify and automate ETL processes for efficient data management.

Language: JavaScript - Size: 3.66 MB - Last synced at: about 18 hours ago - Pushed at: about 1 month ago - Stars: 30 - Forks: 2

runprism/prism

Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.

Language: Python - Size: 2.42 MB - Last synced at: 14 days ago - Pushed at: 6 months ago - Stars: 85 - Forks: 2

NPLinker/nplinker

A python framework for microbial natural products data mining by integrating genomics and metabolomics data

Language: Python - Size: 116 MB - Last synced at: 19 days ago - Pushed at: 26 days ago - Stars: 18 - Forks: 13

saezlab/cosmosR

COSMOS (Causal Oriented Search of Multi-Omic Space) is a method that integrates phosphoproteomics, transcriptomics, and metabolomics data sets.

Language: R - Size: 53.2 MB - Last synced at: 22 days ago - Pushed at: 2 months ago - Stars: 60 - Forks: 16

hetio/hetionet

Hetionet: an integrative network of disease

Language: HTML - Size: 380 MB - Last synced at: 27 days ago - Pushed at: about 2 years ago - Stars: 282 - Forks: 69

mara/mara-pipelines

A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

Language: Python - Size: 3.29 MB - Last synced at: 27 days ago - Pushed at: over 1 year ago - Stars: 2,078 - Forks: 100

armbian/armbian.github.io

Armbian Linux artefacts and data exchange website

Language: Python - Size: 2.73 GB - Last synced at: 6 days ago - Pushed at: 6 days ago - Stars: 3 - Forks: 1

farmacyst99/data-integration

Language: Python - Size: 21.5 KB - Last synced at: 28 days ago - Pushed at: 28 days ago - Stars: 0 - Forks: 0

45592858/DataV5

Simplifying Database Migrations, Data Initialization, and Data Management with Full Control for Developers.

Language: TypeScript - Size: 21.5 KB - Last synced at: 29 days ago - Pushed at: 30 days ago - Stars: 1 - Forks: 0

machuangtao/KG-RAG4SM

Knowledge Graph-based Retrieval-Augmented Generation for Schema Matching

Language: Python - Size: 5.55 MB - Last synced at: 8 days ago - Pushed at: 8 days ago - Stars: 4 - Forks: 1

bytedance/bitsail

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

Language: Java - Size: 26.4 MB - Last synced at: 26 days ago - Pushed at: over 1 year ago - Stars: 1,654 - Forks: 331

glasgowcompbio/pyMultiOmics

Python toolbox for multi-omics data mapping and analysis

Language: Jupyter Notebook - Size: 45.9 MB - Last synced at: 21 days ago - Pushed at: about 2 years ago - Stars: 24 - Forks: 5

VIDA-NYU/magneto-matcher

Repository for developing and evaluating components and algorithms for data integration tasks

Language: Jupyter Notebook - Size: 40 MB - Last synced at: 12 days ago - Pushed at: 12 days ago - Stars: 4 - Forks: 0

cutterkom/remove-na-lgbtiq-queer-knowledge-graph

A knowledge graph on queer history

Language: R - Size: 9.45 MB - Last synced at: 13 days ago - Pushed at: 5 months ago - Stars: 16 - Forks: 1

boxalino/data-integration-magento2

Data Integration Layer for Magento2 sync

Language: PHP - Size: 474 KB - Last synced at: 4 days ago - Pushed at: about 1 month ago - Stars: 1 - Forks: 1

google/megalista 📦

First Party data integration solution built for marketing teams to enable audience and conversion onboarding into Google Marketing products (Google Ads, Campaign Manager, Google Analytics).

Language: Python - Size: 1.34 MB - Last synced at: 21 days ago - Pushed at: 4 months ago - Stars: 137 - Forks: 55

dieterich-lab/Baltica

Baltica: integrated differential junction usage

Language: R - Size: 57.6 MB - Last synced at: 27 days ago - Pushed at: about 2 years ago - Stars: 10 - Forks: 3

mara/mara-etl-tools

Utilities for creating ETL pipelines with mara

Language: PLpgSQL - Size: 54.7 KB - Last synced at: about 3 hours ago - Pushed at: almost 3 years ago - Stars: 36 - Forks: 4

buildersoftio/cortex

Cortex | Data Framework—a cutting-edge SDK that simplifies real-time data processing with intuitive operators, robust state management, and seamless telemetry for efficient, scalable pipelines.

Language: C# - Size: 453 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 10 - Forks: 2

yxwucq/scExtract

Automatic annotation of single-cell RNA-seq data from the literature

Language: Python - Size: 20.5 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 5 - Forks: 0

HrishiDhondge/CroMaSt

CroMaSt (Cross Mapper of domain Structural instances) is an automated iterative workflow to clarify domain definition by cross-mapping of domain structural instances between domain databases.

Language: Common Workflow Language - Size: 918 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 1 - Forks: 0

gbazad93/AirFlow-ML-Data-Integration

An Airflow-based pipeline that fetches data from a free API, cleans and transforms it, and saves it to a database—ready for downstream machine learning.

Language: Python - Size: 857 KB - Last synced at: 4 days ago - Pushed at: about 2 months ago - Stars: 11 - Forks: 1

ceumicrodata/mETL

mito ETL tool

Language: Python - Size: 7.43 MB - Last synced at: 3 days ago - Pushed at: almost 4 years ago - Stars: 163 - Forks: 41

selbouhaddani/OmicsPLS

R package for High dimensional data analysis and integration with O2PLS!

Language: HTML - Size: 31.4 MB - Last synced at: 28 days ago - Pushed at: about 1 month ago - Stars: 32 - Forks: 8

altova/altova-server-docker

This project provides an easy way to install and run the various components of the Altova Server Platform in a Docker environment. It includes configuration files and instructions to set up the server products with minimal effort, leveraging Docker Compose.

Language: Shell - Size: 13.7 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 1 - Forks: 1

dataplane-app/dataplane

Dataplane is an Airflow inspired unified data platform with additional data mesh and RPA capability to automate, schedule and design data pipelines and workflows. Dataplane is written in Golang with a React front end.

Language: JavaScript - Size: 281 MB - Last synced at: 9 days ago - Pushed at: about 1 month ago - Stars: 226 - Forks: 33

kuwala-io/kuwala

Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demographics data b) Point of Interests from Open Street Map c) Google Popular Times

Language: JavaScript - Size: 7.79 MB - Last synced at: about 1 month ago - Pushed at: over 2 years ago - Stars: 792 - Forks: 54