m3dev/gokart

Gokart solves reproducibility, task dependencies, constraints of good code, and ease of use for Machine Learning Pipeline.

Language: Python - Size: 996 KB - Last synced: about 17 hours ago - Pushed: about 18 hours ago - Stars: 299 - Forks: 56

spotify/luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

Language: Python - Size: 10.3 MB - Last synced: 3 days ago - Pushed: 3 days ago - Stars: 17,356 - Forks: 2,373

Ionathan-Hope/Ionathan-Hope

Luigi.xml

Language: Assembly - Size: 4.89 MB - Last synced: 4 days ago - Pushed: 5 days ago - Stars: 0 - Forks: 0

BlueBrain/luigi-tools

Extra tools to work with the Luigi library

Language: Python - Size: 242 KB - Last synced: 4 days ago - Pushed: 5 days ago - Stars: 7 - Forks: 1

rickyringler/AffirME

Demonstrating Flask development in Python, HTML, and CSS.

Language: Python - Size: 43.9 KB - Last synced: 9 days ago - Pushed: 9 days ago - Stars: 1 - Forks: 0

holdenk/luigi-rewrite-rug

WIP Proof of Concept Luigi Pipeline Rewriting for testing using rug

Language: TypeScript - Size: 13.7 KB - Last synced: 10 days ago - Pushed: about 7 years ago - Stars: 0 - Forks: 0

WilliamQLiu/improved-parakeet

Python Data ETL with scrapy, luigi, and pandas

Language: Python - Size: 295 KB - Last synced: 26 days ago - Pushed: over 1 year ago - Stars: 2 - Forks: 0

zhanpon/luigi-examples

Luigi example workflows

Language: Python - Size: 19.5 KB - Last synced: about 1 month ago - Pushed: 9 months ago - Stars: 0 - Forks: 0

optimizely/kubeluigi

Run Luigi tasks as Jobs on K8s - An updated Kubernetes contrib

Language: Python - Size: 94.7 KB - Last synced: 6 days ago - Pushed: 7 months ago - Stars: 8 - Forks: 4

timkpaine/paperboy

A web frontend for scheduling Jupyter notebook reports

Language: Python - Size: 12.5 MB - Last synced: 1 day ago - Pushed: over 2 years ago - Stars: 248 - Forks: 26

maxhollmann/voxceleb-luigi

Luigi pipeline to download VoxCeleb(2) audio from YouTube and extract speaker segments

Language: Python - Size: 80.1 KB - Last synced: 4 days ago - Pushed: about 3 years ago - Stars: 41 - Forks: 4

pharmbio/sciluigi

A light-weight wrapper library around Spotify's Luigi workflow library to make writing scientific workflows more fluent, flexible and modular

Language: Python - Size: 221 KB - Last synced: 21 days ago - Pushed: over 1 year ago - Stars: 331 - Forks: 57

neroist/luigi

Nim bindings for Luigi

Language: C - Size: 480 KB - Last synced: about 1 month ago - Pushed: 9 months ago - Stars: 7 - Forks: 1

mdev-web/testbomi4

Language: TypeScript - Size: 27.3 KB - Last synced: 3 months ago - Pushed: 3 months ago - Stars: 0 - Forks: 0

adilkhash/luigi-telegram

Luigi Tasks status notifications to Telegram

Language: Python - Size: 3.91 KB - Last synced: 3 months ago - Pushed: almost 6 years ago - Stars: 3 - Forks: 0

miku/gluish

Utils around luigi.

Language: Python - Size: 313 KB - Last synced: 5 days ago - Pushed: about 3 years ago - Stars: 65 - Forks: 7

common-citizen-tech/common-data-usage-pipelines

Common Data Usage is a tool for graph analysis of Open Government Data Usage in open-source projects

Language: Python - Size: 586 KB - Last synced: 4 months ago - Pushed: 4 months ago - Stars: 0 - Forks: 0

unnati-xyz/scalable-data-science-platform

Content for architecting a data science platform for products using Luigi, Spark & Flask.

Language: Jupyter Notebook - Size: 22.7 MB - Last synced: 24 days ago - Pushed: over 4 years ago - Stars: 162 - Forks: 29

Xzonn/LuigiMansion

《路易吉洋馆》汉化

Language: Python - Size: 5.18 MB - Last synced: about 2 months ago - Pushed: about 2 months ago - Stars: 21 - Forks: 2

SCIFER99/Super-Mario-Bros-Game

This is a backend development attempt to recreate the 1985 Level 1 of Super Mario Bros (Version 1) !!!

Language: Python - Size: 4.84 MB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 1 - Forks: 0

groupon/luigi-warehouse

A luigi powered analytics / warehouse stack

Language: Python - Size: 62.5 KB - Last synced: 27 days ago - Pushed: about 7 years ago - Stars: 87 - Forks: 18

thehyve/pmc-conversion 📦

Data extraction, transformation and loading pipeline for the Princess Maxima Center

Language: Python - Size: 12.9 MB - Last synced: about 1 month ago - Pushed: 6 months ago - Stars: 2 - Forks: 0

miseyu/luigi-etl

luigi app for own

Language: Python - Size: 17.6 KB - Last synced: 5 months ago - Pushed: over 7 years ago - Stars: 5 - Forks: 0

slub/efre-lod-elasticsearch-tools 📦

collection of python scripts for a toolchain to feed an ElasticSearch-Index with bibliographic metadata and harvest RDF out of it

Language: Python - Size: 555 KB - Last synced: 6 days ago - Pushed: almost 4 years ago - Stars: 8 - Forks: 6

ClarityNLP/ClarityNLP

An NLP framework for clinical phenotyping. Docker | Python | Solr | OMOP. http://claritynlp.readthedocs.io/en/latest/

Language: Jupyter Notebook - Size: 190 MB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 105 - Forks: 41

Pawsanie/Steam_statistics_ETL

This pipeline can be used to collect statistical information about all games, distributed through the Steam platform.

Language: Python - Size: 2.35 MB - Last synced: 5 months ago - Pushed: 5 months ago - Stars: 1 - Forks: 0

binste/pipecutter 📦

pipecutter provides a few tools for luigi such that it works better with data science libraries and environments such as pandas, scikit-learn, and Jupyter notebooks.

Language: Python - Size: 158 KB - Last synced: 7 days ago - Pushed: almost 2 years ago - Stars: 4 - Forks: 0

mcguiremw/world_bank_data

Size: 1000 Bytes - Last synced: 8 months ago - Pushed: over 6 years ago - Stars: 0 - Forks: 2

mcguiremw/CuriosiPy

Language: Python - Size: 6.84 KB - Last synced: 8 months ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

shirosaidev/saisoku

Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.

Language: Python - Size: 121 KB - Last synced: 3 months ago - Pushed: over 3 years ago - Stars: 44 - Forks: 3

BrunoSouza88/Intro_ETL_DIO

Language: Python - Size: 10.6 MB - Last synced: 18 days ago - Pushed: 10 months ago - Stars: 0 - Forks: 0

GoobyCorp/MLSSE 📦

This is an editor for the new 3DS release of Mario and Luigi Superstar Saga.

Language: Python - Size: 22.5 KB - Last synced: 9 months ago - Pushed: almost 5 years ago - Stars: 15 - Forks: 5

SAP-samples/kyma-mtx-faq-management

The FAQ management service allows customers to manage a list of frequently asked questions through an SAP Fiori interface. For this, it leverages multiple SAP technologies such as the Kyma runtime, SAP Cloud Application Programming Model (CAP), SAP Application Router, Luigi, Fundamental Vue, and SAP Fiori elements.

Language: Go - Size: 2.17 MB - Last synced: 9 months ago - Pushed: almost 2 years ago - Stars: 8 - Forks: 8

SAP-samples/business-network-spatial-service-samples

Contains configuration examples, sample code, and relevant documentation for the SAP Business Network's Network Spatial Service (NSS). Documentation is also provided that shows how to integrate NSS into external systems and apps.

Language: JavaScript - Size: 190 KB - Last synced: 9 months ago - Pushed: about 2 years ago - Stars: 1 - Forks: 0

vsevolodbazhan/movies

Luigi-based data pipeline that extracts top-50 movies of different genres from IMDb and stores them in MongoDB.

Language: Python - Size: 41 KB - Last synced: 10 months ago - Pushed: over 3 years ago - Stars: 0 - Forks: 0

alex-feldman9/parsing_ikea_il

Parsing data from www.ikea.co.il by Selenium and Luigi pipeline

Language: Python - Size: 11 MB - Last synced: 10 months ago - Pushed: almost 3 years ago - Stars: 1 - Forks: 1

MouseOnTheKeys/Luigi

Pycon 2017- Creating ETL tasks with Luigi package

Language: Python - Size: 36.1 KB - Last synced: about 1 month ago - Pushed: about 1 month ago - Stars: 0 - Forks: 0

Maseya/SMB-Utility

An SMB editor originally created by M.K.S

Language: C - Size: 562 KB - Last synced: 12 months ago - Pushed: over 4 years ago - Stars: 9 - Forks: 2

mjalajel/luigi-mapreduce

An attempt do create *real* MapReduce with luigi

Language: Python - Size: 13.7 KB - Last synced: about 1 year ago - Pushed: about 8 years ago - Stars: 1 - Forks: 0

1bk/simple-airports-analysis

A simple project to analyse Malaysian airports - an opportunity to play with tools like Luigi, Docker, and Metabase as part of an end-to-end analytics framework.

Language: Python - Size: 7.98 MB - Last synced: 5 months ago - Pushed: 10 months ago - Stars: 9 - Forks: 5

davidkhala/flow

Collection of scheduling tools

Language: Shell - Size: 68.4 KB - Last synced: 10 days ago - Pushed: 6 months ago - Stars: 0 - Forks: 0

spiside/docker-luigi

A project to help develop Luigi pipelines using Docker ✳️

Language: Shell - Size: 170 KB - Last synced: about 1 year ago - Pushed: about 3 years ago - Stars: 76 - Forks: 22

massimobiagioli/luigi-demo

**** EXPERIMENT **** Spotify/Luigi Demo Project

Language: Python - Size: 93.8 KB - Last synced: about 1 month ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

luiscabanzon/dsa

Language: Jupyter Notebook - Size: 394 KB - Last synced: 9 months ago - Pushed: almost 2 years ago - Stars: 3 - Forks: 14

OAODEV/luigi-central-docker

Alpine-based docker container to run a Luigi central scheduler

Language: Python - Size: 13.7 KB - Last synced: about 2 months ago - Pushed: about 1 year ago - Stars: 1 - Forks: 0

live-wire/expyriments 📦

Python experiments :snake:

Language: Python - Size: 9.8 MB - Last synced: 9 months ago - Pushed: over 4 years ago - Stars: 2 - Forks: 0

Pawsanie/Luigi_ETL

Universal Luigi ETL pipeline. Validates data received from external sources. Extracts, transforms them and lands.

Language: Python - Size: 95.7 KB - Last synced: about 1 year ago - Pushed: about 1 year ago - Stars: 0 - Forks: 0

andrewsosa/luigi-demo

Demo of using Luigi to build ETL Pipelines

Language: Python - Size: 32.2 KB - Last synced: about 1 year ago - Pushed: over 4 years ago - Stars: 1 - Forks: 0

colombiaOrg/testsideremoveaccount

Language: TypeScript - Size: 957 KB - Last synced: 5 months ago - Pushed: about 1 year ago - Stars: 0 - Forks: 0

pacifikus/top_music_analysis

Spotify top songs clustering

Language: Jupyter Notebook - Size: 4.96 MB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

MtnFranke/semantic-pdf-splitter 📦

Language: Python - Size: 10.7 KB - Last synced: 21 days ago - Pushed: over 7 years ago - Stars: 3 - Forks: 1

Oslandia/osm-data-classification 📦

Migrated to: https://gitlab.com/Oslandia/osm-data-classification

Language: Python - Size: 108 MB - Last synced: about 1 year ago - Pushed: over 4 years ago - Stars: 23 - Forks: 7

gsi-upm/Sefarad

Language: JavaScript - Size: 179 MB - Last synced: 27 days ago - Pushed: over 5 years ago - Stars: 6 - Forks: 2

fruityloops1/piillo

Mario & Luigi: Dream Team Custom Code

Language: C++ - Size: 119 KB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 4 - Forks: 1

havrikov/covering-input-structure

Replication package for the paper "Systematically Covering Input Structure" published at ASE 2019

Language: Python - Size: 40 KB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 7 - Forks: 1

adilkhash/pycon-ru-2019-etl-examples

Luigi, Airflow, Prefect Examples for PyCon RU 2019 Presentation

Language: Python - Size: 3.91 KB - Last synced: about 1 year ago - Pushed: about 4 years ago - Stars: 3 - Forks: 0

SAP-samples/luigi-micro-frontend-application

Learn how to build a Luigi micro-frontend application with React and UI5. It demonstrates the usage of the Luigi open source project.

Language: JavaScript - Size: 1.72 MB - Last synced: 9 months ago - Pushed: about 1 year ago - Stars: 6 - Forks: 6

cereniyim/Wine-Rating-Predictor-ML-Model

Automated ML pipeline with Python, Docker, Luigi, SciKit-Learn and Pandas to predict wine quality ratings

Language: Jupyter Notebook - Size: 25.4 MB - Last synced: about 1 year ago - Pushed: almost 4 years ago - Stars: 15 - Forks: 12

korniichuk/workflow

Workflow management platforms comparison

Language: Python - Size: 57.6 KB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 4 - Forks: 2

adilkhash/luigi-course-materials

Материалы для курса Введение в Data Engineering: дата пайплайны

Language: Python - Size: 27.3 KB - Last synced: about 1 year ago - Pushed: over 2 years ago - Stars: 10 - Forks: 6

grishenkovp/luigi

Сборник дагов для оркестрации etl-процессов в Luigi

Language: Python - Size: 2.93 KB - Last synced: about 1 year ago - Pushed: about 2 years ago - Stars: 0 - Forks: 0

axiom-data-science/docker-luigi

🐍 🐳 Luigi in Docker - alpine and ubuntu images available

Language: Dockerfile - Size: 79.1 KB - Last synced: 10 days ago - Pushed: about 3 years ago - Stars: 50 - Forks: 20

younggrasshopperza/super-mario-css-animation

A fun Super Mario CSS Animation that you can check out.

Language: CSS - Size: 10.7 KB - Last synced: about 1 year ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

sanogotech/luigi_demo Fork of Kapernikov/luigi_demo

demo / tutorial of python and luigi

Language: Python - Size: 21.5 MB - Last synced: about 1 year ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

sanogotech/Luigi-ETL-Python-Tutorial Fork of kadnan/Luigi-Python-Tutorial

An introductory tutorial about Python Luigi. ETL

Language: Python - Size: 4.88 KB - Last synced: about 1 year ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

shotarok/vw-luigi

luigi workflows to evaluate models trained by vowpal wabbit

Language: Python - Size: 5.86 KB - Last synced: 11 months ago - Pushed: over 2 years ago - Stars: 6 - Forks: 0

nestauk/daps_utils

Tools for setting up and running pipelines in a Data Analytics and Production System (DAPS).

Language: Python - Size: 246 KB - Last synced: 27 days ago - Pushed: over 1 year ago - Stars: 0 - Forks: 1

meliache/b2luigi_summary_talk

For joint meeting with LAW developers

Language: TeX - Size: 8.08 MB - Last synced: about 1 year ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

stejul/smikic-dwh

Personal DWH to test out different tools

Language: Python - Size: 2.21 MB - Last synced: about 1 year ago - Pushed: over 2 years ago - Stars: 0 - Forks: 0

naimazizi/hive-export

Combination between Apache Spark and Sqoop to extract data from Hive table into relational database, integrated with pipeline using luigi.

Language: Python - Size: 11.7 KB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 1 - Forks: 0

ghazimuharam/luigi-etl-pipeline

ETL Data pipeline using Luigi

Language: Python - Size: 1.25 MB - Last synced: about 1 year ago - Pushed: almost 3 years ago - Stars: 1 - Forks: 0

wyattscarpenter/year-of-luigi Fork of cht8687/year-of-snake

Check if it is year of luigi

Language: JavaScript - Size: 11.7 KB - Last synced: 7 days ago - Pushed: almost 3 years ago - Stars: 0 - Forks: 0

neelshah23/micro-frontend-demo-luigi

Sample micro-frontend architecture with use of Luigi framework.

Language: CSS - Size: 5.3 MB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 0 - Forks: 0

anandasaia/Wine-Rating-Predictor-Pipeline

Wine Rating Predictor Pipeline made with Python, Docker, Pandas, Luigi, Scikit-Learn

Language: Jupyter Notebook - Size: 99.6 KB - Last synced: about 1 year ago - Pushed: about 3 years ago - Stars: 0 - Forks: 0

snimmagadda1/Luigi-ETL-Example

🔍 Example of an ETL pipeline using Spotify's Luigi

Language: Python - Size: 34.2 KB - Last synced: about 1 year ago - Pushed: over 5 years ago - Stars: 0 - Forks: 0

mynameisvinn/waluigi

reimplemented luigi

Language: Jupyter Notebook - Size: 3.91 KB - Last synced: about 1 year ago - Pushed: over 3 years ago - Stars: 1 - Forks: 0

maxhollmann/luigi_soft_failures

Allow Luigi tasks to fail softly, without cancelling downstream tasks.

Language: Python - Size: 10.7 KB - Last synced: 6 days ago - Pushed: about 3 years ago - Stars: 3 - Forks: 1

shmokmt/luigi-sample

Luigi sample project.

Language: Python - Size: 10.7 KB - Last synced: about 1 year ago - Pushed: about 4 years ago - Stars: 0 - Forks: 0

joaoariedi/etl_luigi

Using Luigi to ETL data from APIs and DBs

Language: Python - Size: 89.8 KB - Last synced: about 1 year ago - Pushed: over 1 year ago - Stars: 2 - Forks: 1

ohtacaesar-sandbox/luigi-sandbox

Language: Python - Size: 6.84 KB - Last synced: about 1 year ago - Pushed: over 4 years ago - Stars: 0 - Forks: 0

miku/batchdata

Batch data processing with luigi, 90min workshop at PyCon Balkan 2018, Belgrade.

Language: Python - Size: 226 KB - Last synced: about 1 year ago - Pushed: over 5 years ago - Stars: 7 - Forks: 0

Bennyhwanggggg/Twitter-Data-ETL-Pipeline

Data pipeline using Twitter API and MySQL

Language: Python - Size: 7.81 KB - Last synced: about 1 year ago - Pushed: over 4 years ago - Stars: 0 - Forks: 0

hyzhak/luigi-mlflow

mlflow task for luigi pipeline

Language: Python - Size: 6.84 KB - Last synced: 27 days ago - Pushed: almost 5 years ago - Stars: 0 - Forks: 0

jezzarax/luigi_resource_limitation_example

Just a toy example on how to limit the number of tasks accessing the same resource. Couldn't fully get it from the documentation.

Language: Python - Size: 4.88 KB - Last synced: 12 months ago - Pushed: almost 5 years ago - Stars: 1 - Forks: 0

bfemiano/song_plays_workshop_tutorial

Song Plays Workshop Tutorial

Language: Python - Size: 1.84 MB - Last synced: about 1 year ago - Pushed: over 3 years ago - Stars: 14 - Forks: 4

danthelion/luigi-example

Example luigi code

Language: Python - Size: 2.93 KB - Last synced: about 1 year ago - Pushed: almost 7 years ago - Stars: 1 - Forks: 0

omkarprabhu-98/mapreduce-jobs-for-hadoop

Collection of examples for jobs in python for Hadoop using MapReduce programming model

Language: Python - Size: 76.2 KB - Last synced: about 1 year ago - Pushed: almost 6 years ago - Stars: 0 - Forks: 0

antoniopenta/ml-project-structure

Easy and powerful template for ML projects

Language: Python - Size: 33.2 KB - Last synced: about 1 year ago - Pushed: over 5 years ago - Stars: 0 - Forks: 0

jmeisele/datapipelines

Data Pipelines Using Luigi

Language: Jupyter Notebook - Size: 379 KB - Last synced: about 1 year ago - Pushed: over 5 years ago - Stars: 0 - Forks: 0

nel215/file-dependency-check-mixin

Language: Python - Size: 5.86 KB - Last synced: about 1 year ago - Pushed: over 5 years ago - Stars: 0 - Forks: 0

danrg/luigi_poc

Proof of Concept for a basic Luigi-based pipeline

Language: Python - Size: 1.95 KB - Last synced: about 1 year ago - Pushed: over 5 years ago - Stars: 0 - Forks: 0

aktech/luigi-talk

A talk on Data Pipelining with luigi @ PyData London Meetup

Language: Python - Size: 6.36 MB - Last synced: 11 days ago - Pushed: over 6 years ago - Stars: 2 - Forks: 0

goern/peach

This will aggregate buildlogs from Travis

Language: Python - Size: 33.2 KB - Last synced: about 1 year ago - Pushed: about 6 years ago - Stars: 0 - Forks: 0

elsonidoq/bicis

Demand forecast of Buenos Aires public bicycles

Language: Jupyter Notebook - Size: 278 KB - Last synced: about 1 year ago - Pushed: about 6 years ago - Stars: 0 - Forks: 2

sncf-connect-tech/meetup-afpy-nantes-luigi

Slides du meetup Python AFPY à Voyages-sncf.com Nantes du 3 octobre 2017 : luigi & behave

Language: HTML - Size: 1.92 MB - Last synced: 28 days ago - Pushed: over 6 years ago - Stars: 2 - Forks: 0

Tejesh0711/Live-Blog-Analysis

Language: JavaScript - Size: 68.7 MB - Last synced: about 1 year ago - Pushed: over 6 years ago - Stars: 0 - Forks: 0

howawong/luigi_pipeline_demo

Luigi Pipeline Demo for https://www.facebook.com/events/144094749445572/

Language: Python - Size: 4.06 MB - Last synced: about 1 year ago - Pushed: about 7 years ago - Stars: 0 - Forks: 0

xrmx/primadatapipeline

La mia prima data pipeline talk demo project

Language: Python - Size: 14.6 KB - Last synced: 3 days ago - Pushed: about 7 years ago - Stars: 3 - Forks: 0

tilacog/luigi-talk-pydatabh-example

Code used in PyData BH #1

Language: Python - Size: 5.86 KB - Last synced: about 1 year ago - Pushed: about 7 years ago - Stars: 0 - Forks: 0