datafactory | Topic | Ecosyste.ms: Repos

Topic: "datafactory"

Azure-Samples/modern-data-warehouse-dataops

DataOps for Microsoft Data Platform technologies. https://aka.ms/dataops-repo

Language: Shell - Size: 120 MB - Last synced at: 4 days ago - Pushed at: 4 days ago - Stars: 657 - Forks: 506

rlevchenko/terraform-azure-data

Terraform script to deploy almost all Azure Data Services

Language: HCL - Size: 26.4 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 35 - Forks: 26

DataTech-Solutions/Threat-Detection-and-Visualization

Threat Detection and Visualization

Language: TSQL - Size: 11.9 MB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 25 - Forks: 153

NandanHegde15/Azure-DataFactory-Generic-Pipelines

Generic Pipelines / Templates for Data Factory / Synapse Pipelines w.r.t Different MSFT Offering Integrations / Use Cases

Size: 774 KB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 10 - Forks: 4

guderkar/ACIProxy

Azure Container Instances Proxy implemented in Azure Function App (Consumption Plan)

Language: Python - Size: 438 KB - Last synced at: over 2 years ago - Pushed at: almost 3 years ago - Stars: 9 - Forks: 9

SixGod191001/azure-adf-framework

The ADF Universal Framework is an open-source project designed to provide a comprehensive and flexible solution for building scalable and efficient data integration workflows using Azure Data Factory (ADF).

Language: TSQL - Size: 6.05 MB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 3 - Forks: 1

federicopfund/data-engineer

Proceso ETL

Language: Jupyter Notebook - Size: 84.5 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 3 - Forks: 0

tknishh/olympic-data-analysis-azure

End-to-End data engineering project with Azure Databricks as cloud service and Tokyo olympic data

Language: Jupyter Notebook - Size: 1.07 MB - Last synced at: 24 days ago - Pushed at: about 2 years ago - Stars: 3 - Forks: 3

aessing/demo-mdwh

Modern Dataware House Demos with Azure Databricks, Azure Data Factory & Azure Dedicated SQL pool (formerly SQL DW)

Size: 48.3 MB - Last synced at: 2 months ago - Pushed at: over 4 years ago - Stars: 3 - Forks: 2

adharsh277/INTERNSHIP-PROJECT-

End-to-end Azure-based data engineering pipeline for CRM analytics using Data Factory, Databricks, Synapse, and Power BI.

Size: 33.2 MB - Last synced at: about 14 hours ago - Pushed at: about 16 hours ago - Stars: 2 - Forks: 0

snhaider9977/Datafactory-CI-CD

This repository serves as a sample demonstration of implementing CI/CD practices on Azure Data Factory using Azure DevOps. It provides an example approach to configure a pipeline for continuous integration and continuous deployment (CI/CD) workflows on Azure Data Factory

Size: 2.41 MB - Last synced at: about 2 years ago - Pushed at: about 2 years ago - Stars: 2 - Forks: 0

mehroosali/databricks-F1-Project

A data pipeline project build on databricks and azure to demostrate lifecycle of a cloud data project.

Language: Jupyter Notebook - Size: 5.98 MB - Last synced at: about 2 years ago - Pushed at: over 3 years ago - Stars: 2 - Forks: 2

goamegah/daie-book

Data and Artificial Intelligence Engineering book - Build Modern Data Platform with Azure and Databricks

Language: Python - Size: 63.5 KB - Last synced at: about 2 months ago - Pushed at: 3 months ago - Stars: 1 - Forks: 0

venkatakamaiah46/Python

Interesting_programs_written_in_Python_language

Size: 10.7 KB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 1 - Forks: 0

thanaphongK37/Data-Science-and-Data-Analyst-Project

Portfolio Data Analysis and Data Science projects and Data Engineer built using Azure Service, SQL and Python.

Language: Jupyter Notebook - Size: 3.92 MB - Last synced at: 5 months ago - Pushed at: 7 months ago - Stars: 1 - Forks: 0

ayanhussain81/Olympics-Data-ETL

Tokyo Olympic Azure Data Engineering Project

Language: Jupyter Notebook - Size: 734 KB - Last synced at: 5 months ago - Pushed at: 12 months ago - Stars: 1 - Forks: 0

MekWiset/Medallion_DataLakehouse

Building a Data Lakehouse using the Medallion architecture.

Language: Jupyter Notebook - Size: 887 KB - Last synced at: 5 months ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 3

bennyaustin/logicapp-azure-resourcemgmt

Logic App that calls Azure Resource Management REST API to get information about resources in a subscription

Language: PowerShell - Size: 43.9 KB - Last synced at: about 1 year ago - Pushed at: about 1 year ago - Stars: 1 - Forks: 0

shubhammirajkar/tokyo_olympics_de_project

Explore the Tokyo Olympics data journey! We ingested a GitHub CSV into Azure via Data Factory, stored it in Data Lake Storage Gen2, performed transformations in Databricks, conducted advanced analytics in Azure Synapse, and visualized insights in Synapse or Power BI.

Language: Jupyter Notebook - Size: 1.17 MB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 1 - Forks: 0

tuanai-vireox/Azure-LandingZone-Demo

Azure Data Landing Zone - IAC , Config, Code

Language: HCL - Size: 16.6 MB - Last synced at: 17 days ago - Pushed at: almost 2 years ago - Stars: 1 - Forks: 0

BrioGit/DataFactoryAutomation

This application shows how to use the Azure .NET library to create and execute Azure Data Factory from C# code.

Language: C# - Size: 384 KB - Last synced at: over 1 year ago - Pushed at: about 2 years ago - Stars: 1 - Forks: 0

bennyaustin/ADFCustomActivity-AASPartition

Repository for Azure Data Factory (ADF) Custom Activity to dynamically create and process Azure Analysis Service (AAS) Tabular Model Partitions

Language: C# - Size: 22.5 KB - Last synced at: over 1 year ago - Pushed at: about 3 years ago - Stars: 1 - Forks: 1

camposvinicius/azure-etl

This pipeline is an ETL application, which aims to collect data for a few minutes from Binance's open API for cryptocurrencies: BTCUSDT, ADAUSDT, ETHUSDT, BNBUSDT and LTCUSDT.

Language: Smarty - Size: 1.3 MB - Last synced at: over 2 years ago - Pushed at: over 3 years ago - Stars: 1 - Forks: 5

sandman153/WWLogs

KQL Queries for WWLogs Presentation

Size: 9.4 MB - Last synced at: over 2 years ago - Pushed at: over 3 years ago - Stars: 1 - Forks: 0

liliankotvan/Azure-ADF-RefreshPBI

Azure Data Factory template to refresh Power BI Dataset

Size: 8.79 KB - Last synced at: over 2 years ago - Pushed at: over 3 years ago - Stars: 1 - Forks: 0

armdev/rethinkdb-jsf

rethinkdb jsf 2.3 project

Language: Java - Size: 54.7 KB - Last synced at: over 1 year ago - Pushed at: over 4 years ago - Stars: 1 - Forks: 0

akfincode/azure-powershell

PowerShell Tools

Language: PowerShell - Size: 8.79 KB - Last synced at: about 2 years ago - Pushed at: over 4 years ago - Stars: 1 - Forks: 0

Alex-Nettekoven/ETL-Azure-SF-Pipeline

Azure Data Factory & Snowflake ETL Pipeline

Language: Python - Size: 1.95 KB - Last synced at: 11 days ago - Pushed at: 12 days ago - Stars: 0 - Forks: 0

abbgrade/PsDataFactory

PsDataFactory is a PowerShell module to generate DataFactory pipelines and datasets.

Language: PowerShell - Size: 73.2 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 2

jeromedayrit/rg-salescars-adeproject

A data pipeline project that processes car sales data using the Medallion Architecture (Bronze, Silver, Gold) to deliver clean, reliable insights for reporting

Size: 0 Bytes - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

Saikesana31/Netflix

Azure Data engineering project

Language: Python - Size: 1.3 MB - Last synced at: 5 months ago - Pushed at: 5 months ago - Stars: 0 - Forks: 0

Saikesana31/Adventure_Works_DE

Azure Data engineering project

Size: 2.26 MB - Last synced at: 6 months ago - Pushed at: 6 months ago - Stars: 0 - Forks: 0

ryanninodizon/EHR-PoC

End-to-End Concept Project: Azure Data Factory Pipeline, Azure IoT Central, Azure Storage Account, Azure SQL, .NET, Angular

Language: TypeScript - Size: 42.4 MB - Last synced at: 5 months ago - Pushed at: 7 months ago - Stars: 0 - Forks: 0

AlefRP/lakehouse_azure

Language: Jupyter Notebook - Size: 26 MB - Last synced at: 3 months ago - Pushed at: 7 months ago - Stars: 0 - Forks: 0

kathisnehith/Azure-dataIngestion-medicare-claims-Analysis

Analyzing Medicare claims to uncover insights and trends in healthcare expenses through batch Ingestion

Language: TSQL - Size: 207 KB - Last synced at: about 1 month ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

Carolinerocks/azure-data-engineering-end-to-end-project

Language: Jupyter Notebook - Size: 3.36 MB - Last synced at: about 1 month ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

vskvj3/End-to-End-Pipeline-for-Swiggy-Restaurant-Data

This project develops a data engineering pipeline to analyze restaurant data from various cities on the Swiggy platform. Using PySpark, Spark SQL, and Azure Data Factory, the data is processed and transformed to generate insights on ratings, cuisines, and trends, presented through dashboards and reports.

Language: Jupyter Notebook - Size: 18.3 MB - Last synced at: 8 months ago - Pushed at: 8 months ago - Stars: 0 - Forks: 0

jugi92/add_servicebus_ips_for_shir_to_nsg

This script resolves the Azure Service Bus IP addresses used by an integration runtime and adds them to the network security group (NSG) rule for the Synapse / Data Factory self-hosted integration runtime (SHIR).

Language: PowerShell - Size: 1000 Bytes - Last synced at: 5 months ago - Pushed at: 10 months ago - Stars: 0 - Forks: 0

SonuRepo/paris_olympic_azure_project

Explore the Paris Olympics data journey! We ingested a GitHub CSV into Azure via Data Factory, stored it in Data Lake Storage Gen2, performed transformations in Databricks, conducted analytics in Azure Synapse, and visualized insights in Synapse.

Language: Jupyter Notebook - Size: 3.37 MB - Last synced at: 3 months ago - Pushed at: 12 months ago - Stars: 0 - Forks: 0

briandenicola/datafactory-batchjob-demo

This is to demo how to use Azure Data Factory with AKS/KEDA to run batch jobs in Azure

Language: HCL - Size: 124 KB - Last synced at: 6 months ago - Pushed at: about 1 year ago - Stars: 0 - Forks: 0

nazish555/Tokyo-Olympics-Azure-Data-Engineering-Project

This project leverages Azure Cloud services like Azure Data Factory, Azure Databricks, and Synapse Analytics to execute a data engineering workflow. Utilizing data sourced from the Olympic API on GitHub, it involves extracting raw data into Azure Data Lake Storage, transforming it with PySpark on Azure Databricks, and analyzing the transformed data

Language: Jupyter Notebook - Size: 337 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

micvet/data-eng-project-amazon

O objetivo deste projeto foi aplicar os conhecimentos nas ferramentas de extração e tratamento de dados da plataforma Azure.

Language: Jupyter Notebook - Size: 78.1 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

salmanmkc/Azure-Data-Platform

Short guide as to how to set up the ADF Project :)

Size: 105 KB - Last synced at: 8 days ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

hannah0wang/end-to-end-data-reporting

End to end data reporting project using Azure services like Azure Data Factory for data orchestration, Azure Synapse Analytics for data warehousing, Databricks for data transformations, and Power BI for intuitive data visualization and reporting.

Language: Jupyter Notebook - Size: 515 KB - Last synced at: over 1 year ago - Pushed at: over 1 year ago - Stars: 0 - Forks: 0

DataEngel/azure-data-factory-tests

HandsOn for data factory, just for test the service in general

Size: 12.7 KB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0

Kokit0/Tokyo-Olympics-Azure-Data-Eng-Project

Datas Engineer Exploration of the 2021 Tokyo Olympic data using a variety of tools and technologies, with a primary focus on Azure services, Leveraging Azure Data Factory, Data Lake Gen 2, Synapse Analytics, and Azure Databricks to perform data engineering tasks, analysis, and visualization.

Language: Jupyter Notebook - Size: 164 KB - Last synced at: almost 2 years ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 0

epomatti/az-e2e-data-eng-proj

Data engineering with Azure services

Language: HCL - Size: 404 KB - Last synced at: 3 months ago - Pushed at: almost 2 years ago - Stars: 0 - Forks: 1

bobbyngo/Formula1

Formula1 ADF pipeline

Language: Python - Size: 1.31 MB - Last synced at: about 2 years ago - Pushed at: about 2 years ago - Stars: 0 - Forks: 0

EthanBauer/adf

Repository to connect github to Azure Data Factory instance for CI/CD pipeline

Size: 21.5 KB - Last synced at: over 1 year ago - Pushed at: about 2 years ago - Stars: 0 - Forks: 0

luccayz/dataengineer_project_001

Efetuar o download de arquivos da web com Python. Inserir dados de um dataframe na cloud Azure com Azure SQL Database. Efetuar transformações nos dados com Azure Data Factory.

Language: Jupyter Notebook - Size: 11.7 KB - Last synced at: about 2 years ago - Pushed at: about 2 years ago - Stars: 0 - Forks: 0

epomatti/az-adf-s3-blob

Data integration with ADF

Language: HCL - Size: 22.5 KB - Last synced at: 6 months ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0

MuhammadHasaanWahid/Data-Filtering-Pipeline-ETL

This Project Extracts supply chain data from csv file having 180k records and more than 40 columns from the Azure Datalake Gen2 storage account and do some dataanalysis with Python(Pandas) to find the top 3 countries and filtered the data for top 3 countries and finally transferred it to 3 files in datalake again by creating ETL pipeline in ADF.

Language: Jupyter Notebook - Size: 8.79 KB - Last synced at: over 2 years ago - Pushed at: over 2 years ago - Stars: 0 - Forks: 0