GitHub topics: databricks | Ecosyste.ms: Repos

Udau9/Taxi-fare-Prediction

Taxi fare prediction using Machine learning models

Language: HTML - Size: 4.86 MB - Last synced at: 26 days ago - Pushed at: 26 days ago - Stars: 0 - Forks: 0

felipesbonatti/distribuicao-clientes-bancarios

Alocação inteligente de clientes bancários baseada em geolocalização e rentabilidade, utilizando processamento distribuído (Spark) com validações de qualidade.

Language: Jupyter Notebook - Size: 36.1 KB - Last synced at: 26 days ago - Pushed at: 26 days ago - Stars: 1 - Forks: 0

felipesbonatti/databricks-alerts-to-teams

Solução que valida automaticamente pipelines de dados no Databricks, detecta anomalias em tempo real e notifica equipes via Microsoft Teams. Reduziu em 90% o tempo de resposta a falhas em ambientes corporativos.

Language: Jupyter Notebook - Size: 13.7 KB - Last synced at: 26 days ago - Pushed at: 26 days ago - Stars: 1 - Forks: 0

geekwhocodes/pyspark-custom-datasource-template

The PySpark Custom Data Source Template makes it easy to build and test custom data sources for Apache PySpark. It simplifies environment setup, debugging, and test data management while providing a structured, ready-to-use foundation.

Language: Python - Size: 61.5 KB - Last synced at: 3 days ago - Pushed at: 2 months ago - Stars: 1 - Forks: 0

MiChaelinzo/Implant-Insights

Accelerating Healthcare with Databricks. In-development conceptual example of live chat with an AI Agent with AWS:

Language: Python - Size: 1.37 MB - Last synced at: 28 days ago - Pushed at: 28 days ago - Stars: 3 - Forks: 1

getstrm/pace

Data policy IN, dynamic view OUT: PACE is the Policy As Code Engine. It helps you to programatically create and apply a data policy to a processing platform like Databricks, Snowflake or BigQuery (or plain 'ol Postgres, even!) with definitions imported from Collibra, Datahub, ODD and the like.

Language: Kotlin - Size: 13.1 MB - Last synced at: 7 days ago - Pushed at: 7 days ago - Stars: 36 - Forks: 1

alexott/dlt-files-in-repos-demo

Demonstration of using Files in Repos with Databricks Delta Live Tables

Language: HCL - Size: 1.5 MB - Last synced at: 14 days ago - Pushed at: 10 months ago - Stars: 32 - Forks: 25

Hamza88-coder/Real-Time-Recruitment-System-with-AI-and-Data-Analytics

Simulation of job offers and CVs with real-time processing, classification, and analytics using Kafka, Ray, Spark, and Databricks. Includes a Flask-based recommendation system and Tableau visualizations.

Language: Jupyter Notebook - Size: 26.7 MB - Last synced at: 24 days ago - Pushed at: 4 months ago - Stars: 9 - Forks: 3

renardeinside/terrametria

Source code 3D population density map of Germany, with ETL and app logic on top the Databricks Platform.

Language: TypeScript - Size: 2 MB - Last synced at: 22 days ago - Pushed at: 5 months ago - Stars: 5 - Forks: 4

EmilyDanie/Databricks-Certified-Professional-Data-Scientist-Exam-Dumps

Databricks Certified Professional Data Scientist Exam Dumps – Pass with Confidence | PasscertHub

Size: 0 Bytes - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

StevenMatthew7/Databricks-Certified-Data-Engineer-Associate-Exam

Databricks Certified Data Engineer Associate Exam Dumps – Pass with Confidence | PasscertHub

Size: 0 Bytes - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

adidas/lakehouse-engine

The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.

Language: Python - Size: 8.16 MB - Last synced at: 15 days ago - Pushed at: 3 months ago - Stars: 240 - Forks: 44

rayalex/spark-databricks-observability

Monitoring Databricks using Prometheus, Grafana and Pyroscope

Language: HCL - Size: 6.38 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 19 - Forks: 3

yeopster/Data-Engineering-Azure

Basic Data Engineering Project Using Microsoft Azure

Language: Jupyter Notebook - Size: 2.26 MB - Last synced at: 25 days ago - Pushed at: 3 months ago - Stars: 0 - Forks: 0

Shahir-47/front-end Fork of HackHarvard2024-Team/front-end

AI-powered navigation tool for safer travel, avoiding high-risk zones using real-time crime data and intelligent routing.

Language: Vue - Size: 237 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

HackHarvard2024-Team/front-end

AI-powered navigation tool for safer travel, avoiding high-risk zones using real-time crime data and intelligent routing.

Language: Vue - Size: 227 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 1

unytics/catalog_builder

Data Catalogs Made Easy

Language: Python - Size: 2.64 MB - Last synced at: 2 days ago - Pushed at: about 1 month ago - Stars: 22 - Forks: 2

lamastex/scalable-data-science

Scalable Data Science, course sets in big data Using Apache Spark over databricks and their mathematical, statistical and computational foundations using SageMath.

Language: HTML - Size: 1.24 GB - Last synced at: 15 days ago - Pushed at: 3 months ago - Stars: 167 - Forks: 92

davidkhala/gcp-collections

Notebooks for GCP services

Language: Python - Size: 247 KB - Last synced at: 22 days ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

GB1609/ComputerScienceNotes

Simple repository in which I keep track of some notes using OBSIDIAN, a powerful and extensible knowledge base tool.

Size: 21.4 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 1 - Forks: 0

venkatakamaiah46/Python

Interesting_programs_written_in_Python_language

Size: 10.7 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 1 - Forks: 0

Azure-Samples/azure-databricks-mlops-mlflow

Azure Databricks MLOps sample for Python based source code using MLflow without using MLflow Project.

Language: Jupyter Notebook - Size: 3.2 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 84 - Forks: 53

djouallah/Testing_BI_Engine

TPC-H_SF10

Language: Jupyter Notebook - Size: 1.32 MB - Last synced at: 1 day ago - Pushed at: 3 months ago - Stars: 52 - Forks: 16

DarrenDavy12/Azure-Databricks-Setup-Guide-with-Formula1-CSV

Azure Databricks Setup Guide with Formula1 CSV - Azure Databricks, PySpark, Python, Data Lake Storage

Size: 3.42 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

databrickslabs/overwatch

Capture deep metrics on one or all assets within a Databricks workspace

Language: Scala - Size: 37.6 MB - Last synced at: about 1 month ago - Pushed at: 5 months ago - Stars: 232 - Forks: 68

souvik-databricks/dlt-with-debug

A lightweight helper utility which allows developers to do interactive pipeline development by having a unified source code for both DLT run and Non-DLT interactive notebook run.

Language: Python - Size: 88.9 KB - Last synced at: 5 days ago - Pushed at: over 2 years ago - Stars: 49 - Forks: 8

edisedis777/PySpark-ML-Features

A PySpark implementation of 6 lesser-known Scikit-Learn features optimized for Azure Databricks. This project translates powerful machine learning techniques from Scikit-Learn into PySpark's distributed computing framework.

Language: Python - Size: 8.79 KB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 1 - Forks: 0

databrickslabs/dolly

Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform

Language: Python - Size: 96.7 KB - Last synced at: about 1 month ago - Pushed at: almost 2 years ago - Stars: 10,816 - Forks: 1,160

lucashomuniz/Project-05

DATA ENGINEERING FOR OLYMPICS USING AZURE, SQL AND PBI

Language: Jupyter Notebook - Size: 1.38 MB - Last synced at: 28 days ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

yokawasa/databricks-notebooks

Collection of Sample Databricks Spark Notebooks ( mostly for Azure Databricks )

Language: Jupyter Notebook - Size: 15.6 KB - Last synced at: 20 days ago - Pushed at: almost 7 years ago - Stars: 86 - Forks: 75

lhbench/lhbench

Lakehouse storage system benchmark

Language: Scala - Size: 8.94 MB - Last synced at: 23 days ago - Pushed at: about 2 years ago - Stars: 72 - Forks: 9

Mauriciorodriguez94/BrickLayers

Interlocking Layers Post-Processing Script for PrusaSlicer, OrcaSlicer, and BambuStudio

Language: Python - Size: 1.57 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

tahir007malik/adventureWorksDataAnalytics

This repository showcases an end-to-end ETL pipeline leveraging Azure services, including ADF, ADLS Gen2, Databricks, and Synapse Analytics, to enhance data processing efficiency.

Language: Jupyter Notebook - Size: 3.5 MB - Last synced at: 27 days ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 1

Saikesana31/Adventure_Works_DE

Azure Data engineering project

Size: 2.26 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

VaibhavBansal26/Design-Medallion-Architecture-Azure

Azure Data Factory, Azure Databricks, DBT

Size: 22.5 KB - Last synced at: 28 days ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

databrickslabs/dbx

🧱 Databricks CLI eXtensions - aka dbx is a CLI tool for development and advanced Databricks workflows management.

Language: Python - Size: 1.78 MB - Last synced at: about 1 month ago - Pushed at: 7 months ago - Stars: 450 - Forks: 124

DadaNanjesha/AzureDataEngine

AzureDataEngine is a robust, scalable batch processing data architecture built on the Azure platform. It efficiently extracts, transforms, and loads massive datasets for machine learning applications, leveraging Azure Blob Storage, PostgreSQL, Databricks, and Key Vault to ensure reliability and maintainability.

Language: Python - Size: 1.05 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 1 - Forks: 0

hackolade/DeltaLake

Hackolade(https://hackolade.com) plugin for Delta Lake on Databricks

Language: JavaScript - Size: 11.7 MB - Last synced at: 13 days ago - Pushed at: 13 days ago - Stars: 4 - Forks: 12

ak-abhilash/FIFA-World-Cup-Data-Pipeline-with-Azure

This repository showcases a data pipeline built using Azure services to process and analyze the FIFA World Cup dataset. The pipeline includes data ingestion, transformation, and analytics, culminating in visualizations using Power BI, Looker Studio, and Tableau.

Language: Jupyter Notebook - Size: 2.14 MB - Last synced at: about 1 month ago - Pushed at: about 1 month ago - Stars: 0 - Forks: 0

rafaelpierre/pyjaws

PyJaws: A Pythonic Way to Define Databricks Jobs and Workflows

Language: Python - Size: 3.46 MB - Last synced at: 5 days ago - Pushed at: 10 months ago - Stars: 43 - Forks: 3

databrickslabs/splunk-integration

Databricks Add-on for Splunk

Language: Python - Size: 71.5 MB - Last synced at: about 1 month ago - Pushed at: about 2 months ago - Stars: 26 - Forks: 18

databricks/unity-catalog-setup

Notebooks, terraform, tools to enable setting up Unity Catalog

Size: 1.42 MB - Last synced at: 7 days ago - Pushed at: over 2 years ago - Stars: 47 - Forks: 32

benitomartin/mlops-databricks-credit-default

End-to-end MLOps Credit Default Project using DABs

Language: Python - Size: 1.08 MB - Last synced at: 17 days ago - Pushed at: 5 months ago - Stars: 17 - Forks: 16

data-miner00/spark

A laboratory to carry out experiments with PySpark

Language: Jupyter Notebook - Size: 245 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0

Azure/azure-cosmosdb-spark 📦

Apache Spark Connector for Azure Cosmos DB

Size: 192 MB - Last synced at: 2 days ago - Pushed at: about 2 months ago - Stars: 203 - Forks: 121

dkirrane/kafka-to-databricks-dlt

Fully automated Databricks PySpark DLT pipeline for integrating with Kafka Topics & Kafka Schema Registry

Language: Python - Size: 175 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 1 - Forks: 0

ndomah/The-Data-Engineering-Academy

Materials from The Data Engineering Academy

Size: 18.5 MB - Last synced at: about 1 month ago - Pushed at: 3 months ago - Stars: 1 - Forks: 0

lynnlangit/learn-databricks-genai

Learning Databricks GenAI

Language: Jupyter Notebook - Size: 1.32 MB - Last synced at: 3 days ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0

the69ma/IntensivoSQLnoDatabricks

IntensivoSQLnoDatabricks is a comprehensive GitHub repository dedicated to providing intense training and resources on SQL within the Databricks environment. It covers a wide range of topics from basic SQL queries to advanced database management and optimization techniques.

Size: 1000 Bytes - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0

kgelli/Apple-Data-Analysis---Apache-Spark

Modular ETL pipeline for analyzing Apple product purchase patterns using Apache Spark on Databricks with factory design patterns.

Language: Jupyter Notebook - Size: 202 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0

julianolaurentino/Databricks-notebooks-vendidos

Utilizando Databricks para analisar uma base em SQL de vendas de notebooks

Language: SQL - Size: 43 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 1 - Forks: 0

dan1elt0m/unitycatalog-migrate

Migrate Databricks Unity Catalog to OSS Unity Catalog

Language: Python - Size: 264 KB - Last synced at: 1 day ago - Pushed at: about 2 months ago - Stars: 3 - Forks: 0

Toloka/dbxio

High-level Databricks client

Language: Python - Size: 250 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 12 - Forks: 0

airscholar/modern-data-eng-dbt-databricks-azure

In this project, we setup and end to end data engineering using Apache Spark, Azure Databricks, Data Build Tool (DBT) using Azure as our cloud provider.

Size: 118 KB - Last synced at: 18 days ago - Pushed at: over 1 year ago - Stars: 27 - Forks: 13

vedanthv/data-engineering-portfolio

Cool DE Projects

Language: Jupyter Notebook - Size: 89.9 MB - Last synced at: 13 days ago - Pushed at: about 2 months ago - Stars: 25 - Forks: 4

soorajpazeekal/logistics-real-time-poc

A Data engineering based Proof of Concept demonstrating cutting-edge logistics solutions for a US-based Grocery Delivery Platform

Language: Jupyter Notebook - Size: 30.3 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 2 - Forks: 4

databrickslabs/pytester

Python Testing for Databricks

Language: Python - Size: 296 KB - Last synced at: about 1 month ago - Pushed at: about 2 months ago - Stars: 72 - Forks: 7

BlueGranite/DatabricksTraining

Repository for Microsoft Databricks Training Events - Hosted by BlueGranite

Language: Python - Size: 14 MB - Last synced at: 26 days ago - Pushed at: over 5 years ago - Stars: 15 - Forks: 8

santiagortiiz/Advanced-Data-Engineering-with-Databricks

Databricks. Incremental data processing, task orchestration, and production job monitoring.

Language: Python - Size: 121 KB - Last synced at: 12 days ago - Pushed at: about 1 year ago - Stars: 18 - Forks: 35

ricardolsmendes/fine-grained-demand-forecasting-infra

Infrastructure provisioning for a customized approach to the Databricks Fine-grained Demand Forecasting accelerator

Language: HCL - Size: 16.6 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0

reisdebora/awesome-databricks

A curated list of awesome Databricks resources, including Spark

Size: 27.3 KB - Last synced at: 2 days ago - Pushed at: 10 months ago - Stars: 17 - Forks: 3

kabeera1007/Cricket-ETL-Pipeline

End-to-End ETL Pipeline for Live Cricket Streaming Data Using AWS (Lambda, Glue, Step Functions, S3), Snowflake, and Power BI for Visualization

Language: Jupyter Notebook - Size: 311 KB - Last synced at: about 1 month ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0

TatevKaren/free-resources-books-papers

Books and Papers in Mathematics, Econometrics, Machine Learning, Finance etc for different levels that can be useful for Data Scientists, Developers and everyone whoo is interesting in STEM.

Size: 79.8 MB - Last synced at: about 1 month ago - Pushed at: about 4 years ago - Stars: 40 - Forks: 9

gustavoeso/PhishingURL-predictive-model

Projeto de Machine learning para desenvolver um modelo preditivo de busca de email que potencialmente seja phishing. Esse projeto foi desenvolvido no DataBricks, utilizando cluster voltado a MachineLearning.

Language: Jupyter Notebook - Size: 297 KB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

kgelli/Sales-Data-Analytics---Azure-End-to-End-Data-Engineering

End-to-end data engineering solution that transforms sales data from on-premise SQL Server to cloud-based analytics using Azure services (Data Factory, Data Lake Storage Gen2, Databricks, Synapse Analytics, and Power BI).

Language: Jupyter Notebook - Size: 2.15 MB - Last synced at: about 1 month ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

Alekssuwinska/sales-data-analysis

Analysis of sales data of a Toy Manufacturer performed using DataBricks.

Language: HTML - Size: 169 KB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

dev-vivekkumarverma/pyspark-databricks

spark, databricks, kafka, batch and stream-processing

Language: Jupyter Notebook - Size: 11.4 MB - Last synced at: about 1 month ago - Pushed at: 3 months ago - Stars: 1 - Forks: 0

jihyeonseong/ESG-AI-investment-by-streamlit

ESG-investment AI

Language: Jupyter Notebook - Size: 32.6 MB - Last synced at: 20 days ago - Pushed at: 6 months ago - Stars: 28 - Forks: 7

Vivi-Figueiredo/ETL-Databricks

Processo de ETL (Extract, Transform, Load) no Databricks com extração de dados via API.

Language: Jupyter Notebook - Size: 0 Bytes - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 1 - Forks: 0

VirginioC/analisi-wikipedia

Analisi esplorativa degli articoli di Wikipedia e modello di Machine Learning per la classificazione dei nuovi articoli in 15 categorie tematiche utilizzando PySpark, Spark SQL e MLlib su Databricks.

Size: 3.55 MB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

ricardolsmendes/fine-grained-demand-forecasting Fork of databricks-industry-solutions/fine-grained-demand-forecasting

Customized approach to the Databricks Fine-grained Demand Forecasting accelerator, adapted for the Medallion Architecture and Unity Catalog

Language: R - Size: 54.7 KB - Last synced at: about 2 months ago - Pushed at: about 2 months ago - Stars: 0 - Forks: 0

data-platform-hq/terraform-databricks-unity-catalog

Terraform module for creation of new or managemet of already existing Databricks Unity Catalog Metastore

Language: HCL - Size: 70.3 KB - Last synced at: 3 days ago - Pushed at: 2 months ago - Stars: 2 - Forks: 2

rohit180497/NBFI-Loan-Repayment

This project aims to build an end-to-end loan default prediction system for a Non-Banking Financial Institution (NBFI). The system is designed to ingest, clean, process, and predict loan default probabilities while ensuring model deployment, monitoring, and automated CI/CD.

Language: HTML - Size: 106 MB - Last synced at: about 1 month ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

tom474/data_pipeline_with_databricks

[RMIT 2024C] EEET2574 - Big Data for Engineering - MongoDB and Spark

Language: Jupyter Notebook - Size: 15.9 MB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 1 - Forks: 0

thanaphongK37/Data-Science-and-Data-Analyst-Project

Portfolio Data Analysis and Data Science projects and Data Engineer built using Azure Service, SQL and Python.

Language: Jupyter Notebook - Size: 3.92 MB - Last synced at: 29 days ago - Pushed at: 2 months ago - Stars: 1 - Forks: 0

Karimangfn/Databricks-Intro-Basics

Repositório de Exercicios básicos com o Databricks

Language: Jupyter Notebook - Size: 25.4 KB - Last synced at: 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

santoshshinde2012/medallion-architecture-databrics

Medallion Architecture: Principles and Practical Exploration

Language: Jupyter Notebook - Size: 18.8 MB - Last synced at: 4 days ago - Pushed at: 2 months ago - Stars: 0 - Forks: 1

ndomah/Azure-Medallion-Pipeline

An end-to-end Azure pipeline using Medallion Architecture.

Size: 1.03 MB - Last synced at: about 2 months ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

ndomah/Data-Engineering

Links to data engineering projects and learning materials.

Size: 4.88 KB - Last synced at: 18 days ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

getyourguide/dbq

Run Databricks queries from your terminal or editor

Language: Python - Size: 271 KB - Last synced at: 16 days ago - Pushed at: 2 months ago - Stars: 6 - Forks: 1

abhilash-1/pyspark-project

This is the first project where we worked on apache spark, In this project what we have done is that we downloaded the datasets from KAGGLE where everyone is aware of, we have downloaded loan, customers credit card and transactions datasets . After downloading the datsaets we have cleaned the data . Then after by using new tools and technologies like spark, HDFS, Hive and many more we have executed new use cases on the datasets we have downloaded from kaggle. As we all know apache spark is a framework that can quickly process the very large datsets.

Language: Jupyter Notebook - Size: 1.87 MB - Last synced at: 17 days ago - Pushed at: over 3 years ago - Stars: 17 - Forks: 13

data-platform-hq/terraform-databricks-runtime

Language: HCL - Size: 50.8 KB - Last synced at: 16 days ago - Pushed at: 2 months ago - Stars: 0 - Forks: 0

benc-uk/batcomputer 📦

A working example of DevOps & operationalisation applied to Machine Learning and AI

Language: Python - Size: 17.8 MB - Last synced at: about 2 months ago - Pushed at: over 4 years ago - Stars: 14 - Forks: 4