GitHub / kstrassheim / datawarehouse-crawler

This is a content and schema crawler tool to receive, update and import various kinds of data into a Onprem or Cloud based SQLServer or Azure-Synapse-Analysis (Azure Datawarehouse SQLServer). As source it supports SQLServer Tables, ODATA Endpoints, CSV Files or Excel Files. For multiple sources it can run in parallel mode where it would make a thread for each connection. The speciality of this crawler is that it creates the target tables by himself using the additional info from source.json. In case of Azure-Synapse-Analysis it would estimate the distribution type and keys. The syncing works completely without SQL Transactions by using a consistency correction algorithm for very frequent fact tables. There are 5 Syncing Algorithms (see Manual/Insert) which can be selected as well as one Update Algorithm.

JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/kstrassheim%2Fdatawarehouse-crawler

Topics: azure-data-warehouse, azure-synapse-analytics, business-intelligence, crawler, csv, data-import, data-science, datawarehouse, datawarehousing, docker, dotnet-core-2, excel, integration-testing, odata, parallel-computing, sql

Files