Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub / gfelot / DEND-DateLake-Spark

Use of Spark to get data from S3 then wrangle it to make available back in S3 with a better schema

JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/gfelot%2FDEND-DateLake-Spark

Stars: 3
Forks: 0
Open Issues: 24

License: None
Language: Python
Repo Size: 576 KB
Dependencies: 517

Created: almost 5 years ago
Updated: about 4 years ago
Last pushed: over 1 year ago
Last synced: about 1 year ago

Topics: aws, datalake, python3, spark, udacity-data-engineer-nanodegree, udacity-nanodegree

Files

Readme

Dependencies

environment.yaml conda

alabaster 0.7.12.*
anaconda-client 1.7.2.*
anaconda-navigator 1.9.7.*
anaconda-project 0.8.2.*
appnope 0.1.0.*
appscript 1.0.1.*
asn1crypto 0.24.0.*
astroid 2.2.5.*
astropy 3.1.2.*
atomicwrites 1.3.0.*
attrs 19.1.0.*
babel 2.6.0.*
backcall 0.1.0.*
backports 1.0.*
backports.os 0.1.1.*
backports.shutil_get_terminal_size 1.0.0.*
beautifulsoup4 4.7.1.*
bitarray 0.9.0.*
bkcharts 0.2.*
blaze 0.11.3.*
bleach 3.1.0.*
blinker 1.4.*
blosc 1.16.3.*
bokeh 1.1.0.*
boto 2.49.0.*
boto3 1.9.134.*
botocore 1.12.134.*
bottleneck 1.2.1.*
bz2file 0.98.*
bzip2 1.0.6.*
ca-certificates 2019.3.9.*
cassandra-driver 3.11.0.*
certifi 2019.3.9.*
cffi 1.12.3.*
chardet 3.0.4.*
click 7.0.*
cloudpickle 0.8.1.*
clyent 1.2.2.*
colorama 0.4.1.*
conda 4.6.14.*
conda-build 3.17.8.*
conda-verify 3.1.1.*
contextlib2 0.5.5.*
cryptography 2.6.1.*
cycler 0.10.0.*
cython 0.29.7.*
cytoolz 0.9.0.1.*
dask 1.2.0.*
dask-core 1.2.0.*
datashape 0.5.4.*
dbus 1.13.6.*
decorator 4.4.0.*
defusedxml 0.6.0.*
distributed 1.27.0.*
docutils 0.14.*
entrypoints 0.3.*
et_xmlfile 1.0.1.*
expat 2.2.5.*
fastcache 1.0.2.*
filelock 3.0.10.*
flask 1.0.2.*
flask-cors 3.0.7.*
freetype 2.10.0.*
future 0.17.1.*
gensim 3.4.0.*
gettext 0.19.8.1.*
gevent 1.4.0.*
glib 2.58.3.*
glob2 0.6.*
gmp 6.1.2.*
gmpy2 2.0.8.*
greenlet 0.4.15.*
h5py 2.9.0.*
hdf5 1.10.4.*
heapdict 1.0.0.*
html5lib 1.0.1.*
icu 58.2.*
idna 2.8.*
imageio 2.5.0.*
imagesize 1.1.0.*
importlib_metadata 0.11.*
intel-openmp 2019.3.*
ipykernel 5.1.0.*
ipython 7.5.0.*
ipython-sql 0.3.9.*
ipython_genutils 0.2.0.*
ipywidgets 7.4.2.*
isort 4.3.17.*
itsdangerous 1.1.0.*
jdcal 1.4.*
jedi 0.13.3.*
jinja2 2.10.1.*
jmespath 0.9.4.*
jpeg 9c.*
jsonschema 3.0.1.*
jupyter 1.0.0.*
jupyter_client 5.2.4.*
jupyter_console 6.0.0.*
jupyter_core 4.4.0.*
jupyterlab 0.35.5.*
jupyterlab_server 0.2.0.*
keyring 18.0.0.*
kiwisolver 1.0.1.*
krb5 1.16.3.*
lazy-object-proxy 1.3.1.*
libarchive 3.3.3.*
libblas 3.8.0.*
libcblas 3.8.0.*
libcurl 7.64.1.*
libcxx 4.0.1.*
libcxxabi 4.0.1.*
libedit 3.1.20181209.*
libev 4.25.*
libffi 3.2.1.*
libgfortran 3.0.1.*
libiconv 1.15.*
liblapack 3.8.0.*
liblief 0.9.0.*
libpng 1.6.37.*
libpq 11.3.*
libsodium 1.0.16.*
libssh2 1.8.2.*
libtiff 4.0.10.*
libxml2 2.9.9.*
libxslt 1.1.32.*
llvmlite 0.28.0.*
locket 0.2.0.*
lxml 4.3.3.*
lz4-c 1.8.3.*
lzo 2.10.*
markupsafe 1.1.1.*
matplotlib 3.0.3.*
matplotlib-base 3.0.3.*
mccabe 0.6.1.*
mistune 0.8.4.*
mkl 2019.3.*
mkl_fft 1.0.13.*
mkl_random 1.0.4.*
mock 3.0.5.*
more-itertools 7.0.0.*
mpc 1.1.0.*
mpfr 4.0.2.*
mpmath 1.1.0.*
msgpack-python 0.6.1.*
multipledispatch 0.6.0.*
navigator-updater 0.2.1.*
nbconvert 5.4.1.*
nbformat 4.4.0.*
ncurses 6.1.*
networkx 2.3.*
nltk 3.4.1.*
nose 1.3.7.*
notebook 5.7.8.*
numba 0.43.1.*
numexpr 2.6.9.*
numpy 1.16.3.*
numpydoc 0.9.1.*
oauthlib 3.0.1.*
odo 0.5.1.*
olefile 0.46.*
openblas 0.3.6.*
openpyxl 2.6.2.*
openssl 1.1.1b.*
packaging 19.0.*
pandas 0.24.2.*
pandoc 2.7.2.*
pandocfilters 1.4.2.*
parso 0.4.0.*
partd 0.3.10.*
path.py 12.0.1.*
pathlib2 2.3.3.*
patsy 0.5.1.*
pcre 8.41.*
pep8 1.7.1.*
pexpect 4.7.0.*
pickleshare 0.7.5.*
pillow 6.0.0.*
pip 19.1.*
pkginfo 1.5.0.1.*
pluggy 0.9.0.*
ply 3.11.*
prettytable 0.7.2.*
prometheus_client 0.6.0.*
prompt_toolkit 2.0.9.*
psutil 5.6.1.*
psycopg2 2.7.6.1.*
ptyprocess 0.6.0.*
py 1.8.0.*
py-lief 0.9.0.*
pycodestyle 2.5.0.*
pycosat 0.6.3.*
pycparser 2.19.*
pycrypto 2.6.1.*
pycurl 7.43.0.2.*
pyflakes 2.1.1.*
pygments 2.3.1.*
pyjwt 1.7.1.*
pylint 2.3.1.*
pyodbc 4.0.26.*
pyopenssl 19.0.0.*
pyparsing 2.4.0.*
pyqt 5.9.2.*
pyrsistent 0.14.11.*
pysocks 1.6.8.*
pytables 3.5.1.*
pytest 4.4.1.*
pytest-arraydiff 0.3.*
pytest-astropy 0.5.0.*
pytest-doctestplus 0.3.0.*
pytest-openfiles 0.3.2.*
pytest-remotedata 0.3.1.*
python 3.6.7.*
python-crfsuite 0.9.6.*
python-dateutil 2.8.0.*
python-libarchive-c 2.8.*
python.app 1.2.*
pytz 2019.1.*
pywavelets 1.0.3.*
pyyaml 5.1.*
pyzmq 18.0.0.*
qt 5.9.7.*
qtawesome 0.5.7.*
qtconsole 4.4.3.*
qtpy 1.7.0.*
readline 7.0.*
requests 2.21.0.*
requests-oauthlib 1.2.0.*
rope 0.14.0.*
ruamel_yaml 0.15.71.*
s3transfer 0.2.0.*
scikit-image 0.15.0.*
scikit-learn 0.20.3.*
scipy 1.2.1.*
seaborn 0.9.0.*
send2trash 1.5.0.*
setuptools 41.0.1.*
simplegeneric 0.8.1.*
singledispatch 3.4.0.3.*
sip 4.19.8.*
six 1.12.0.*
smart_open 1.8.3.*
snowballstemmer 1.2.1.*
sortedcollections 1.1.2.*
sortedcontainers 2.1.0.*
soupsieve 1.8.*
sphinx 2.0.1.*
sphinxcontrib-applehelp 1.0.1.*
sphinxcontrib-devhelp 1.0.1.*
sphinxcontrib-htmlhelp 1.0.2.*
sphinxcontrib-jsmath 1.0.1.*
sphinxcontrib-qthelp 1.0.2.*
sphinxcontrib-serializinghtml 1.1.3.*
sphinxcontrib-websupport 1.1.0.*
spyder 3.3.4.*
spyder-kernels 0.4.4.*
sqlalchemy 1.3.3.*
sqlite 3.28.0.*
sqlparse 0.3.0.*
statsmodels 0.9.0.*
sympy 1.4.*
tblib 1.3.2.*
terminado 0.8.2.*
testpath 0.4.2.*
tk 8.6.9.*
toolz 0.9.0.*
tornado 6.0.2.*
tqdm 4.31.1.*
traitlets 4.3.2.*
twython 3.7.0.*
typed-ast 1.3.4.*
unicodecsv 0.14.1.*
unixodbc 2.3.7.*
urllib3 1.24.2.*
wcwidth 0.1.7.*
webencodings 0.5.1.*
werkzeug 0.15.2.*
wheel 0.33.4.*
widgetsnbextension 3.4.2.*
wrapt 1.11.1.*
wurlitzer 1.0.2.*
xlrd 1.2.0.*
xlsxwriter 1.1.7.*
xlwings 0.15.5.*
xlwt 1.2.0.*
xz 5.2.4.*
yaml 0.1.7.*
zeromq 4.2.5.*
zict 0.1.4.*
zipp 0.3.3.*
zlib 1.2.11.*
zstd 1.3.3.*

requirements.txt pypi

Babel ==2.6.0
Bottleneck ==1.2.1
Click ==7.0
Cython ==0.29.7
Flask ==1.0.2
Flask-Cors ==3.0.7
Jinja2 ==2.10.1
MarkupSafe ==1.1.1
Pillow ==6.0.0
PyJWT ==1.7.1
PySocks ==1.6.8
PyWavelets ==1.0.3
PyYAML ==5.1
Pygments ==2.3.1
QtAwesome ==0.5.7
QtPy ==1.7.0
SQLAlchemy ==1.3.3
Send2Trash ==1.5.0
Sphinx ==2.0.1
Werkzeug ==0.15.2
XlsxWriter ==1.1.7
alabaster ==0.7.12
anaconda-client ==1.7.2
anaconda-navigator ==1.9.7
anaconda-project ==0.8.2
appnope ==0.1.0
appscript ==1.0.1
asn1crypto ==0.24.0
astroid ==2.2.5
astropy ==3.1.2
atomicwrites ==1.3.0
attrs ==19.1.0
backcall ==0.1.0
backports.os ==0.1.1
backports.shutil-get-terminal-size ==1.0.0
beautifulsoup4 ==4.7.1
bitarray ==0.9.0
bkcharts ==0.2
blaze ==0.11.3
bleach ==3.1.0
blinker ==1.4
bokeh ==1.1.0
boto ==2.49.0
boto3 ==1.9.134
botocore ==1.12.134
bz2file ==0.98
cassandra-driver ==3.11.0
certifi ==2019.3.9
cffi ==1.12.3
chardet ==3.0.4
cloudpickle ==0.8.1
clyent ==1.2.2
colorama ==0.4.1
conda ==4.6.14
conda-build ==3.17.8
conda-verify ==3.1.1
contextlib2 ==0.5.5
cryptography ==2.6.1
cycler ==0.10.0
cytoolz ==0.9.0.1
dask ==1.2.0
datashape ==0.5.4
decorator ==4.4.0
defusedxml ==0.6.0
distributed ==1.27.0
docutils ==0.14
entrypoints ==0.3
et-xmlfile ==1.0.1
fastcache ==1.0.2
filelock ==3.0.10
future ==0.17.1
gensim ==3.4.0
gevent ==1.4.0
glob2 ==0.6
gmpy2 ==2.0.8
greenlet ==0.4.15
h5py ==2.9.0
heapdict ==1.0.0
html5lib ==1.0.1
idna ==2.8
imageio ==2.5.0
imagesize ==1.1.0
importlib-metadata ==0.0.0
ipykernel ==5.1.0
ipython ==7.5.0
ipython-genutils ==0.2.0
ipython-sql ==0.3.9
ipywidgets ==7.4.2
isort ==4.3.17
itsdangerous ==1.1.0
jdcal ==1.4
jedi ==0.13.3
jmespath ==0.9.4
jsonschema ==3.0.1
jupyter-client ==5.2.4
jupyter-console ==6.0.0
jupyter-core ==4.4.0
jupyterlab ==0.35.5
jupyterlab-server ==0.2.0
keyring ==18.0.0
kiwisolver ==1.0.1
lazy-object-proxy ==1.3.1
libarchive-c ==2.8
lief ==0.9.0
llvmlite ==0.28.0
locket ==0.2.0
lxml ==4.3.3
matplotlib ==3.0.3
mccabe ==0.6.1
mistune ==0.8.4
mkl-fft ==1.0.13
mkl-random ==1.0.4
mock ==3.0.5
more-itertools ==7.0.0
mpmath ==1.1.0
msgpack ==0.6.1
multipledispatch ==0.6.0
navigator-updater ==0.2.1
nbconvert ==5.4.1
nbformat ==4.4.0
networkx ==2.3
nltk ==3.4.1
nose ==1.3.7
notebook ==5.7.8
numba ==0.43.1
numexpr ==2.6.9
numpy ==1.16.3
numpydoc ==0.9.1
oauthlib ==3.0.1
odo ==0.5.1
olefile ==0.46
openpyxl ==2.6.2
packaging ==19.0
pandas ==0.24.2
pandocfilters ==1.4.2
parso ==0.4.0
partd ==0.3.10
path.py ==12.0.1
pathlib2 ==2.3.3
patsy ==0.5.1
pep8 ==1.7.1
pexpect ==4.7.0
pickleshare ==0.7.5
pkginfo ==1.5.0.1
pluggy ==0.9.0
ply ==3.11
prettytable ==0.7.2
prometheus-client ==0.6.0
prompt-toolkit ==2.0.9
psutil ==5.6.1
psycopg2 ==2.7.6.1
ptyprocess ==0.6.0
py ==1.8.0
pyOpenSSL ==19.0.0
pycodestyle ==2.5.0
pycosat ==0.6.3
pycparser ==2.19
pycrypto ==2.6.1
pycurl ==7.43.0.2
pyflakes ==2.1.1
pylint ==2.3.1
pyodbc ==4.0.26
pyparsing ==2.4.0
pyrsistent ==0.14.11
pytest ==4.4.1
pytest-arraydiff ==0.3
pytest-astropy ==0.5.0
pytest-doctestplus ==0.3.0
pytest-openfiles ==0.3.2
pytest-remotedata ==0.3.1
python-crfsuite ==0.9.6
python-dateutil ==2.8.0
pytz ==2019.1
pyzmq ==18.0.0
qtconsole ==4.4.3
requests ==2.21.0
requests-oauthlib ==1.2.0
rope ==0.14.0
ruamel-yaml ==0.15.71
s3transfer ==0.2.0
scikit-image ==0.15.0
scikit-learn ==0.20.3
scipy ==1.2.1
seaborn ==0.9.0
simplegeneric ==0.8.1
singledispatch ==3.4.0.3
six ==1.12.0
smart-open ==1.8.3
snowballstemmer ==1.2.1
sortedcollections ==1.1.2
sortedcontainers ==2.1.0
soupsieve ==1.8
sphinxcontrib-applehelp ==1.0.1
sphinxcontrib-devhelp ==1.0.1
sphinxcontrib-htmlhelp ==1.0.2
sphinxcontrib-jsmath ==1.0.1
sphinxcontrib-qthelp ==1.0.2
sphinxcontrib-serializinghtml ==1.1.3
sphinxcontrib-websupport ==1.1.0
spyder ==3.3.4
spyder-kernels ==0.4.4
sqlparse ==0.3.0
statsmodels ==0.9.0
sympy ==1.4
tables ==3.5.1
tblib ==1.3.2
terminado ==0.8.2
testpath ==0.4.2
toolz ==0.9.0
tornado ==6.0.2
tqdm ==4.31.1
traitlets ==4.3.2
twython ==3.7.0
typed-ast ==1.3.4
unicodecsv ==0.14.1
urllib3 ==1.24.2
wcwidth ==0.1.7
webencodings ==0.5.1
widgetsnbextension ==3.4.2
wrapt ==1.11.1
wurlitzer ==1.0.2
xlrd ==1.2.0
xlwings ==0.15.5
xlwt ==1.2.0
zict ==0.1.4
zipp ==0.3.3