Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub / IBM / data-prep-kit

Open source project for data preparation of LLM application builders

JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/IBM%2Fdata-prep-kit

Stars: 34
Forks: 16
Open Issues: 52

License: apache-2.0
Language: Python
Repo Size: 129 MB
Dependencies: 70

Created: 2 months ago
Updated: 6 days ago
Last pushed: 7 days ago
Last synced: 7 days ago

Topics: data, data-prep, data-preparation, data-preprocessing, data-preprocessing-pipelines, finetuning, llm, llmapps

Files

Readme

Dependencies

.github/workflows/build-images.yml actions

actions/checkout v4 composite

.github/workflows/build-library.yml actions

actions/checkout v4 composite

.github/workflows/deploy-docs.yml actions

actions/checkout v2 composite
actions/setup-python v2 composite

.github/workflows/deploy-library.yml actions

actions/checkout v4 composite
actions/download-artifact v4 composite
pypa/gh-action-pypi-publish release/v1 composite

.github/workflows/deploy-transforms.yml actions

actions/checkout v4 composite

.github/workflows/test.yml actions

actions/checkout v4 composite

kfp/kfp_ray_components/Dockerfile docker

docker.io/rayproject/ray 2.9.3-py310 build

tools/ingest2parquet/Dockerfile docker

docker.io/rayproject/ray 2.9.3-py310 build

kfp/kfp_ray_components/requirements.txt pypi

data-prep-lab-kfp ==0.1.6
scipy ==1.13.0

kfp/kfp_support_lib/pyproject.toml pypi

data-prep-lab ==0.1.5
kfp ==1.8.22
requests *

tools/ingest2parquet/requirements.txt pypi

pandas *
parameterized *

transforms/code/code_quality/ray/Dockerfile docker

docker.io/rayproject/ray 2.9.3-py310 build

transforms/code/malware/ray/Dockerfile docker

base latest build
docker.io/clamav/clamav latest build
docker.io/rayproject/ray 2.9.3-py310 build

transforms/code/proglang_select/ray/Dockerfile docker

docker.io/rayproject/ray 2.9.3-py310 build

transforms/universal/doc_id/ray/Dockerfile docker

docker.io/rayproject/ray 2.9.3-py310 build

transforms/universal/ededup/ray/Dockerfile docker

docker.io/rayproject/ray 2.9.3-py310 build

transforms/universal/fdedup/ray/Dockerfile docker

docker.io/rayproject/ray 2.9.3-py310 build

transforms/universal/filter/ray/Dockerfile docker

docker.io/rayproject/ray 2.9.3-py310 build

transforms/universal/noop/ray/Dockerfile docker

docker.io/rayproject/ray 2.9.3-py310 build

transforms/universal/tokenization/ray/Dockerfile docker

docker.io/rayproject/ray 2.9.3-py310 build

data-processing-lib/ray/pyproject.toml pypi

argparse *
boto3 ==1.34.69
fastapi >=0.109.1
mmh3 *
pillow >=10.2.0
pyarrow ==15.0.2
ray [default]==2.9.3

examples/requirements.txt pypi

transforms/code/code_quality/ray/requirements.txt pypi

bs4 ==0.0.2
transformers ==4.38.2

transforms/code/malware/ray/requirements.txt pypi

clamd ==1.0.2

transforms/code/proglang_select/ray/requirements.txt pypi

transforms/universal/doc_id/ray/requirements.txt pypi

transforms/universal/ededup/ray/requirements.txt pypi

mmh3 *
tqdm ==4.64.0
xxhash *

transforms/universal/fdedup/ray/requirements.txt pypi

mmh3 *
scipy *
tqdm ==4.64.0
xxhash *

transforms/universal/tokenization/ray/requirements.txt pypi

transformers ==4.38.0

data-processing-lib/spark/Dockerfile docker

eclipse-temurin ${OPENJRE_TAG} build
python 3.11.9-slim-bookworm build

transforms/universal/doc_id/spark/Dockerfile docker

${BASE_IMAGE} latest build

transforms/universal/filter/python/Dockerfile docker

docker.io/python 3.10.14-slim-bullseye build

transforms/universal/filter/spark/Dockerfile docker

${BASE_IMAGE} latest build

transforms/universal/noop/python/Dockerfile docker

docker.io/python 3.10.14-slim-bullseye build

transforms/universal/noop/spark/Dockerfile docker

${BASE_IMAGE} latest build

data-processing-lib/python/pyproject.toml pypi

argparse *
boto3 ==1.34.69
mmh3 *
pyarrow ==15.0.2

data-processing-lib/spark/pyproject.toml pypi

data-prep-toolkit ==0.2.0.dev6
pyspark >=3.5.1
pyyaml >=6.0.1

transforms/universal/doc_id/spark/requirements.txt pypi

transforms/universal/filter/python/pyproject.toml pypi

data-prep-toolkit ==0.2.0.dev6
duckdb ==0.10.1

transforms/universal/filter/ray/pyproject.toml pypi

data-prep-toolkit-ray ==0.2.0.dev6
dpk-filter-transform-python ==0.9.0.dev6

transforms/universal/filter/spark/requirements.txt pypi

transforms/universal/noop/python/pyproject.toml pypi

data-prep-toolkit ==0.2.0.dev6

transforms/universal/noop/ray/pyproject.toml pypi

data-prep-toolkit-ray ==0.2.0.dev6
dpk-noop-transform-python ==0.9.0.dev6

transforms/universal/noop/spark/pyproject.toml pypi

data-prep-toolkit-spark ==0.2.0.dev6
dpk-noop-transform-python ==0.9.0.dev6