Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub / hpthreatresearch / subcrawl

SubCrawl is a modular framework for discovering open directories, identifying unique content through signatures and organizing the data with optional output modules, such as MISP.

JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/hpthreatresearch%2Fsubcrawl

Stars: 136
Forks: 32
Open Issues: 4

License: mit
Language: Python
Repo Size: 11.9 MB
Dependencies: 26

Created: about 3 years ago
Updated: 3 months ago
Last pushed: 8 months ago
Last synced: 2 months ago

Topics: opendir, threat-hunting, threat-intelligence

Files

Readme

Dependencies

crawler/Dockerfile docker

python 3.8-slim-buster build

crawler/docker-compose.yml docker

confluentinc/cp-kafka latest
confluentinc/cp-zookeeper latest

analysis/requirements.txt pypi

matplotlib ==3.3.4
networkx ==2.5
numpy ==1.20.1
py_tlsh ==4.5.0
pymisp ==2.4.140
requests ==2.22.0

crawler/requirements.txt pypi

Jinja2 ==3.0.1
PyYAML ==5.4.1
beautifulsoup4 ==4.9.3
clamd ==1.0.2
elasticsearch ==7.15.1
falcon ==2.0.0
gunicorn ==20.0.4
kafka-python ==2.0.2
mergedeep ==1.3.4
peewee ==3.14.0
py_tlsh ==4.5.0
pyjarm ==0.0.5
pymisp ==2.4.140
python_magic ==0.4.22
requests ==2.25.0
timeloop ==1.0.2
yara_python ==4.0.5