Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub / jstrosch / subcrawl

SubCrawl is a modular framework for discovering open directories, identifying unique content through signatures and organizing the data with optional output modules, such as Elastic.

JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/jstrosch%2Fsubcrawl

Fork of hpthreatresearch/subcrawl
Stars: 36
Forks: 4
Open Issues: 2

License: mit
Language: Python
Repo Size: 12 MB
Dependencies: 26

Created: over 1 year ago
Updated: about 2 months ago
Last pushed: 29 days ago
Last synced: 29 days ago

Topics: crawler, cyber, elastic, opendirectory, threat-hunting, threat-intelligence, yara

Files

Readme

Dependencies

crawler/Dockerfile docker

python 3.8-slim-buster build

crawler/docker-compose.yml docker

confluentinc/cp-kafka latest
confluentinc/cp-zookeeper latest

analysis/requirements.txt pypi

matplotlib ==3.3.4
networkx ==2.5
numpy ==1.20.1
py_tlsh ==4.5.0
pymisp ==2.4.140
requests ==2.22.0

crawler/requirements.txt pypi

Jinja2 ==3.0.1
PyYAML ==5.4.1
beautifulsoup4 ==4.9.3
clamd ==1.0.2
elasticsearch ==7.15.1
falcon ==2.0.0
gunicorn ==20.0.4
kafka-python ==2.0.2
mergedeep ==1.3.4
peewee ==3.14.0
py_tlsh ==4.5.0
pyjarm ==0.0.5
pymisp ==2.4.140
python_magic ==0.4.22
requests ==2.25.0
timeloop ==1.0.2
yara_python ==4.0.5