Ecosyste.ms: Repos

An open API service providing repository metadata for many open source software ecosystems.

GitHub / ukwa / webarchive-discovery

WARC and ARC indexing and discovery tools.

JSON API: https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/ukwa%2Fwebarchive-discovery

Stars: 113
Forks: 24
Open Issues: 94

License: None
Language: Java
Repo Size: 13 MB
Dependencies: 89

Created: over 11 years ago
Updated: about 2 months ago
Last pushed: about 2 months ago
Last synced: 20 days ago

Commit Stats

Commits: 1358
Authors: 29
Mean commits per author: 46.83
Development Distribution Score: 0.317
More commit stats: https://commits.ecosyste.ms/hosts/GitHub/repositories/ukwa/webarchive-discovery

Files

Loading...

Readme

Loading...

Dependencies

digipres-tika/pom.xml maven

com.github.stephenc.java-iso-tools:loopy-core 1.2.2 compile
bouncycastle:bcmail-jdk16 140
bouncycastle:bcprov-jdk16 140
com.google.guava:guava 18.0
com.itextpdf:itextpdf 5.5.12
eu.scape-project.nanite:nanite-core 1.4.1-97
info.picocli:picocli 4.5.2
org.apache.commons:commons-imaging 1.0-alpha1
org.apache.tika:tika-core 1.24.1
org.apache.tika:tika-langdetect 1.24.1
org.apache.tika:tika-parsers 1.24.1
org.apache.xmlgraphics:xmlgraphics-commons 2.6
org.bouncycastle:bctsp-jdk16 1.46
org.netpreserve.commons:webarchive-commons ${webarchive.commons.version}
org.netpreserve.openwayback:openwayback-core 2.4.0
junit:junit 4.13.1 test
org.slf4j:slf4j-simple 1.7.30 test

pom.xml maven

org.slf4j:jcl-over-slf4j 1.5.11
org.slf4j:slf4j-api 1.5.11

warc-analysers-oscar4/pom.xml maven

uk.bl.wa.discovery:warc-indexer 3.2.0-SNAPSHOT provided
uk.ac.cam.ch.wwmm.oscar:oscar4-api 4.2.2

warc-hadoop-crawler-utils/pom.xml maven

org.archive.heritrix:heritrix-commons 3.2.0
org.archive.heritrix:heritrix-modules 3.2.0
uk.bl.wa.discovery:warc-hadoop-recordreaders 2.2.0-BETA-6-SNAPSHOT

warc-hadoop-indexer/pom.xml maven

org.apache.hadoop:hadoop-core ${hadoop.version} provided
log4j:log4j 1.2.17
org.apache.solr:solr-solrj ${solr.version}
org.slf4j:slf4j-log4j12 ${slf4j.version.override}
uk.bl.wa.discovery:warc-hadoop-recordreaders 3.2.0-SNAPSHOT
uk.bl.wa.discovery:warc-indexer 3.2.0-SNAPSHOT
org.apache.hadoop:hadoop-test ${hadoop.version} test
org.apache.mrunit:mrunit 0.9.0-incubating test
uk.bl.wa.discovery:warc-hadoop-recordreaders 3.2.0-SNAPSHOT test

warc-hadoop-recordreaders/pom.xml maven

org.apache.hadoop:hadoop-core ${hadoop.version} provided
log4j:log4j 1.2.17
org.apache.solr:solr-solrj ${solr.version}
org.netpreserve.commons:webarchive-commons ${webarchive.commons.version}
org.netpreserve.openwayback:openwayback-core 2.4.0
org.slf4j:slf4j-log4j12 ${slf4j.version.override}
junit:junit 4.13.1 test
org.apache.hadoop:hadoop-test ${hadoop.version} test
org.apache.mrunit:mrunit 0.9.0-incubating test

warc-indexer/pom.xml maven

com.fasterxml.jackson.core:jackson-annotations ${jackson.version}
com.fasterxml.jackson.core:jackson-core ${jackson.version}
com.fasterxml.jackson.core:jackson-databind ${jackson.version}
com.google.guava:guava 18.0
com.typesafe:config 1.0.2
commons-codec:commons-codec 1.8
commons-io:commons-io 2.7
commons-lang:commons-lang 2.6
net.sf.opencsv:opencsv 2.3
org.apache.logging.log4j:log4j-1.2-api 2.17.1
org.apache.logging.log4j:log4j-core 2.17.1
org.apache.lucene:lucene-core 8.7.0
org.apache.pdfbox:preflight 2.0.21
org.apache.solr:solr-core ${solr.version}
org.apache.solr:solr-solrj ${solr.version}
org.brotli:dec 0.1.2
org.jdom:jdom 1.1
org.jsoup:jsoup 1.14.2
org.netpreserve.commons:webarchive-commons ${webarchive.commons.version}
org.netpreserve.openwayback:openwayback-core 2.4.0
org.opensearch.client:opensearch-rest-high-level-client 1.1.0
org.opensearch:opensearch 1.1.0
uk.bl.wa.bitwiser:bitwiser 0.0.2
uk.bl.wa.discovery:digipres-tika 3.2.0-SNAPSHOT
uk.bl.wa.sentimentalj:sentimentalj 1.0.2
xerces:xercesImpl 2.12.2
javax.servlet:servlet-api 2.5 test
junit:junit 4.13.1 test

warc-nlp/pom.xml maven

uk.bl.wa.discovery:warc-hadoop-indexer 3.2.0-SNAPSHOT provided
uk.bl.wa.discovery:warc-indexer 3.2.0-SNAPSHOT provided
edu.stanford.nlp:stanford-corenlp 4.4.0
org.deeplearning4j:deeplearning4j-nlp 1.0.0-beta7
org.deeplearning4j:deeplearning4j-ui 1.0.0-beta7
org.nd4j:nd4j-native 1.0.0-beta7

warc-openimaj/pom.xml maven

uk.bl.wa.discovery:warc-indexer 3.2.0-SNAPSHOT provided
org.openimaj:faces 1.3.10

warc-weka/pom.xml maven

weka:weka 3.5.7

.github/workflows/ci-build-and-push.yml actions

actions/checkout v2 composite
actions/setup-java v2 composite
docker/build-push-action v2 composite
docker/login-action v1 composite
docker/metadata-action v3 composite

Dockerfile docker

maven 3-jdk-8 build
openjdk 8-jre build

warc-indexer/src/main/opensearch/os1/docker-compose.yml docker

opensearchproject/opensearch 1.1.0
opensearchproject/opensearch-dashboards 1.1.0

warc-indexer/src/main/solr/Dockerfile docker

solr 6 build