Bump github.com/beeker1121/goque

Bumps [github.com/beeker1121/goque](https://github.com/beeker1121/goque) from 2.0.1+incompatible to 2.1.0+incompatible. - [Release notes](https://github.com/beeker1121/goque/releases) - [Commits](https://github.com/beeker1121/goque/compare/v2.0.1...v2.1.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Update config.yml
2025-12-13 15:19:03 +00:00 · 2019-11-05 07:27:14 +00:00 · 2019-04-05 09:30:20 -04:00 · 2019-03-14 10:57:44 +01:00 · 2019-03-12 19:37:06 +01:00 · 2019-03-12 04:48:03 +00:00
5 changed files with 24 additions and 10 deletions
--- a/5
+++ b/5
@@ -2,7 +2,10 @@ FROM golang:alpine as builder
 ADD . /go/src/github.com/terorie/od-database-crawler
 RUN apk add git \
 && go get -d -v github.com/terorie/od-database-crawler \
- && CGO_ENABLED=0 go install -a -installsuffix cgo github.com/terorie/od-database-crawler
+ && CGO_ENABLED=0 go install -a \
+    -installsuffix cgo \
+    -ldflags="-s -w" \
+    github.com/terorie/od-database-crawler

 FROM scratch
 COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
--- a/README.md
+++ b/README.md
@@ -1,11 +1,14 @@
-# od-database Go crawler 🚀
+# OD-Database Crawler 🕷
 [![Build Status](https://travis-ci.org/terorie/od-database-crawler.svg?branch=master)](https://travis-ci.org/terorie/od-database-crawler)
-> by terorie 2018 :P
+[![](https://tokei.rs/b1/github/terorie/od-database-crawler)](https://github.com/terorie/od-database-crawler)
+[![CodeFactor](https://www.codefactor.io/repository/github/terorie/od-database-crawler/badge/master)](https://www.codefactor.io/repository/github/terorie/od-database-crawler/overview/master)

 * Crawler for [__OD-Database__](https://github.com/simon987/od-database)
+ * In production at https://od-db.the-eye.eu/
+ * Over 880 TB actively crawled
 * Crawls HTTP open directories (standard Web Server Listings)
 * Gets name, path, size and modification time of all files
- * Lightweight and fast: __over 9000 requests per second__ on a standard laptop
+ * Lightweight and fast

 https://od-db.the-eye.eu/

--- a/config.yml
+++ b/config.yml
@@ -47,7 +47,7 @@ output:
 # Crawler settings
 crawl:
  # Number of sites that can be processed at once
-  tasks: 100
+  tasks: 25

  # Number of connections per site
  # Please be careful with this setting!
@@ -81,4 +81,4 @@ crawl:
  # in memory.
  # A negative value will cause all jobs
  # to be stored in memory. (Don't do this)
-  job_buffer: 5000
+  job_buffer: -1
--- a/go.mod
+++ b/go.mod
@@ -1,13 +1,13 @@
 module github.com/terorie/od-database-crawler

 require (
-	github.com/beeker1121/goque v2.0.1+incompatible
+	github.com/beeker1121/goque v2.1.0+incompatible
 	github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db // indirect
-	github.com/sirupsen/logrus v1.3.0
+	github.com/sirupsen/logrus v1.4.0
 	github.com/spf13/cobra v0.0.3
-	github.com/spf13/viper v1.3.1
+	github.com/spf13/viper v1.3.2
 	github.com/syndtr/goleveldb v0.0.0-20181128100959-b001fa50d6b2 // indirect
-	github.com/valyala/fasthttp v1.1.0
+	github.com/valyala/fasthttp v1.2.0
 	golang.org/x/crypto v0.0.0-20190131182504-b8fe1690c613
 	golang.org/x/net v0.0.0-20180911220305-26e67e76b6c3
 )
--- a/go.sum
+++ b/go.sum
@@ -1,6 +1,8 @@
 github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
 github.com/beeker1121/goque v2.0.1+incompatible h1:5nJHPMqQLxUvGFc8m/NW2QzxKyc0zICmqs/JUsmEjwE=
 github.com/beeker1121/goque v2.0.1+incompatible/go.mod h1:L6dOWBhDOnxUVQsb0wkLve0VCnt2xJW/MI8pdRX4ANw=
+github.com/beeker1121/goque v2.1.0+incompatible h1:m5pZ5b8nqzojS2DF2ioZphFYQUqGYsDORq6uefUItPM=
+github.com/beeker1121/goque v2.1.0+incompatible/go.mod h1:L6dOWBhDOnxUVQsb0wkLve0VCnt2xJW/MI8pdRX4ANw=
 github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
 github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
 github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
@@ -25,6 +27,8 @@ github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/9
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/sirupsen/logrus v1.3.0 h1:hI/7Q+DtNZ2kINb6qt/lS+IyXnHQe9e90POfeewL/ME=
 github.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+github.com/sirupsen/logrus v1.4.0 h1:yKenngtzGh+cUSSh6GWbxW2abRqhYUSR/t/6+2QqNvE=
+github.com/sirupsen/logrus v1.4.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
 github.com/spf13/afero v1.1.2 h1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=
 github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
 github.com/spf13/cast v1.3.0 h1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=
@@ -37,6 +41,8 @@ github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
 github.com/spf13/viper v1.3.1 h1:5+8j8FTpnFV4nEImW/ofkzEt8VoOiLXxdYIDsB73T38=
 github.com/spf13/viper v1.3.1/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
+github.com/spf13/viper v1.3.2 h1:VUFqw5KcqRf7i70GOzW7N+Q7+gxVBkSSqiXB12+JQ4M=
+github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/syndtr/goleveldb v0.0.0-20181128100959-b001fa50d6b2 h1:GnOzE5fEFN3b2zDhJJABEofdb51uMRNb8eqIVtdducs=
@@ -46,6 +52,8 @@ github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6Kllzaw
 github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
 github.com/valyala/fasthttp v1.1.0 h1:3BohG7mqwj4lq7PTX//7gLbUlzNvZSPmuHFnloXT0lw=
 github.com/valyala/fasthttp v1.1.0/go.mod h1:4vX61m6KN+xDduDNwXrhIAVZaZaZiQ1luJk8LWSxF3s=
+github.com/valyala/fasthttp v1.2.0 h1:dzZJf2IuMiclVjdw0kkT+f9u4YdrapbNyGAN47E/qnk=
+github.com/valyala/fasthttp v1.2.0/go.mod h1:4vX61m6KN+xDduDNwXrhIAVZaZaZiQ1luJk8LWSxF3s=
 github.com/valyala/tcplisten v0.0.0-20161114210144-ceec8f93295a/go.mod h1:v3UYOV9WzVtRmSR+PDvWpU/qWl4Wa5LApYYX4ZtKbio=
 github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
 golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
Author	SHA1	Message	Date
dependabot-preview[bot]	9223edbfc5	Bump github.com/beeker1121/goque Bumps [github.com/beeker1121/goque](https://github.com/beeker1121/goque) from 2.0.1+incompatible to 2.1.0+incompatible. - [Release notes](https://github.com/beeker1121/goque/releases) - [Commits](https://github.com/beeker1121/goque/compare/v2.0.1...v2.1.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>	2019-11-05 07:27:14 +00:00
Simon Fortier	88bf634cb6	Update config.yml	2019-04-05 09:30:20 -04:00
dependabot[bot]	796cf6ac23	Bump github.com/spf13/viper from 1.3.1 to 1.3.2	2019-03-14 10:57:44 +01:00
Richard Patel	defaf54e66	Bump github.com/sirupsen/logrus from 1.3.0 to 1.4.0	2019-03-12 19:37:06 +01:00
dependabot[bot]	230824c58f	Bump github.com/sirupsen/logrus from 1.3.0 to 1.4.0 Bumps [github.com/sirupsen/logrus](https://github.com/sirupsen/logrus) from 1.3.0 to 1.4.0. - [Release notes](https://github.com/sirupsen/logrus/releases) - [Changelog](https://github.com/sirupsen/logrus/blob/master/CHANGELOG.md) - [Commits](https://github.com/sirupsen/logrus/compare/v1.3.0...v1.4.0) Signed-off-by: dependabot[bot] <support@dependabot.com>	2019-03-12 04:48:03 +00:00
terorie	d3c199b738	Update README Add some badges and update description	2019-02-28 23:57:50 +01:00
dependabot[bot]	0b3f0d87fe	Upgrade fasthttp to 1.2.0 Bumps [github.com/valyala/fasthttp](https://github.com/valyala/fasthttp) from 1.1.0 to 1.2.0. - [Release notes](https://github.com/valyala/fasthttp/releases) - [Commits](https://github.com/valyala/fasthttp/compare/v1.1.0...v1.2.0) Thanks bot Signed-off-by: dependabot[bot] <support@dependabot.com>	2019-02-28 22:42:40 +01:00
terorie	da9c75e392	Reduce Docker image size	2019-02-22 21:37:04 +01:00