simon987/od-database-crawler

Fork 0

mirror of https://github.com/terorie/od-database-crawler.git synced 2025-04-10 05:56:42 +00:00

Go to file

Richard Patel b1bf59adef

Add The Eye DB to README.md

2018-11-27 17:40:12 +01:00

ds/redblackhash

Fix ton of bugs

2018-11-17 04:18:22 +01:00

fasturl

Ignore FTP URLs

2018-11-17 04:50:59 +01:00

.gitignore

Add output files to .gitignore

2018-11-20 03:51:42 +01:00

config.go

Add job buffer size parameter

2018-11-20 03:42:32 +01:00

config.yml

Add job buffer size parameter

2018-11-20 03:42:32 +01:00

crawl.go

Add TCP timeout option

2018-11-20 03:29:10 +01:00

errors.go

Unescape results & don't recrawl 404

2018-11-17 01:21:20 +01:00

LICENSE

Add LICENSE

2018-11-05 21:42:59 +01:00

main.go

Bump to v1.1.0

2018-11-20 03:46:36 +01:00

model.go

Performance patch

2018-11-20 02:33:50 +01:00

queue.go

Add job buffer size parameter

2018-11-20 03:42:32 +01:00

README.md

Add The Eye DB to README.md

2018-11-27 17:40:12 +01:00

release.sh

Add freebsd to release.sh

2018-11-18 14:38:18 +01:00

scheduler.go

Delete URL queues after crawling

2018-11-20 03:05:43 +01:00

server.go

Bump the upload retry interval up to 30s

2018-11-20 04:13:20 +01:00

stats.go

Fix queue_count in log

2018-11-20 02:49:03 +01:00

util.go

Add log file support

2018-11-18 14:46:52 +01:00

worker.go

Performance patch

2018-11-20 02:33:50 +01:00

README.md

od-database Go crawler 🚀

by terorie 2018 :P

Crawler for OD-Database
Crawls HTTP open directories (standard Web Server Listings)
Gets name, path, size and modification time of all files
Lightweight and fast: over 9000 requests per second on a standard laptop

https://od-db.the-eye.eu/