simon987/od-database-crawler

Fork 0

mirror of https://github.com/terorie/od-database-crawler.git synced 2025-04-10 05:56:42 +00:00

Go to file

Richard Patel 17ba5583c9 Add .travis.yml

2019-02-02 23:18:03 +01:00

ds/redblackhash

Fix ton of bugs

2018-11-17 04:18:22 +01:00

fasturl

Ignore FTP URLs

2018-11-17 04:50:59 +01:00

.gitignore

Add output files to .gitignore

2018-11-20 03:51:42 +01:00

.travis.yml

Add .travis.yml

2019-02-02 23:18:03 +01:00

config.go

Minor cleanup

2018-12-18 15:31:33 +01:00

config.yml

Wait time control in config

2018-11-27 22:11:57 +01:00

crawl_apache2_test.go

Add parser tests

2018-12-18 15:31:09 +01:00

crawl_nginx_test.go

Add parser tests

2018-12-18 15:31:09 +01:00

crawl_test.go

Benchmark: Reference parser

2018-12-18 15:39:41 +01:00

crawl.go

Add parser tests

2018-12-18 15:31:09 +01:00

errors.go

Unescape results & don't recrawl 404

2018-11-17 01:21:20 +01:00

go.mod

Add go.mod

2019-02-02 23:15:52 +01:00

go.sum

Add go.mod

2019-02-02 23:15:52 +01:00

LICENSE

Add LICENSE

2018-11-05 21:42:59 +01:00

main.go

Bump version to v1.1.1

2018-11-27 22:11:57 +01:00

model.go

Performance patch

2018-11-20 02:33:50 +01:00

queue.go

Add job buffer size parameter

2018-11-20 03:42:32 +01:00

README.md

Add The Eye DB to README.md

2018-11-27 17:40:12 +01:00

release.sh

Add freebsd to release.sh

2018-11-18 14:38:18 +01:00

scheduler.go

Delete URL queues after crawling

2018-11-20 03:05:43 +01:00

server.go

Wait time control in config

2018-11-27 22:11:57 +01:00

stats.go

Minor cleanup

2018-12-18 15:31:33 +01:00

util.go

Add log file support

2018-11-18 14:46:52 +01:00

worker.go

Minor cleanup

2018-12-18 15:31:33 +01:00

README.md

od-database Go crawler 🚀

by terorie 2018 :P

Crawler for OD-Database
Crawls HTTP open directories (standard Web Server Listings)
Gets name, path, size and modification time of all files
Lightweight and fast: over 9000 requests per second on a standard laptop

https://od-db.the-eye.eu/