simon987/od-database-crawler

Fork 0

mirror of https://github.com/terorie/od-database-crawler.git synced 2025-12-10 22:18:55 +00:00

Go to file

Richard Patel 4f3140a39f Fix queue_count in log

2018-11-20 02:49:03 +01:00

ds/redblackhash

Fix ton of bugs

2018-11-17 04:18:22 +01:00

fasturl

Ignore FTP URLs

2018-11-17 04:50:59 +01:00

.gitignore

Rename project

2018-11-17 01:33:15 +01:00

config.go

Add log file support

2018-11-18 14:46:52 +01:00

config.yml

Add log file support

2018-11-18 14:46:52 +01:00

crawl.go

Fix crawl loop

2018-11-18 18:45:06 +01:00

errors.go

Unescape results & don't recrawl 404

2018-11-17 01:21:20 +01:00

LICENSE

Add LICENSE

2018-11-05 21:42:59 +01:00

main.go

Job queue disk offloading

2018-11-20 02:03:10 +01:00

model.go

Performance patch

2018-11-20 02:33:50 +01:00

queue.go

Fix queue_count in log

2018-11-20 02:49:03 +01:00

README.md

Update README.md

2018-11-17 01:36:07 +01:00

release.sh

Add freebsd to release.sh

2018-11-18 14:38:18 +01:00

scheduler.go

Fix queue_count in log

2018-11-20 02:49:03 +01:00

server.go

Limit retries to 10

2018-11-18 21:05:26 +01:00

stats.go

Fix queue_count in log

2018-11-20 02:49:03 +01:00

util.go

Add log file support

2018-11-18 14:46:52 +01:00

worker.go

Performance patch

2018-11-20 02:33:50 +01:00

README.md

od-database Go crawler 🚀

by terorie 2018 :P

Crawler for OD-Database
Crawls HTTP open directories (standard Web Server Listings)
Gets name, path, size and modification time of all files
Lightweight and fast: over 9000 requests per second on a standard laptop