simon987/od-database-crawler

Fork 0

mirror of https://github.com/terorie/od-database-crawler.git synced 2025-04-10 14:06:42 +00:00

Go to file

Richard Patel 1625d6c888

Bump to v1.0.2

2018-11-18 18:53:57 +01:00

ds/redblackhash

Fix ton of bugs

2018-11-17 04:18:22 +01:00

fasturl

Ignore FTP URLs

2018-11-17 04:50:59 +01:00

.gitignore

Rename project

2018-11-17 01:33:15 +01:00

config.go

Add log file support

2018-11-18 14:46:52 +01:00

config.yml

Add log file support

2018-11-18 14:46:52 +01:00

crawl.go

Fix crawl loop

2018-11-18 18:45:06 +01:00

errors.go

Unescape results & don't recrawl 404

2018-11-17 01:21:20 +01:00

LICENSE

Add LICENSE

2018-11-05 21:42:59 +01:00

main.go

Bump to v1.0.2

2018-11-18 18:53:57 +01:00

model.go

Fix ton of bugs

2018-11-17 04:18:22 +01:00

README.md

Update README.md

2018-11-17 01:36:07 +01:00

release.sh

Add freebsd to release.sh

2018-11-18 14:38:18 +01:00

scheduler.go

Refactor uploading & chunk size parameter

2018-11-18 00:19:43 +01:00

server.go

Retry /task/upload

2018-11-18 18:33:26 +01:00

stats.go

Fix ton of bugs

2018-11-17 04:18:22 +01:00

util.go

Add log file support

2018-11-18 14:46:52 +01:00

worker.go

Ignore all HTTP errors

2018-11-18 14:25:06 +01:00

README.md

od-database Go crawler 🚀

by terorie 2018 :P

Crawler for OD-Database
Crawls HTTP open directories (standard Web Server Listings)
Gets name, path, size and modification time of all files
Lightweight and fast: over 9000 requests per second on a standard laptop