simon987/od-database-crawler

Fork 0

mirror of https://github.com/terorie/od-database-crawler.git synced 2025-04-10 14:06:42 +00:00

Go to file

Richard Patel 6dbec8c789

Add release script

2018-11-18 02:36:22 +01:00

ds/redblackhash

Fix ton of bugs

2018-11-17 04:18:22 +01:00

fasturl

Ignore FTP URLs

2018-11-17 04:50:59 +01:00

.gitignore

Rename project

2018-11-17 01:33:15 +01:00

config.go

Ignore HTTPS errors

2018-11-18 00:37:30 +01:00

config.yml

Add recheck and timeout parameters

2018-11-18 00:29:29 +01:00

crawl.go

Ignore HTTPS errors

2018-11-18 00:37:30 +01:00

errors.go

Unescape results & don't recrawl 404

2018-11-17 01:21:20 +01:00

LICENSE

Add LICENSE

2018-11-05 21:42:59 +01:00

main.go

Bump to 1.0

2018-11-18 00:54:58 +01:00

model.go

Fix ton of bugs

2018-11-17 04:18:22 +01:00

README.md

Update README.md

2018-11-17 01:36:07 +01:00

release.sh

Add release script

2018-11-18 02:36:22 +01:00

scheduler.go

Refactor uploading & chunk size parameter

2018-11-18 00:19:43 +01:00

server.go

Don't call /task/upload for websites with no results

2018-11-18 01:42:57 +01:00

stats.go

Fix ton of bugs

2018-11-17 04:18:22 +01:00

util.go

Add resource stats logging

2018-11-05 22:41:17 +01:00

worker.go

Grammar

2018-11-17 13:35:29 +01:00

README.md

od-database Go crawler 🚀

by terorie 2018 :P

Crawler for OD-Database
Crawls HTTP open directories (standard Web Server Listings)
Gets name, path, size and modification time of all files
Lightweight and fast: over 9000 requests per second on a standard laptop