simon987/od-database-crawler

Fork 0

mirror of https://github.com/terorie/od-database-crawler.git synced 2025-12-12 14:58:51 +00:00

Go to file

Richard Patel 50952791c5 Almost done

2018-11-16 03:12:26 +01:00

.github

Update README.md

2018-11-06 02:41:20 +01:00

ds/redblackhash

Performance improvements

2018-11-06 00:34:22 +01:00

fasturl

Almost done

2018-11-16 03:12:26 +01:00

.gitignore

Refactor a bit

2018-10-28 13:43:45 +01:00

config.go

Timeout and results saving

2018-11-15 20:14:31 +01:00

config.yml

Timeout and results saving

2018-11-15 20:14:31 +01:00

crawl.go

Less tokenizer allocations

2018-11-16 00:22:40 +01:00

errors.go

Detect directory symlinks

2018-10-28 18:37:18 +01:00

LICENSE

Add LICENSE

2018-11-05 21:42:59 +01:00

main.go

Bump version

2018-11-16 00:25:11 +01:00

model.go

Timeout and results saving

2018-11-15 20:14:31 +01:00

README.md

Update README.md

2018-11-06 02:41:20 +01:00

scheduler.go

Timeout and results saving

2018-11-15 20:14:31 +01:00

server.go

Refactor a bit

2018-10-28 13:43:45 +01:00

stats.go

Add resource stats logging

2018-11-05 22:41:17 +01:00

tasks.go

Bits of ODDB API

2018-10-27 04:10:08 +02:00

util.go

Add resource stats logging

2018-11-05 22:41:17 +01:00

worker.go

Timeout and results saving

2018-11-15 20:14:31 +01:00

README.md

oddb Go crawler 🚀

by terorie 2018 :P

Crawls HTTP open directories (standard Web Server Listings)
Gets name, path, size and modification time of all files
Soon: Will work as a crawler for OD-Database!

Stress test crawling pandoradir on an average laptop (~10K requests per second, 4 connections):

Memory usage is being optimized :P