mirror of
https://github.com/terorie/od-database-crawler.git
synced 2025-04-20 10:56:44 +00:00
13 lines
463 B
Markdown
13 lines
463 B
Markdown
# oddb Go crawler 🚀
|
|
> by terorie 2018 :P
|
|
|
|
* Crawls HTTP open directories (standard Web Server Listings)
|
|
* Gets name, path, size and modification time of all files
|
|
* Soon: Will work as a crawler for [OD-Database](https://github.com/simon987/od-database)!
|
|
|
|
Stress test crawling [pandoradir](https://github.com/terorie/pandoradir)
|
|
on an average laptop (~10K requests per second, 4 connections):
|
|
|
|
|
|
Memory usage is being optimized :P
|