simon987/od-database-crawler
1
0
Fork 0
mirror of https://github.com/terorie/od-database-crawler.git synced 2025-04-16 08:56:44 +00:00
Code Issues Projects Releases Wiki Activity
od-database-crawler/README.md
Richard Patel 93272e1da1
Update README.md
2018-11-06 02:41:20 +01:00

463 B
Raw Blame History

oddb Go crawler 🚀

by terorie 2018 :P

  • Crawls HTTP open directories (standard Web Server Listings)
  • Gets name, path, size and modification time of all files
  • Soon: Will work as a crawler for OD-Database!

Stress test crawling pandoradir on an average laptop (~10K requests per second, 4 connections): image

Memory usage is being optimized :P