simon987/od-database-crawler
1
0
Fork 0
mirror of https://github.com/terorie/od-database-crawler.git synced 2025-04-16 08:56:44 +00:00
Code Issues Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Richard Patel 2018-11-06 02:41:20 +01:00
parent 0344a120ff
commit 93272e1da1
No known key found for this signature in database
GPG Key ID: C268B2BBDA2ABECB
2 changed files with 11 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
.github/stress.png vendored Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 369 KiB

12
README.md
View File

@ -1,2 +1,12 @@
# oddb Go crawler
# oddb Go crawler 🚀
> by terorie 2018 :P
* Crawls HTTP open directories (standard Web Server Listings)
* Gets name, path, size and modification time of all files
* Soon: Will work as a crawler for [OD-Database](https://github.com/simon987/od-database)!
Stress test crawling [pandoradir](https://github.com/terorie/pandoradir)
on an average laptop (~10K requests per second, 4 connections):
![image](.github/stress.png)
Memory usage is being optimized :P