simon987/od-database-crawler
1
0
Fork 0
mirror of https://github.com/terorie/od-database-crawler.git synced 2025-04-17 09:26:43 +00:00
Code Issues Projects Releases Wiki Activity

Update README.md

Browse Source
This commit is contained in:
Richard Patel 2018-11-17 01:36:07 +01:00
parent 718f9d7fbc
commit 0fe97a8058
No known key found for this signature in database
GPG Key ID: C268B2BBDA2ABECB
2 changed files with 3 additions and 8 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
.github/stress.png vendored
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 369 KiB

11
README.md
View File

@ -1,12 +1,7 @@
# oddb Go crawler 🚀 # od-database Go crawler 🚀
> by terorie 2018 :P > by terorie 2018 :P
* Crawler for [__OD-Database__](https://github.com/simon987/od-database)
* Crawls HTTP open directories (standard Web Server Listings) * Crawls HTTP open directories (standard Web Server Listings)
* Gets name, path, size and modification time of all files * Gets name, path, size and modification time of all files
* Soon: Will work as a crawler for [OD-Database](https://github.com/simon987/od-database)! * Lightweight and fast: __over 9000 requests per second__ on a standard laptop
Stress test crawling [pandoradir](https://github.com/terorie/pandoradir)
on an average laptop (~10K requests per second, 4 connections):
![image](.github/stress.png)
Memory usage is being optimized :P