simon987/od-database-crawler
1
0
Fork 0
mirror of https://github.com/terorie/od-database-crawler.git synced 2025-12-10 22:18:55 +00:00
Code Issues Projects Releases Wiki Activity
83 Commits 12 Branches 8 Tags
dc4e4212a0b2ca6bc1fd5c221f6185b13d0c4899
Go to file
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE
Richard Patel dc4e4212a0 Add freebsd to release.sh
2018-11-18 14:38:18 +01:00
ds/redblackhash
Fix ton of bugs
2018-11-17 04:18:22 +01:00
fasturl
Ignore FTP URLs
2018-11-17 04:50:59 +01:00
.gitignore
Rename project
2018-11-17 01:33:15 +01:00
config.go
Add User-Agent parameter
2018-11-18 14:24:04 +01:00
config.yml
Add User-Agent parameter
2018-11-18 14:24:04 +01:00
crawl.go
Add User-Agent parameter
2018-11-18 14:24:04 +01:00
errors.go
Unescape results & don't recrawl 404
2018-11-17 01:21:20 +01:00
LICENSE
Add LICENSE
2018-11-05 21:42:59 +01:00
main.go
Bump to 1.0
2018-11-18 00:54:58 +01:00
model.go
Fix ton of bugs
2018-11-17 04:18:22 +01:00
README.md
Update README.md
2018-11-17 01:36:07 +01:00
release.sh
Add freebsd to release.sh
2018-11-18 14:38:18 +01:00
scheduler.go
Refactor uploading & chunk size parameter
2018-11-18 00:19:43 +01:00
server.go
Don't call /task/upload for websites with no results
2018-11-18 01:42:57 +01:00
stats.go
Fix ton of bugs
2018-11-17 04:18:22 +01:00
util.go
Add resource stats logging
2018-11-05 22:41:17 +01:00
worker.go
Ignore all HTTP errors
2018-11-18 14:25:06 +01:00

README.md

od-database Go crawler 🚀

by terorie 2018 :P

  • Crawler for OD-Database
  • Crawls HTTP open directories (standard Web Server Listings)
  • Gets name, path, size and modification time of all files
  • Lightweight and fast: over 9000 requests per second on a standard laptop
Reference in New Issue View Git Blame Copy Permalink
Description
No description provided
crawlerfasthttpgogolangod-database
Readme 858 KiB
Languages
Go 99.9%