simon987/od-database-crawler
1
0
Fork 0
mirror of https://github.com/terorie/od-database-crawler.git synced 2025-12-12 14:58:51 +00:00
Code Issues Projects Releases Wiki Activity
32 Commits 12 Branches 8 Tags
hexa
Commit Graph

32 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Simon
2d72ff3402 Micro optimization pt. 2 2018-11-05 17:38:33 -05:00
Simon
1b5e6bb7f4 Micro optimization 2018-11-05 17:08:54 -05:00
Richard Patel
add6581804 Add resource stats logging 2018-11-05 22:41:17 +01:00
Richard Patel
395a6f30b2 Fix pprof 2018-11-05 21:55:07 +01:00
Richard Patel
a4e53053b9 Add LICENSE 
oi m8 got a loicense for that
 2018-11-05 21:42:59 +01:00
Richard Patel
e39565377e Add pprof debug server 2018-11-05 21:39:15 +01:00
Richard Patel
77cb45dbec Detect directory symlinks 2018-10-28 18:37:18 +01:00
Richard Patel
fa37d45378 Remove too many crawler block 
More logging
 2018-10-28 18:17:04 +01:00
Richard Patel
bfd7302be8 Add urfave/cli app 2018-10-28 17:59:46 +01:00
Richard Patel
b1c40767e0 Remember scanned URLs 2018-10-28 17:07:30 +01:00
Richard Patel
c196b6f20d Better config 2018-10-28 14:19:09 +01:00
Richard Patel
ddfdce9d0f Refactor a bit 2018-10-28 13:43:45 +01:00
Richard Patel
7c4ed9d41e Remove WIP disclaimer 2018-10-28 03:48:33 +01:00
Richard Patel
ab5874129f Don't retry on 401/403 2018-10-28 03:47:29 +01:00
Richard Patel
faad19f121 more stuff 2018-10-28 03:41:16 +01:00
Richard Patel
4ea5f8a410 Handle HTTP statuses 2018-10-28 03:22:25 +01:00
Richard Patel
1c33346f45 Fix crawl descent 2018-10-28 03:06:18 +01:00
Richard Patel
a507110787 Add stats interval parameter 2018-10-28 02:47:20 +02:00
Richard Patel
79f540bf29 Scheduler 2018-10-28 02:40:12 +02:00
Richard Patel
5ac9fc10a1 Merge branch 'rewrite' 2018-10-27 17:27:52 +02:00
Richard Patel
941899e304 Merge branch 'queue' 2018-10-27 17:27:36 +02:00
Richard Patel
3fb4d4bde9 More logs 2018-10-27 17:25:32 +02:00
Richard Patel
76c8c13d49 Use finite state machine 2018-10-27 16:55:00 +02:00
Richard Patel
442a2cf8a7 Compare finite state machine and Regex 2018-10-27 16:53:45 +02:00
Richard Patel
9e090d109d Header state machine 2018-10-27 16:29:10 +02:00
Richard Patel
d748be72cd File HEAD requests 2018-10-27 16:22:01 +02:00
Richard Patel
2844d344ec Working listing 2018-10-27 15:00:20 +02:00
Richard Patel
6b73cf75b8 Add .gitignore 2018-10-27 04:13:05 +02:00
Richard Patel
7b98db9e78 WIP disclaimer 2018-10-27 04:12:33 +02:00
Richard Patel
abf069f946 Bits of ODDB API 2018-10-27 04:10:08 +02:00
Richard Patel
f2d2b620fa Simple queue crawler 2018-10-27 04:08:32 +02:00
Richard Patel
dc816146cc Initial commit 2018-10-27 04:07:42 +02:00