simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-04-19 10:26:44 +00:00
Code Issues Projects Releases Wiki Activity
256 Commits 2 Branches 0 Tags
Commit Graph

84 Commits

Author SHA1 Message Date
Simon
2ef60a05a5 proper user agent for pycurl 2018-09-29 11:38:52 -04:00
Simon
8b13de4a6b Re-init curl handle on error 2018-08-25 16:46:53 -04:00
Simon
faa9ac3ccb Closing curl handle manually just to make sure 2018-08-23 12:48:15 -04:00
Simon
dff4125c9f Bugfix post-pycurl update pt. 3 (Sorry!) 2018-08-23 12:47:17 -04:00
Simon
25e1e58828 Bugfix post-pycurl update pt. 3 (Sorry!) 2018-08-23 12:46:00 -04:00
Simon
6ffc43601b Bugfix post-pycurl update pt. 2 2018-08-23 12:40:13 -04:00
Simon
484a0baf9d Bugfix post-pycurl update 2018-08-23 12:37:27 -04:00
Simon
cadaf14c1b Small bugfix 2018-08-23 12:12:23 -04:00
Simon
54b4d2d5b4 removed debug lines 2018-08-23 12:02:07 -04:00
Simon
d42be56dee More debug info 2018-08-23 11:59:23 -04:00
Simon
85c3aa918d replaced requests by pycurl 2018-08-23 11:47:09 -04:00
Simon
cc4c70f400 Request content is read all at once 2018-08-11 13:05:24 -04:00
Simon
458641654c Minimal configuration for reddit comment callback 2018-08-08 21:24:55 -04:00
Simon
5ff198b88a Fix for negative sizes 2018-07-25 11:37:12 -04:00
Simon
34d1f375a8 Crawler performance improvements 2018-07-25 11:27:50 -04:00
Simon
f12d5d524a exceptions during push_result are logged instead of raised 2018-07-21 10:45:17 -04:00
Simon
d43cf3b0ce Empty queue timeout increased to avoid that all workers die before the website is dropped 2018-07-20 14:11:17 -04:00
Simon
d3801adf74 Typo 2018-07-20 13:39:23 -04:00
Simon
1df5d194d2 Very slow websites are skipped. Should fix infinite waiting bug 2018-07-20 13:34:40 -04:00
Simon
004ade8935 Misc bug fixes 2018-07-20 10:35:17 -04:00
Simon
8ef1d36c9d Bug when directory is empty with new file upload 2018-07-17 18:24:05 -04:00
Simon
898ffcf410 File upload is made in small chunks 2018-07-17 17:52:17 -04:00
Simon
55a0fde19d Skip 'Parent directory' links more efficiently 2018-07-17 11:20:58 -04:00
Simon
756e331c83 Fixed bug in crawler when file count in a directory is greater than 150 2018-07-17 11:03:10 -04:00
Simon
112400886e Crawler no longer crashes when website has no files 2018-07-15 10:46:48 -04:00
Simon
e18ded7ac1 Temporarily removed logger in async methods (https://stackoverflow.com/questions/37907350) 2018-07-15 10:35:13 -04:00
Simon
152a6f20fb Re-enabled multi threaded file requests for large directories 2018-07-15 08:54:36 -04:00
Simon
f452d0f8b2 file lists now deleted after indexing 2018-07-14 20:41:20 -04:00
Simon
51a47b3628 Removed debug line 2018-07-14 17:36:16 -04:00
Simon
fe1d29aaea Crawl tasks are now fetched by the crawlers instead of pushed by the server 2018-07-14 17:31:18 -04:00
Simon
d138db8f06 Added filter to check if a website can be scanned from its parent directory 2018-07-10 10:14:23 -04:00
Simon
e384efd403 Bugfix for http crawler 2018-06-25 20:36:31 -04:00
Simon
d7ce1670a8 Logging and bugfix for http crawler 2018-06-25 14:36:16 -04:00
Simon
5fd00f22af Task logs now stored on main server 2018-06-24 20:32:02 -04:00
Simon
f6ee338c0f Removed unused statement 2018-06-24 18:24:41 -04:00
Simon
e11343de23 More FTP crawler bug fixes 2018-06-24 18:05:30 -04:00
Simon
ab35ce96cc FTP crawler bug fixes 2018-06-24 16:44:21 -04:00
Simon
8e937e69c0 Should fix some FTP errors 2018-06-24 13:50:55 -04:00
Simon
a6d753c6ee Added redispatch button and fixed typo in load balancing code 2018-06-24 10:07:46 -04:00
Simon
1ac510ff53 Slots can be updated without removing & adding 2018-06-24 09:39:44 -04:00
Simon
348914aba9 Removing unused module 2018-06-22 17:34:10 -04:00
Simon
e824b2bf3c Updated readme and UI fixes 2018-06-22 13:22:58 -04:00
Simon
7a4432e4d0 More bugfixes for looping directories, some work on task dispatching 2018-06-21 20:50:26 -04:00
Simon
14d384e366 Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default 2018-06-21 19:44:27 -04:00
Simon
098ad2be72 Should fix unknown encoding errors + removed https warnings 2018-06-21 19:23:01 -04:00
Simon
073551df3c Attempt to handle looping directories 2018-06-21 11:54:40 -04:00
Simon
dd93d40a55 Small bugfix for ftp crawler 2018-06-20 21:56:38 -04:00
Simon
c5deafbea5 Should fix some odd http listings 2018-06-20 13:34:41 -04:00
Simon
d8486104b4 Fix for odd html listings 2018-06-19 12:14:50 -04:00
Simon
e54609972c Overwrite document on re-index, update website last_modified on task complete, delete website files on index complete 2018-06-19 11:24:28 -04:00