simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-04-20 19:06:42 +00:00
Code Issues Projects Releases Wiki Activity
206 Commits 2 Branches 0 Tags
Commit Graph

73 Commits

Author SHA1 Message Date
Simon
cc4c70f400 Request content is read all at once 2018-08-11 13:05:24 -04:00
Simon
458641654c Minimal configuration for reddit comment callback 2018-08-08 21:24:55 -04:00
Simon
5ff198b88a Fix for negative sizes 2018-07-25 11:37:12 -04:00
Simon
34d1f375a8 Crawler performance improvements 2018-07-25 11:27:50 -04:00
Simon
f12d5d524a exceptions during push_result are logged instead of raised 2018-07-21 10:45:17 -04:00
Simon
d43cf3b0ce Empty queue timeout increased to avoid that all workers die before the website is dropped 2018-07-20 14:11:17 -04:00
Simon
d3801adf74 Typo 2018-07-20 13:39:23 -04:00
Simon
1df5d194d2 Very slow websites are skipped. Should fix infinite waiting bug 2018-07-20 13:34:40 -04:00
Simon
004ade8935 Misc bug fixes 2018-07-20 10:35:17 -04:00
Simon
8ef1d36c9d Bug when directory is empty with new file upload 2018-07-17 18:24:05 -04:00
Simon
898ffcf410 File upload is made in small chunks 2018-07-17 17:52:17 -04:00
Simon
55a0fde19d Skip 'Parent directory' links more efficiently 2018-07-17 11:20:58 -04:00
Simon
756e331c83 Fixed bug in crawler when file count in a directory is greater than 150 2018-07-17 11:03:10 -04:00
Simon
112400886e Crawler no longer crashes when website has no files 2018-07-15 10:46:48 -04:00
Simon
e18ded7ac1 Temporarily removed logger in async methods (https://stackoverflow.com/questions/37907350) 2018-07-15 10:35:13 -04:00
Simon
152a6f20fb Re-enabled multi threaded file requests for large directories 2018-07-15 08:54:36 -04:00
Simon
f452d0f8b2 file lists now deleted after indexing 2018-07-14 20:41:20 -04:00
Simon
51a47b3628 Removed debug line 2018-07-14 17:36:16 -04:00
Simon
fe1d29aaea Crawl tasks are now fetched by the crawlers instead of pushed by the server 2018-07-14 17:31:18 -04:00
Simon
d138db8f06 Added filter to check if a website can be scanned from its parent directory 2018-07-10 10:14:23 -04:00
Simon
e384efd403 Bugfix for http crawler 2018-06-25 20:36:31 -04:00
Simon
d7ce1670a8 Logging and bugfix for http crawler 2018-06-25 14:36:16 -04:00
Simon
5fd00f22af Task logs now stored on main server 2018-06-24 20:32:02 -04:00
Simon
f6ee338c0f Removed unused statement 2018-06-24 18:24:41 -04:00
Simon
e11343de23 More FTP crawler bug fixes 2018-06-24 18:05:30 -04:00
Simon
ab35ce96cc FTP crawler bug fixes 2018-06-24 16:44:21 -04:00
Simon
8e937e69c0 Should fix some FTP errors 2018-06-24 13:50:55 -04:00
Simon
a6d753c6ee Added redispatch button and fixed typo in load balancing code 2018-06-24 10:07:46 -04:00
Simon
1ac510ff53 Slots can be updated without removing & adding 2018-06-24 09:39:44 -04:00
Simon
348914aba9 Removing unused module 2018-06-22 17:34:10 -04:00
Simon
e824b2bf3c Updated readme and UI fixes 2018-06-22 13:22:58 -04:00
Simon
7a4432e4d0 More bugfixes for looping directories, some work on task dispatching 2018-06-21 20:50:26 -04:00
Simon
14d384e366 Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default 2018-06-21 19:44:27 -04:00
Simon
098ad2be72 Should fix unknown encoding errors + removed https warnings 2018-06-21 19:23:01 -04:00
Simon
073551df3c Attempt to handle looping directories 2018-06-21 11:54:40 -04:00
Simon
dd93d40a55 Small bugfix for ftp crawler 2018-06-20 21:56:38 -04:00
Simon
c5deafbea5 Should fix some odd http listings 2018-06-20 13:34:41 -04:00
Simon
d8486104b4 Fix for odd html listings 2018-06-19 12:14:50 -04:00
Simon
e54609972c Overwrite document on re-index, update website last_modified on task complete, delete website files on index complete 2018-06-19 11:24:28 -04:00
Simon
8486555426 Ignore 'parent directory' links 2018-06-19 10:36:09 -04:00
Simon
677bfa03ea Another fix for encoding problems 2018-06-18 20:30:18 -04:00
Simon
8768e39f08 Added stats page 2018-06-18 19:56:25 -04:00
Simon
7923647ea3 Made the ftp crawler work with the latest changes 2018-06-18 15:46:03 -04:00
Simon
8a73142ff8 Support for more than just utf-8 and removed some debug info 2018-06-18 13:44:19 -04:00
Simon
b63c7190c3 Improved external link detection 2018-06-18 12:14:05 -04:00
Simon
400abc9a3c Added crawl logs page 2018-06-18 11:41:26 -04:00
Simon
99d64b658b Disabled thread pool for headers requests in listing 2018-06-18 10:33:33 -04:00
Simon
b97b8f6784 Temporary fix for decoding errors 2018-06-17 22:17:21 -04:00
Simon
344e7274d7 Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations 2018-06-17 22:10:46 -04:00
Simon
07d51a75cc Increased queue.get() timeouts 2018-06-17 10:07:06 -04:00