simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-04-20 19:06:42 +00:00
Code Issues Projects Releases Wiki Activity
167 Commits 2 Branches 0 Tags
Commit Graph

61 Commits

Author SHA1 Message Date
Simon
756e331c83 Fixed bug in crawler when file count in a directory is greater than 150 2018-07-17 11:03:10 -04:00
Simon
112400886e Crawler no longer crashes when website has no files 2018-07-15 10:46:48 -04:00
Simon
e18ded7ac1 Temporarily removed logger in async methods (https://stackoverflow.com/questions/37907350) 2018-07-15 10:35:13 -04:00
Simon
152a6f20fb Re-enabled multi threaded file requests for large directories 2018-07-15 08:54:36 -04:00
Simon
f452d0f8b2 file lists now deleted after indexing 2018-07-14 20:41:20 -04:00
Simon
51a47b3628 Removed debug line 2018-07-14 17:36:16 -04:00
Simon
fe1d29aaea Crawl tasks are now fetched by the crawlers instead of pushed by the server 2018-07-14 17:31:18 -04:00
Simon
d138db8f06 Added filter to check if a website can be scanned from its parent directory 2018-07-10 10:14:23 -04:00
Simon
e384efd403 Bugfix for http crawler 2018-06-25 20:36:31 -04:00
Simon
d7ce1670a8 Logging and bugfix for http crawler 2018-06-25 14:36:16 -04:00
Simon
5fd00f22af Task logs now stored on main server 2018-06-24 20:32:02 -04:00
Simon
f6ee338c0f Removed unused statement 2018-06-24 18:24:41 -04:00
Simon
e11343de23 More FTP crawler bug fixes 2018-06-24 18:05:30 -04:00
Simon
ab35ce96cc FTP crawler bug fixes 2018-06-24 16:44:21 -04:00
Simon
8e937e69c0 Should fix some FTP errors 2018-06-24 13:50:55 -04:00
Simon
a6d753c6ee Added redispatch button and fixed typo in load balancing code 2018-06-24 10:07:46 -04:00
Simon
1ac510ff53 Slots can be updated without removing & adding 2018-06-24 09:39:44 -04:00
Simon
348914aba9 Removing unused module 2018-06-22 17:34:10 -04:00
Simon
e824b2bf3c Updated readme and UI fixes 2018-06-22 13:22:58 -04:00
Simon
7a4432e4d0 More bugfixes for looping directories, some work on task dispatching 2018-06-21 20:50:26 -04:00
Simon
14d384e366 Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default 2018-06-21 19:44:27 -04:00
Simon
098ad2be72 Should fix unknown encoding errors + removed https warnings 2018-06-21 19:23:01 -04:00
Simon
073551df3c Attempt to handle looping directories 2018-06-21 11:54:40 -04:00
Simon
dd93d40a55 Small bugfix for ftp crawler 2018-06-20 21:56:38 -04:00
Simon
c5deafbea5 Should fix some odd http listings 2018-06-20 13:34:41 -04:00
Simon
d8486104b4 Fix for odd html listings 2018-06-19 12:14:50 -04:00
Simon
e54609972c Overwrite document on re-index, update website last_modified on task complete, delete website files on index complete 2018-06-19 11:24:28 -04:00
Simon
8486555426 Ignore 'parent directory' links 2018-06-19 10:36:09 -04:00
Simon
677bfa03ea Another fix for encoding problems 2018-06-18 20:30:18 -04:00
Simon
8768e39f08 Added stats page 2018-06-18 19:56:25 -04:00
Simon
7923647ea3 Made the ftp crawler work with the latest changes 2018-06-18 15:46:03 -04:00
Simon
8a73142ff8 Support for more than just utf-8 and removed some debug info 2018-06-18 13:44:19 -04:00
Simon
b63c7190c3 Improved external link detection 2018-06-18 12:14:05 -04:00
Simon
400abc9a3c Added crawl logs page 2018-06-18 11:41:26 -04:00
Simon
99d64b658b Disabled thread pool for headers requests in listing 2018-06-18 10:33:33 -04:00
Simon
b97b8f6784 Temporary fix for decoding errors 2018-06-17 22:17:21 -04:00
Simon
344e7274d7 Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations 2018-06-17 22:10:46 -04:00
Simon
07d51a75cc Increased queue.get() timeouts 2018-06-17 10:07:06 -04:00
Simon
e6175c84c9 Re-added timeout that was accidentally deleted 2018-06-16 22:20:15 -04:00
Simon
1283cc9599 Should fix memory usage problem when crawling (part three) 2018-06-16 20:32:50 -04:00
Simon
86144935e3 Attempt to fix Unicode errors part two 2018-06-16 15:30:44 -04:00
Simon
c309aa25c8 Attempt to fix unicode decode errors 2018-06-16 15:20:23 -04:00
Simon
9d0a0a8b42 Should fix memory usage problem when crawling (part two) 2018-06-16 14:53:48 -04:00
Simon
adb94cf326 Should fix memory usage problem when crawling 2018-06-14 23:36:54 -04:00
Simon
81fde6cc30 Bug fixes with html parsing 2018-06-14 20:02:06 -04:00
Simon
f3c7b551d2 Some adjustments to make it work on Stretch server 2018-06-14 17:09:05 -04:00
Simon
dffd032659 Indexing after crawling is a bit more efficient 2018-06-14 16:41:43 -04:00
Simon
83ca579ec7 Started working on post-crawl callbacks and basic auth for crawl servers 2018-06-14 15:05:56 -04:00
Simon
1bd58468eb Bug fixes for FTP crawler 2018-06-13 15:54:45 -04:00
Simon
2fe81e4b06 Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart 2018-06-12 22:28:36 -04:00