13 Commits

Author	SHA1	Message	Date
Simon	7a4432e4d0	More bugfixes for looping directories, some work on task dispatching	2018-06-21 20:50:26 -04:00
Simon	14d384e366	Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default	2018-06-21 19:44:27 -04:00
Simon	073551df3c	Attempt to handle looping directories	2018-06-21 11:54:40 -04:00
Simon	8a73142ff8	Support for more than just utf-8 and removed some debug info	2018-06-18 13:44:19 -04:00
Simon	99d64b658b	Disabled thread pool for headers requests in listing	2018-06-18 10:33:33 -04:00
Simon	344e7274d7	Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations	2018-06-17 22:10:46 -04:00
Simon	07d51a75cc	Increased queue.get() timeouts	2018-06-17 10:07:06 -04:00
Simon	9d0a0a8b42	Should fix memory usage problem when crawling (part two)	2018-06-16 14:53:48 -04:00
Simon	adb94cf326	Should fix memory usage problem when crawling	2018-06-14 23:36:54 -04:00
Simon	f3c7b551d2	Some adjustments to make it work on Stretch server	2018-06-14 17:09:05 -04:00
Simon	1bd58468eb	Bug fixes for FTP crawler	2018-06-13 15:54:45 -04:00
Simon	af2601ee70	Fixed file duplication problem	2018-06-12 15:55:52 -04:00
Simon	d61fd75890	Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)	2018-06-12 13:44:03 -04:00