20 Commits

Author	SHA1	Message	Date
Simon	098ad2be72	Should fix unknown encoding errors + removed https warnings	2018-06-21 19:23:01 -04:00
Simon	073551df3c	Attempt to handle looping directories	2018-06-21 11:54:40 -04:00
Simon	c5deafbea5	Should fix some odd http listings	2018-06-20 13:34:41 -04:00
Simon	d8486104b4	Fix for odd html listings	2018-06-19 12:14:50 -04:00
Simon	8486555426	Ignore 'parent directory' links	2018-06-19 10:36:09 -04:00
Simon	677bfa03ea	Another fix for encoding problems	2018-06-18 20:30:18 -04:00
Simon	8a73142ff8	Support for more than just utf-8 and removed some debug info	2018-06-18 13:44:19 -04:00
Simon	b63c7190c3	Improved external link detection	2018-06-18 12:14:05 -04:00
Simon	99d64b658b	Disabled thread pool for headers requests in listing	2018-06-18 10:33:33 -04:00
Simon	b97b8f6784	Temporary fix for decoding errors	2018-06-17 22:17:21 -04:00
Simon	344e7274d7	Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations	2018-06-17 22:10:46 -04:00
Simon	e6175c84c9	Re-added timeout that was accidentally deleted	2018-06-16 22:20:15 -04:00
Simon	1283cc9599	Should fix memory usage problem when crawling (part three)	2018-06-16 20:32:50 -04:00
Simon	86144935e3	Attempt to fix Unicode errors part two	2018-06-16 15:30:44 -04:00
Simon	c309aa25c8	Attempt to fix unicode decode errors	2018-06-16 15:20:23 -04:00
Simon	9d0a0a8b42	Should fix memory usage problem when crawling (part two)	2018-06-16 14:53:48 -04:00
Simon	adb94cf326	Should fix memory usage problem when crawling	2018-06-14 23:36:54 -04:00
Simon	81fde6cc30	Bug fixes with html parsing	2018-06-14 20:02:06 -04:00
Simon	83ca579ec7	Started working on post-crawl callbacks and basic auth for crawl servers	2018-06-14 15:05:56 -04:00
Simon	d61fd75890	Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)	2018-06-12 13:44:03 -04:00