33 Commits

Author	SHA1	Message	Date
Simon	34d1f375a8	Crawler performance improvements	2018-07-25 11:27:50 -04:00
Simon	d3801adf74	Typo	2018-07-20 13:39:23 -04:00
Simon	1df5d194d2	Very slow websites are skipped. Should fix infinite waiting bug	2018-07-20 13:34:40 -04:00
Simon	55a0fde19d	Skip 'Parent directory' links more efficiently	2018-07-17 11:20:58 -04:00
Simon	756e331c83	Fixed bug in crawler when file count in a directory is greater than 150	2018-07-17 11:03:10 -04:00
Simon	152a6f20fb	Re-enabled multi threaded file requests for large directories	2018-07-15 08:54:36 -04:00
Simon	f452d0f8b2	file lists now deleted after indexing	2018-07-14 20:41:20 -04:00
Simon	d138db8f06	Added filter to check if a website can be scanned from its parent directory	2018-07-10 10:14:23 -04:00
Simon	e384efd403	Bugfix for http crawler	2018-06-25 20:36:31 -04:00
Simon	d7ce1670a8	Logging and bugfix for http crawler	2018-06-25 14:36:16 -04:00
Simon	1ac510ff53	Slots can be updated without removing & adding	2018-06-24 09:39:44 -04:00
Simon	e824b2bf3c	Updated readme and UI fixes	2018-06-22 13:22:58 -04:00
Simon	7a4432e4d0	More bugfixes for looping directories, some work on task dispatching	2018-06-21 20:50:26 -04:00
Simon	098ad2be72	Should fix unknown encoding errors + removed https warnings	2018-06-21 19:23:01 -04:00
Simon	073551df3c	Attempt to handle looping directories	2018-06-21 11:54:40 -04:00
Simon	c5deafbea5	Should fix some odd http listings	2018-06-20 13:34:41 -04:00
Simon	d8486104b4	Fix for odd html listings	2018-06-19 12:14:50 -04:00
Simon	8486555426	Ignore 'parent directory' links	2018-06-19 10:36:09 -04:00
Simon	677bfa03ea	Another fix for encoding problems	2018-06-18 20:30:18 -04:00
Simon	8a73142ff8	Support for more than just utf-8 and removed some debug info	2018-06-18 13:44:19 -04:00
Simon	b63c7190c3	Improved external link detection	2018-06-18 12:14:05 -04:00
Simon	99d64b658b	Disabled thread pool for headers requests in listing	2018-06-18 10:33:33 -04:00
Simon	b97b8f6784	Temporary fix for decoding errors	2018-06-17 22:17:21 -04:00
Simon	344e7274d7	Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations	2018-06-17 22:10:46 -04:00
Simon	e6175c84c9	Re-added timeout that was accidentally deleted	2018-06-16 22:20:15 -04:00
Simon	1283cc9599	Should fix memory usage problem when crawling (part three)	2018-06-16 20:32:50 -04:00
Simon	86144935e3	Attempt to fix Unicode errors part two	2018-06-16 15:30:44 -04:00
Simon	c309aa25c8	Attempt to fix unicode decode errors	2018-06-16 15:20:23 -04:00
Simon	9d0a0a8b42	Should fix memory usage problem when crawling (part two)	2018-06-16 14:53:48 -04:00
Simon	adb94cf326	Should fix memory usage problem when crawling	2018-06-14 23:36:54 -04:00
Simon	81fde6cc30	Bug fixes with html parsing	2018-06-14 20:02:06 -04:00
Simon	83ca579ec7	Started working on post-crawl callbacks and basic auth for crawl servers	2018-06-14 15:05:56 -04:00
Simon	d61fd75890	Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)	2018-06-12 13:44:03 -04:00