44 Commits

Author	SHA1	Message	Date
Simon	2ef60a05a5	proper user agent for pycurl	2018-09-29 11:38:52 -04:00
Simon	8b13de4a6b	Re-init curl handle on error	2018-08-25 16:46:53 -04:00
Simon	faa9ac3ccb	Closing curl handle manually just to make sure	2018-08-23 12:48:15 -04:00
Simon	dff4125c9f	Bugfix post-pycurl update pt. 3 (Sorry!)	2018-08-23 12:47:17 -04:00
Simon	25e1e58828	Bugfix post-pycurl update pt. 3 (Sorry!)	2018-08-23 12:46:00 -04:00
Simon	6ffc43601b	Bugfix post-pycurl update pt. 2	2018-08-23 12:40:13 -04:00
Simon	484a0baf9d	Bugfix post-pycurl update	2018-08-23 12:37:27 -04:00
Simon	cadaf14c1b	Small bugfix	2018-08-23 12:12:23 -04:00
Simon	54b4d2d5b4	removed debug lines	2018-08-23 12:02:07 -04:00
Simon	85c3aa918d	replaced requests by pycurl	2018-08-23 11:47:09 -04:00
Simon	cc4c70f400	Request content is read all at once	2018-08-11 13:05:24 -04:00
Simon	34d1f375a8	Crawler performance improvements	2018-07-25 11:27:50 -04:00
Simon	d3801adf74	Typo	2018-07-20 13:39:23 -04:00
Simon	1df5d194d2	Very slow websites are skipped. Should fix infinite waiting bug	2018-07-20 13:34:40 -04:00
Simon	55a0fde19d	Skip 'Parent directory' links more efficiently	2018-07-17 11:20:58 -04:00
Simon	756e331c83	Fixed bug in crawler when file count in a directory is greater than 150	2018-07-17 11:03:10 -04:00
Simon	152a6f20fb	Re-enabled multi threaded file requests for large directories	2018-07-15 08:54:36 -04:00
Simon	f452d0f8b2	file lists now deleted after indexing	2018-07-14 20:41:20 -04:00
Simon	d138db8f06	Added filter to check if a website can be scanned from its parent directory	2018-07-10 10:14:23 -04:00
Simon	e384efd403	Bugfix for http crawler	2018-06-25 20:36:31 -04:00
Simon	d7ce1670a8	Logging and bugfix for http crawler	2018-06-25 14:36:16 -04:00
Simon	1ac510ff53	Slots can be updated without removing & adding	2018-06-24 09:39:44 -04:00
Simon	e824b2bf3c	Updated readme and UI fixes	2018-06-22 13:22:58 -04:00
Simon	7a4432e4d0	More bugfixes for looping directories, some work on task dispatching	2018-06-21 20:50:26 -04:00
Simon	098ad2be72	Should fix unknown encoding errors + removed https warnings	2018-06-21 19:23:01 -04:00
Simon	073551df3c	Attempt to handle looping directories	2018-06-21 11:54:40 -04:00
Simon	c5deafbea5	Should fix some odd http listings	2018-06-20 13:34:41 -04:00
Simon	d8486104b4	Fix for odd html listings	2018-06-19 12:14:50 -04:00
Simon	8486555426	Ignore 'parent directory' links	2018-06-19 10:36:09 -04:00
Simon	677bfa03ea	Another fix for encoding problems	2018-06-18 20:30:18 -04:00
Simon	8a73142ff8	Support for more than just utf-8 and removed some debug info	2018-06-18 13:44:19 -04:00
Simon	b63c7190c3	Improved external link detection	2018-06-18 12:14:05 -04:00
Simon	99d64b658b	Disabled thread pool for headers requests in listing	2018-06-18 10:33:33 -04:00
Simon	b97b8f6784	Temporary fix for decoding errors	2018-06-17 22:17:21 -04:00
Simon	344e7274d7	Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations	2018-06-17 22:10:46 -04:00
Simon	e6175c84c9	Re-added timeout that was accidentally deleted	2018-06-16 22:20:15 -04:00
Simon	1283cc9599	Should fix memory usage problem when crawling (part three)	2018-06-16 20:32:50 -04:00
Simon	86144935e3	Attempt to fix Unicode errors part two	2018-06-16 15:30:44 -04:00
Simon	c309aa25c8	Attempt to fix unicode decode errors	2018-06-16 15:20:23 -04:00
Simon	9d0a0a8b42	Should fix memory usage problem when crawling (part two)	2018-06-16 14:53:48 -04:00
Simon	adb94cf326	Should fix memory usage problem when crawling	2018-06-14 23:36:54 -04:00
Simon	81fde6cc30	Bug fixes with html parsing	2018-06-14 20:02:06 -04:00
Simon	83ca579ec7	Started working on post-crawl callbacks and basic auth for crawl servers	2018-06-14 15:05:56 -04:00
Simon	d61fd75890	Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)	2018-06-12 13:44:03 -04:00