33 Commits

Author	SHA1	Message	Date
Simon	677bfa03ea	Another fix for encoding problems	2018-06-18 20:30:18 -04:00
Simon	8768e39f08	Added stats page	2018-06-18 19:56:25 -04:00
Simon	7923647ea3	Made the ftp crawler work with the latest changes	2018-06-18 15:46:03 -04:00
Simon	8a73142ff8	Support for more than just utf-8 and removed some debug info	2018-06-18 13:44:19 -04:00
Simon	b63c7190c3	Improved external link detection	2018-06-18 12:14:05 -04:00
Simon	400abc9a3c	Added crawl logs page	2018-06-18 11:41:26 -04:00
Simon	99d64b658b	Disabled thread pool for headers requests in listing	2018-06-18 10:33:33 -04:00
Simon	b97b8f6784	Temporary fix for decoding errors	2018-06-17 22:17:21 -04:00
Simon	344e7274d7	Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations	2018-06-17 22:10:46 -04:00
Simon	07d51a75cc	Increased queue.get() timeouts	2018-06-17 10:07:06 -04:00
Simon	e6175c84c9	Re-added timeout that was accidentally deleted	2018-06-16 22:20:15 -04:00
Simon	1283cc9599	Should fix memory usage problem when crawling (part three)	2018-06-16 20:32:50 -04:00
Simon	86144935e3	Attempt to fix Unicode errors part two	2018-06-16 15:30:44 -04:00
Simon	c309aa25c8	Attempt to fix unicode decode errors	2018-06-16 15:20:23 -04:00
Simon	9d0a0a8b42	Should fix memory usage problem when crawling (part two)	2018-06-16 14:53:48 -04:00
Simon	adb94cf326	Should fix memory usage problem when crawling	2018-06-14 23:36:54 -04:00
Simon	81fde6cc30	Bug fixes with html parsing	2018-06-14 20:02:06 -04:00
Simon	f3c7b551d2	Some adjustments to make it work on Stretch server	2018-06-14 17:09:05 -04:00
Simon	dffd032659	Indexing after crawling is a bit more efficient	2018-06-14 16:41:43 -04:00
Simon	83ca579ec7	Started working on post-crawl callbacks and basic auth for crawl servers	2018-06-14 15:05:56 -04:00
Simon	1bd58468eb	Bug fixes for FTP crawler	2018-06-13 15:54:45 -04:00
Simon	2fe81e4b06	Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart	2018-06-12 22:28:36 -04:00
Simon	24ef493245	Websites being indexed now show up on the homepage	2018-06-12 21:51:02 -04:00
Simon	e266a50197	Website stats now works with elasticsearch	2018-06-12 20:17:30 -04:00
Simon	4b60ac62fc	Added website url & date in search results & fixed threading problem	2018-06-12 17:48:15 -04:00
Simon	af2601ee70	Fixed file duplication problem	2018-06-12 15:55:52 -04:00
Simon	1718bb91ca	Files are indexed into ES when task is complete	2018-06-12 15:45:00 -04:00
Simon	6c912ea8c5	Completed tasks are now fetched by the TaskDispatcher	2018-06-12 14:16:05 -04:00
Simon	d61fd75890	Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)	2018-06-12 13:44:03 -04:00
Simon	6d48f1f780	Task crawl result now logged in a database	2018-06-12 11:03:45 -04:00
Simon	72495275b0	Elasticsearch search engine (import from json)	2018-06-11 22:35:49 -04:00
Simon	fcfd7d4acc	Bug fixes + export to json	2018-06-11 20:02:30 -04:00
Simon	d849227798	barebones crawl_server microservice	2018-06-11 19:00:43 -04:00