24 Commits

Author	SHA1	Message	Date
Simon	a6d753c6ee	Added redispatch button and fixed typo in load balancing code	2018-06-24 10:07:46 -04:00
Simon	9d3fc2d71b	typo (again)	2018-06-21 21:26:44 -04:00
Simon	efd1981e6f	typo	2018-06-21 21:00:50 -04:00
Simon	7a4432e4d0	More bugfixes for looping directories, some work on task dispatching	2018-06-21 20:50:26 -04:00
Simon	14d384e366	Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default	2018-06-21 19:44:27 -04:00
Simon	e54609972c	Overwrite document on re-index, update website last_modified on task complete, delete website files on index complete	2018-06-19 11:24:28 -04:00
Simon	81d52a4551	Changed UI to fit the-eye.eu	2018-06-18 22:37:05 -04:00
Simon	8768e39f08	Added stats page	2018-06-18 19:56:25 -04:00
Simon	400abc9a3c	Added crawl logs page	2018-06-18 11:41:26 -04:00
Simon	81fde6cc30	Bug fixes with html parsing	2018-06-14 20:02:06 -04:00
Simon	dffd032659	Indexing after crawling is a bit more efficient	2018-06-14 16:41:43 -04:00
Simon	83ca579ec7	Started working on post-crawl callbacks and basic auth for crawl servers	2018-06-14 15:05:56 -04:00
Simon	9bde8cb629	uWSGI config and bugfix with file extensions	2018-06-13 14:11:27 -04:00
Simon	2fe81e4b06	Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart	2018-06-12 22:28:36 -04:00
Simon	24ef493245	Websites being indexed now show up on the homepage	2018-06-12 21:51:02 -04:00
Simon	1718bb91ca	Files are indexed into ES when task is complete	2018-06-12 15:45:00 -04:00
Simon	6c912ea8c5	Completed tasks are now fetched by the TaskDispatcher	2018-06-12 14:16:05 -04:00
Simon	d61fd75890	Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)	2018-06-12 13:44:03 -04:00
Simon	d849227798	barebones crawl_server microservice	2018-06-11 19:00:43 -04:00
Simon	0304c98a31	Added basic ftp spider for scrapy	2018-06-10 14:12:55 -04:00
Simon	4b1fce309c	Mark reddit post as crawled even if too small to comment	2018-06-06 16:01:07 -04:00
Simon	270ab1335a	Added reply to comments option, fixed some bugs	2018-06-02 17:26:15 -04:00
Simon	bb872a9248	Changed from mime to extension for graph and added script to clear invalid websites	2018-05-31 10:51:59 -04:00
Simon	ad645490f6	Initial commit	2018-05-28 20:35:04 -04:00