11 Commits

Author	SHA1	Message	Date
Simon	2fe81e4b06	Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart	2018-06-12 22:28:36 -04:00
Simon	24ef493245	Websites being indexed now show up on the homepage	2018-06-12 21:51:02 -04:00
Simon	1718bb91ca	Files are indexed into ES when task is complete	2018-06-12 15:45:00 -04:00
Simon	6c912ea8c5	Completed tasks are now fetched by the TaskDispatcher	2018-06-12 14:16:05 -04:00
Simon	d61fd75890	Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)	2018-06-12 13:44:03 -04:00
Simon	d849227798	barebones crawl_server microservice	2018-06-11 19:00:43 -04:00
Simon	0304c98a31	Added basic ftp spider for scrapy	2018-06-10 14:12:55 -04:00
Simon	4b1fce309c	Mark reddit post as crawled even if too small to comment	2018-06-06 16:01:07 -04:00
Simon	270ab1335a	Added reply to comments option, fixed some bugs	2018-06-02 17:26:15 -04:00
Simon	bb872a9248	Changed from mime to extension for graph and added script to clear invalid websites	2018-05-31 10:51:59 -04:00
Simon	ad645490f6	Initial commit	2018-05-28 20:35:04 -04:00