10 Commits

Author	SHA1	Message	Date
Simon	d7ce1670a8	Logging and bugfix for http crawler	2018-06-25 14:36:16 -04:00
Simon	5fd00f22af	Task logs now stored on main server	2018-06-24 20:32:02 -04:00
Simon	a6d753c6ee	Added redispatch button and fixed typo in load balancing code	2018-06-24 10:07:46 -04:00
Simon	400abc9a3c	Added crawl logs page	2018-06-18 11:41:26 -04:00
Simon	83ca579ec7	Started working on post-crawl callbacks and basic auth for crawl servers	2018-06-14 15:05:56 -04:00
Simon	1718bb91ca	Files are indexed into ES when task is complete	2018-06-12 15:45:00 -04:00
Simon	6c912ea8c5	Completed tasks are now fetched by the TaskDispatcher	2018-06-12 14:16:05 -04:00
Simon	d61fd75890	Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)	2018-06-12 13:44:03 -04:00
Simon	6d48f1f780	Task crawl result now logged in a database	2018-06-12 11:03:45 -04:00
Simon	d849227798	barebones crawl_server microservice	2018-06-11 19:00:43 -04:00