20 Commits

Author	SHA1	Message	Date
simon	e8f0f96148	More bug fixes...	2020-01-25 12:54:26 -05:00
simon	ae0fb9b1a6	Docker tweaking & bug fixes	2020-01-25 10:16:24 -05:00
simon	df8ab7727b	docker-compose setup (wip)	2019-11-13 13:03:43 -05:00
simon987	6843900ec6	Change max assign time	2019-04-06 13:33:32 -04:00
simon987	b8b531f511	Move recrawl task to cron job	2019-04-06 10:50:15 -04:00
simon987	0c3d0b38e6	Don't use multiprocessing for recrawl task	2019-04-06 09:21:02 -04:00
simon987	06ae89f4d2	Only queue http tasks (temp)	2019-04-06 09:07:17 -04:00
simon987	5b680be770	Update readme	2019-03-30 09:16:54 -04:00
simon987	e02d08ca62	bugfix	2019-03-28 21:04:38 -04:00
simon987	2046b36f9a	Bug fixes	2019-03-28 20:29:34 -04:00
simon987	b9f25630b4	Switch to postgresql, finish minimum viable task_tracker/ws_bucket integration	2019-03-27 19:34:05 -04:00
simon987	4ffe805b8d	Use task_tracker for task tracking	2019-03-24 20:23:05 -04:00
simon987	00e3fd7340	Remove task tracking	2019-03-09 13:26:05 -05:00
Simon	254d6ea44e	More debug logging	2018-11-18 11:15:23 -05:00
Simon	98f43f817a	revert task queuing pt. 3 ._.	2018-11-18 11:13:51 -05:00
Simon	801d056da8	revert task queuing pt. 2	2018-11-18 11:11:38 -05:00
Simon	4ce807c8a0	revert task queuing	2018-11-18 11:10:44 -05:00
Simon	1812ec932d	Completed tasks queue	2018-11-17 21:36:29 -05:00
Simon	112400886e	Crawler no longer crashes when website has no files	2018-07-15 10:46:48 -04:00
Simon	fe1d29aaea	Crawl tasks are now fetched by the crawlers instead of pushed by the server	2018-07-14 17:31:18 -04:00