7 Commits

Author	SHA1	Message	Date
Simon	fe1d29aaea	Crawl tasks are now fetched by the crawlers instead of pushed by the server	2018-07-14 17:31:18 -04:00
Simon	5afdfb2b3c	fixed navbar icon for mobile	2018-06-19 21:13:36 -04:00
Simon	344e7274d7	Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations	2018-06-17 22:10:46 -04:00
Simon	dffd032659	Indexing after crawling is a bit more efficient	2018-06-14 16:41:43 -04:00
Simon	83ca579ec7	Started working on post-crawl callbacks and basic auth for crawl servers	2018-06-14 15:05:56 -04:00
Simon	1bd58468eb	Bug fixes for FTP crawler	2018-06-13 15:54:45 -04:00
Simon	9bde8cb629	uWSGI config and bugfix with file extensions	2018-06-13 14:11:27 -04:00