91 Commits

Author	SHA1	Message	Date
Simon	a6d753c6ee	Added redispatch button and fixed typo in load balancing code	2018-06-24 10:07:46 -04:00
Simon	1ac510ff53	Slots can be updated without removing & adding	2018-06-24 09:39:44 -04:00
Simon	e824b2bf3c	Updated readme and UI fixes	2018-06-22 13:22:58 -04:00
Simon	14d384e366	Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default	2018-06-21 19:44:27 -04:00
Simon	80aa8933e6	Added rescan button	2018-06-21 13:02:16 -04:00
Simon	cf51bb381c	Added top websites scatter graph	2018-06-20 12:21:34 -04:00
Simon	7400bdc2a9	Added admin blacklist control in dashboard	2018-06-20 11:28:06 -04:00
Simon	35837463cd	Added admin clear & delete buttons for websites	2018-06-20 10:48:51 -04:00
Simon	8768e39f08	Added stats page	2018-06-18 19:56:25 -04:00
Simon	83f4b8def9	Enhanced search results page	2018-06-18 15:01:49 -04:00
Simon	400abc9a3c	Added crawl logs page	2018-06-18 11:41:26 -04:00
Simon	1283cc9599	Should fix memory usage problem when crawling (part three)	2018-06-16 20:32:50 -04:00
Simon	83ca579ec7	Started working on post-crawl callbacks and basic auth for crawl servers	2018-06-14 15:05:56 -04:00
Simon	9bde8cb629	uWSGI config and bugfix with file extensions	2018-06-13 14:11:27 -04:00
Simon	e91572a06f	Homepage stats now work with elasticsearch	2018-06-12 23:19:57 -04:00
Simon	2fe81e4b06	Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart	2018-06-12 22:28:36 -04:00
Simon	24ef493245	Websites being indexed now show up on the homepage	2018-06-12 21:51:02 -04:00
Simon	bccb1d0dfd	Website link list works with elasticsearch	2018-06-12 21:26:44 -04:00
Simon	e266a50197	Website stats now works with elasticsearch	2018-06-12 20:17:30 -04:00
Simon	4b60ac62fc	Added website url & date in search results & fixed threading problem	2018-06-12 17:48:15 -04:00
Simon	0127b3a51d	Basic searching integrated with elasticsearch + highlighting	2018-06-12 16:29:05 -04:00
Simon	d61fd75890	Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)	2018-06-12 13:44:03 -04:00
Simon	f2d914060b	Removed unsuitable scrapy spider and implemented custom crawler	2018-06-10 20:08:59 -04:00
Simon	f1e8183cdf	Bulk insert captcha	2018-06-10 07:21:44 -04:00
Simon	4523a4335c	Added bulk insert feature	2018-06-10 07:20:58 -04:00
Simon	a25976d24a	Generate and delete API tokens	2018-06-09 12:41:28 -04:00
Simon	20d0f97ffb	Logout button	2018-06-08 11:48:11 -04:00
Simon	dc0cde61a0	Basic admin page	2018-06-08 11:40:54 -04:00
Simon	537228444b	Duplicate website w/ reddit post + refactor	2018-06-08 10:40:58 -04:00
Simon	b79b0ca58c	Results per page now configurable	2018-06-07 13:49:08 -04:00
Simon	306b0ed0fe	Added option to choose results per page	2018-06-07 13:19:41 -04:00
Simon	06d3a09e11	Quick hack for search order options	2018-06-07 11:22:35 -04:00
Simon	4f6d7f32ad	Option to turn off SSL, moved secret keys to config.py, switched to sqlite WAL mode to avoid locked database problems	2018-06-07 10:33:35 -04:00
Simon	a92d8fe44e	SSL support pt.2	2018-06-06 11:35:55 -04:00
Simon	abe5935847	SSL support	2018-06-06 11:34:07 -04:00
Simon	820caa171b	Download link for export	2018-06-06 11:27:23 -04:00
Simon	7ac7972fd6	Exports are downloadable	2018-06-06 11:12:24 -04:00
Simon	0b1d76f478	Added blacklist feature (untested)	2018-06-06 10:17:30 -04:00
Simon	270ab1335a	Added reply to comments option, fixed some bugs	2018-06-02 17:26:15 -04:00
Simon	ca651278d0	Renamed package	2018-05-28 20:39:34 -04:00
Simon	ad645490f6	Initial commit	2018-05-28 20:35:04 -04:00