68 Commits

Author	SHA1	Message	Date
Simon	e54609972c	Overwrite document on re-index, update website last_modified on task complete, delete website files on index complete	2018-06-19 11:24:28 -04:00
Simon	e5e38a6faf	Elasticsearch export to csv	2018-06-19 09:48:44 -04:00
Simon	81d52a4551	Changed UI to fit the-eye.eu	2018-06-18 22:37:05 -04:00
Simon	8768e39f08	Added stats page	2018-06-18 19:56:25 -04:00
Simon	adb94cf326	Should fix memory usage problem when crawling	2018-06-14 23:36:54 -04:00
Simon	9aed18c2d2	Should fix timeout error when indexing	2018-06-14 20:07:50 -04:00
Simon	dffd032659	Indexing after crawling is a bit more efficient	2018-06-14 16:41:43 -04:00
Simon	9bde8cb629	uWSGI config and bugfix with file extensions	2018-06-13 14:11:27 -04:00
Simon	e91572a06f	Homepage stats now work with elasticsearch	2018-06-12 23:19:57 -04:00
Simon	2fe81e4b06	Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart	2018-06-12 22:28:36 -04:00
Simon	bccb1d0dfd	Website link list works with elasticsearch	2018-06-12 21:26:44 -04:00
Simon	e266a50197	Website stats now works with elasticsearch	2018-06-12 20:17:30 -04:00
Simon	4b60ac62fc	Added website url & date in search results & fixed threading problem	2018-06-12 17:48:15 -04:00
Simon	0127b3a51d	Basic searching integrated with elasticsearch + highlighting	2018-06-12 16:29:05 -04:00
Simon	af2601ee70	Fixed file duplication problem	2018-06-12 15:55:52 -04:00
Simon	1718bb91ca	Files are indexed into ES when task is complete	2018-06-12 15:45:00 -04:00
Simon	011b8455a7	Elasticsearch search engine (search & scroll)	2018-06-11 23:06:41 -04:00
Simon	72495275b0	Elasticsearch search engine (import from json)	2018-06-11 22:35:49 -04:00