69 Commits

Author	SHA1	Message	Date
simon	31877283b3	Bug fixes, ES7	2019-06-14 14:25:41 -04:00
simon987	4ffe805b8d	Use task_tracker for task tracking	2019-03-24 20:23:05 -04:00
simon987	8ced4859f3	hotfix attempt 1	2019-02-02 09:17:59 -05:00
simon987	dd3775cecd	Fix ES Settings	2019-01-13 13:52:49 -05:00
simon987	7f857d641f	Change ES settings, big refactor, removed recaptcha	2019-01-13 12:48:39 -05:00
Simon	a33d4f2dca	retry on index fail/timeout	2018-11-22 10:52:42 -05:00
Simon	750940d148	error handling for delete	2018-11-18 11:19:34 -05:00
Simon	254d6ea44e	More debug logging	2018-11-18 11:15:23 -05:00
Simon	64ce9379c3	Replace delete_by_query with bulk delete (cleanup)	2018-11-18 11:07:34 -05:00
Simon	2f6ae3cb35	Replace delete_by_query with bulk delete	2018-11-18 11:07:18 -05:00
Simon	876a511b54	Lowered search timeout	2018-11-17 12:08:04 -05:00
Simon	372c10b5ab	Nicer search logging	2018-11-17 12:04:59 -05:00
Simon	4996de6aa9	Logging for search and better error handling	2018-11-17 11:19:09 -05:00
Simon	6e80791264	Search filter	2018-11-16 16:49:23 -05:00
Simon	85c3aa918d	replaced requests by pycurl	2018-08-23 11:47:09 -04:00
Simon	bbe8ed07a8	Reset page number on search	2018-08-14 16:20:00 -04:00
Simon	c92f2f4937	Should fix export problem	2018-08-14 12:21:34 -04:00
Simon	edede200f4	Decresed number of indexed documents per second	2018-08-12 14:58:27 -04:00
Simon	bab68819df	Increased stats generation interval	2018-08-10 15:27:37 -04:00
Simon	c94cf5b313	Adjusted timeout values (again)	2018-08-10 11:46:16 -04:00
Simon	faeff701de	Increased search timeout value	2018-08-09 18:33:35 -04:00
Simon	ffeed4192e	Refresh index before reddit comment callback	2018-08-09 16:19:21 -04:00
Simon	8ffd9179d2	Increased stats timeout value	2018-08-09 14:26:22 -04:00
Simon	1ff1e039f5	Increased stats generation interval	2018-07-25 12:29:14 -04:00
Simon	3d63184287	Increased website scatter size	2018-07-25 11:33:21 -04:00
Simon	fbbe952e4d	Stats are generated in background and stored to file instead of on-demand	2018-07-24 20:29:25 -04:00
Simon	004ade8935	Misc bug fixes	2018-07-20 10:35:17 -04:00
Simon	2f1b0c96f1	retry if deleting docs fail	2018-07-14 21:34:25 -04:00
Simon	fe1d29aaea	Crawl tasks are now fetched by the crawlers instead of pushed by the server	2018-07-14 17:31:18 -04:00
Simon	d9e9f53f92	Website should stay online even if elasticsearch is down / timing out	2018-07-12 12:06:45 -04:00
Simon	f202caece8	Increased cooldown time for indexing	2018-07-12 11:45:50 -04:00
Simon	c0327fecda	Increased delete_docs timeout value	2018-07-12 11:30:16 -04:00
Simon	290322dfa7	Indexing is a bit gentler on server resources and some pages have been memory cached	2018-07-12 11:26:20 -04:00
Simon	1ee1c3c35d	safer document deletion	2018-07-10 19:26:06 -04:00
Simon	711e8282ef	'Go to random website' button, and navigation in the website list	2018-07-08 10:42:14 -04:00
Simon	9ff21e7943	Fixed indentation error	2018-06-28 22:47:14 -04:00
Simon	4c9d79fdbf	Added filter for large files in stats	2018-06-28 10:40:54 -04:00
Simon	14037c5f21	Added more extension types and adjusted global stats histograms	2018-06-27 11:53:32 -04:00
Simon	6a3d540de2	Added date filter in search options and github banner on homepage	2018-06-27 10:05:33 -04:00
Simon	b570e81bec	More search options	2018-06-26 21:38:26 -04:00
Simon	4abd8d12e2	Added size filter	2018-06-26 20:21:24 -04:00
Simon	8ea57967e6	Filter by extension type	2018-06-26 19:02:46 -04:00
Simon	a0bd45c829	Increased ES timeouts	2018-06-26 17:01:17 -04:00
Simon	14d384e366	Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default	2018-06-21 19:44:27 -04:00
Simon	a7e4e3ae1f	Fixed stats bug	2018-06-20 18:07:55 -04:00
Simon	cf51bb381c	Added top websites scatter graph	2018-06-20 12:21:34 -04:00
Simon	35837463cd	Added admin clear & delete buttons for websites	2018-06-20 10:48:51 -04:00
Simon	5f07e7d340	File types color based on type	2018-06-19 22:34:44 -04:00
Simon	c99400994b	Modified graph of file types	2018-06-19 20:17:20 -04:00
Simon	76ed03a82e	Dates and sizes graphs styling	2018-06-19 19:44:04 -04:00