276 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Simon	e384efd403	Bugfix for http crawler	2018-06-25 20:36:31 -04:00
Simon	d7ce1670a8	Logging and bugfix for http crawler	2018-06-25 14:36:16 -04:00
Simon	5fd00f22af	Task logs now stored on main server	2018-06-24 20:32:02 -04:00
Simon	059d9fd366	Added button to queue empty websites	2018-06-24 19:33:15 -04:00
Simon	f6ee338c0f	Removed unused statement	2018-06-24 18:24:41 -04:00
Simon	e11343de23	More FTP crawler bug fixes	2018-06-24 18:05:30 -04:00
Simon	ab35ce96cc	FTP crawler bug fixes	2018-06-24 16:44:21 -04:00
Simon	f603f41754	Updated readme	2018-06-24 14:27:44 -04:00
Simon	8e937e69c0	Should fix some FTP errors	2018-06-24 13:50:55 -04:00
Simon	a6d753c6ee	Added redispatch button and fixed typo in load balancing code	2018-06-24 10:07:46 -04:00
Simon	1ac510ff53	Slots can be updated without removing & adding	2018-06-24 09:39:44 -04:00
Simon	348914aba9	Removing unused module	2018-06-22 17:34:10 -04:00
Simon	e824b2bf3c	Updated readme and UI fixes	2018-06-22 13:22:58 -04:00
Simon	9d3fc2d71b	typo (again)	2018-06-21 21:26:44 -04:00
Simon	efd1981e6f	typo	2018-06-21 21:00:50 -04:00
Simon	7a4432e4d0	More bugfixes for looping directories, some work on task dispatching	2018-06-21 20:50:26 -04:00
Simon	14d384e366	Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default	2018-06-21 19:44:27 -04:00
Simon	098ad2be72	Should fix unknown encoding errors + removed https warnings	2018-06-21 19:23:01 -04:00
Simon	80aa8933e6	Added rescan button	2018-06-21 13:02:16 -04:00
Simon	073551df3c	Attempt to handle looping directories	2018-06-21 11:54:40 -04:00
Simon	dd93d40a55	Small bugfix for ftp crawler	2018-06-20 21:56:38 -04:00
Simon	4ca56d2317	Fixes #2	2018-06-20 21:52:45 -04:00
Simon	a7e4e3ae1f	Fixed stats bug	2018-06-20 18:07:55 -04:00
Simon	c5deafbea5	Should fix some odd http listings	2018-06-20 13:34:41 -04:00
Simon	cf51bb381c	Added top websites scatter graph	2018-06-20 12:21:34 -04:00
Simon	7400bdc2a9	Added admin blacklist control in dashboard	2018-06-20 11:28:06 -04:00
Simon	35837463cd	Added admin clear & delete buttons for websites	2018-06-20 10:48:51 -04:00
Simon	cef9e2c8a1	Added some file types association	2018-06-19 22:41:43 -04:00
Simon	5f07e7d340	File types color based on type	2018-06-19 22:34:44 -04:00
Simon	5afdfb2b3c	fixed navbar icon for mobile	2018-06-19 21:13:36 -04:00
Simon	c99400994b	Modified graph of file types	2018-06-19 20:17:20 -04:00
Simon	76ed03a82e	Dates and sizes graphs styling	2018-06-19 19:44:04 -04:00
Simon	8236b04c2e	Dates and sizes graphs	2018-06-19 19:04:12 -04:00
Simon	e0b5aad654	Preview icon for images	2018-06-19 13:56:00 -04:00
Simon	d8486104b4	Fix for odd html listings	2018-06-19 12:14:50 -04:00
Simon	e54609972c	Overwrite document on re-index, update website last_modified on task complete, delete website files on index complete	2018-06-19 11:24:28 -04:00
Simon	8486555426	Ignore 'parent directory' links	2018-06-19 10:36:09 -04:00
Simon	8f311e52ee	Typo in csv export	2018-06-19 10:17:15 -04:00
Simon	5bdfa9985c	Small adjustments for csv export (again)	2018-06-19 10:04:55 -04:00
Simon	4f5f0f76be	Small adjustments for csv export	2018-06-19 10:01:15 -04:00
Simon	e5e38a6faf	Elasticsearch export to csv	2018-06-19 09:48:44 -04:00
Simon	81d52a4551	Changed UI to fit the-eye.eu	2018-06-18 22:37:05 -04:00
Simon	677bfa03ea	Another fix for encoding problems	2018-06-18 20:30:18 -04:00
Simon	788d3749d4	Homepage now compatible with new stats	2018-06-18 20:04:49 -04:00
Simon	8768e39f08	Added stats page	2018-06-18 19:56:25 -04:00
Simon	7923647ea3	Made the ftp crawler work with the latest changes	2018-06-18 15:46:03 -04:00
Simon	83f4b8def9	Enhanced search results page	2018-06-18 15:01:49 -04:00
Simon	8a73142ff8	Support for more than just utf-8 and removed some debug info	2018-06-18 13:44:19 -04:00
Simon	7c47b0f00c	Added delta column in crawl logs	2018-06-18 12:21:00 -04:00
Simon	b63c7190c3	Improved external link detection	2018-06-18 12:14:05 -04:00