simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-04-19 10:26:44 +00:00
Code Issues Projects Releases Wiki Activity
267 Commits 2 Branches 0 Tags
Commit Graph

267 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Simon
a6d753c6ee Added redispatch button and fixed typo in load balancing code 2018-06-24 10:07:46 -04:00
Simon
1ac510ff53 Slots can be updated without removing & adding 2018-06-24 09:39:44 -04:00
Simon
348914aba9 Removing unused module 2018-06-22 17:34:10 -04:00
Simon
e824b2bf3c Updated readme and UI fixes 2018-06-22 13:22:58 -04:00
Simon
9d3fc2d71b typo (again) 2018-06-21 21:26:44 -04:00
Simon
efd1981e6f typo 2018-06-21 21:00:50 -04:00
Simon
7a4432e4d0 More bugfixes for looping directories, some work on task dispatching 2018-06-21 20:50:26 -04:00
Simon
14d384e366 Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default 2018-06-21 19:44:27 -04:00
Simon
098ad2be72 Should fix unknown encoding errors + removed https warnings 2018-06-21 19:23:01 -04:00
Simon
80aa8933e6 Added rescan button 2018-06-21 13:02:16 -04:00
Simon
073551df3c Attempt to handle looping directories 2018-06-21 11:54:40 -04:00
Simon
dd93d40a55 Small bugfix for ftp crawler 2018-06-20 21:56:38 -04:00
Simon
4ca56d2317 Fixes #2 2018-06-20 21:52:45 -04:00
Simon
a7e4e3ae1f Fixed stats bug 2018-06-20 18:07:55 -04:00
Simon
c5deafbea5 Should fix some odd http listings 2018-06-20 13:34:41 -04:00
Simon
cf51bb381c Added top websites scatter graph 2018-06-20 12:21:34 -04:00
Simon
7400bdc2a9 Added admin blacklist control in dashboard 2018-06-20 11:28:06 -04:00
Simon
35837463cd Added admin clear & delete buttons for websites 2018-06-20 10:48:51 -04:00
Simon
cef9e2c8a1 Added some file types association 2018-06-19 22:41:43 -04:00
Simon
5f07e7d340 File types color based on type 2018-06-19 22:34:44 -04:00
Simon
5afdfb2b3c fixed navbar icon for mobile 2018-06-19 21:13:36 -04:00
Simon
c99400994b Modified graph of file types 2018-06-19 20:17:20 -04:00
Simon
76ed03a82e Dates and sizes graphs styling 2018-06-19 19:44:04 -04:00
Simon
8236b04c2e Dates and sizes graphs 2018-06-19 19:04:12 -04:00
Simon
e0b5aad654 Preview icon for images 2018-06-19 13:56:00 -04:00
Simon
d8486104b4 Fix for odd html listings 2018-06-19 12:14:50 -04:00
Simon
e54609972c Overwrite document on re-index, update website last_modified on task complete, delete website files on index complete 2018-06-19 11:24:28 -04:00
Simon
8486555426 Ignore 'parent directory' links 2018-06-19 10:36:09 -04:00
Simon
8f311e52ee Typo in csv export 2018-06-19 10:17:15 -04:00
Simon
5bdfa9985c Small adjustments for csv export (again) 2018-06-19 10:04:55 -04:00
Simon
4f5f0f76be Small adjustments for csv export 2018-06-19 10:01:15 -04:00
Simon
e5e38a6faf Elasticsearch export to csv 2018-06-19 09:48:44 -04:00
Simon
81d52a4551 Changed UI to fit the-eye.eu 2018-06-18 22:37:05 -04:00
Simon
677bfa03ea Another fix for encoding problems 2018-06-18 20:30:18 -04:00
Simon
788d3749d4 Homepage now compatible with new stats 2018-06-18 20:04:49 -04:00
Simon
8768e39f08 Added stats page 2018-06-18 19:56:25 -04:00
Simon
7923647ea3 Made the ftp crawler work with the latest changes 2018-06-18 15:46:03 -04:00
Simon
83f4b8def9 Enhanced search results page 2018-06-18 15:01:49 -04:00
Simon
8a73142ff8 Support for more than just utf-8 and removed some debug info 2018-06-18 13:44:19 -04:00
Simon
7c47b0f00c Added delta column in crawl logs 2018-06-18 12:21:00 -04:00
Simon
b63c7190c3 Improved external link detection 2018-06-18 12:14:05 -04:00
Simon
400abc9a3c Added crawl logs page 2018-06-18 11:41:26 -04:00
Simon
99d64b658b Disabled thread pool for headers requests in listing 2018-06-18 10:33:33 -04:00
Simon
b97b8f6784 Temporary fix for decoding errors 2018-06-17 22:17:21 -04:00
Simon
344e7274d7 Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations 2018-06-17 22:10:46 -04:00
Simon
07d51a75cc Increased queue.get() timeouts 2018-06-17 10:07:06 -04:00
Simon
e6175c84c9 Re-added timeout that was accidentally deleted 2018-06-16 22:20:15 -04:00
Simon
1283cc9599 Should fix memory usage problem when crawling (part three) 2018-06-16 20:32:50 -04:00
Simon
86144935e3 Attempt to fix Unicode errors part two 2018-06-16 15:30:44 -04:00
Simon
c309aa25c8 Attempt to fix unicode decode errors 2018-06-16 15:20:23 -04:00