simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-11-04 06:56:52 +00:00
Code Issues Projects Releases Wiki Activity
304 Commits 2 Branches 0 Tags
Commit Graph

304 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Simon
a7e4e3ae1f Fixed stats bug 2018-06-20 18:07:55 -04:00
Simon
c5deafbea5 Should fix some odd http listings 2018-06-20 13:34:41 -04:00
Simon
cf51bb381c Added top websites scatter graph 2018-06-20 12:21:34 -04:00
Simon
7400bdc2a9 Added admin blacklist control in dashboard 2018-06-20 11:28:06 -04:00
Simon
35837463cd Added admin clear & delete buttons for websites 2018-06-20 10:48:51 -04:00
Simon
cef9e2c8a1 Added some file types association 2018-06-19 22:41:43 -04:00
Simon
5f07e7d340 File types color based on type 2018-06-19 22:34:44 -04:00
Simon
5afdfb2b3c fixed navbar icon for mobile 2018-06-19 21:13:36 -04:00
Simon
c99400994b Modified graph of file types 2018-06-19 20:17:20 -04:00
Simon
76ed03a82e Dates and sizes graphs styling 2018-06-19 19:44:04 -04:00
Simon
8236b04c2e Dates and sizes graphs 2018-06-19 19:04:12 -04:00
Simon
e0b5aad654 Preview icon for images 2018-06-19 13:56:00 -04:00
Simon
d8486104b4 Fix for odd html listings 2018-06-19 12:14:50 -04:00
Simon
e54609972c Overwrite document on re-index, update website last_modified on task complete, delete website files on index complete 2018-06-19 11:24:28 -04:00
Simon
8486555426 Ignore 'parent directory' links 2018-06-19 10:36:09 -04:00
Simon
8f311e52ee Typo in csv export 2018-06-19 10:17:15 -04:00
Simon
5bdfa9985c Small adjustments for csv export (again) 2018-06-19 10:04:55 -04:00
Simon
4f5f0f76be Small adjustments for csv export 2018-06-19 10:01:15 -04:00
Simon
e5e38a6faf Elasticsearch export to csv 2018-06-19 09:48:44 -04:00
Simon
81d52a4551 Changed UI to fit the-eye.eu 2018-06-18 22:37:05 -04:00
Simon
677bfa03ea Another fix for encoding problems 2018-06-18 20:30:18 -04:00
Simon
788d3749d4 Homepage now compatible with new stats 2018-06-18 20:04:49 -04:00
Simon
8768e39f08 Added stats page 2018-06-18 19:56:25 -04:00
Simon
7923647ea3 Made the ftp crawler work with the latest changes 2018-06-18 15:46:03 -04:00
Simon
83f4b8def9 Enhanced search results page 2018-06-18 15:01:49 -04:00
Simon
8a73142ff8 Support for more than just utf-8 and removed some debug info 2018-06-18 13:44:19 -04:00
Simon
7c47b0f00c Added delta column in crawl logs 2018-06-18 12:21:00 -04:00
Simon
b63c7190c3 Improved external link detection 2018-06-18 12:14:05 -04:00
Simon
400abc9a3c Added crawl logs page 2018-06-18 11:41:26 -04:00
Simon
99d64b658b Disabled thread pool for headers requests in listing 2018-06-18 10:33:33 -04:00
Simon
b97b8f6784 Temporary fix for decoding errors 2018-06-17 22:17:21 -04:00
Simon
344e7274d7 Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations 2018-06-17 22:10:46 -04:00
Simon
07d51a75cc Increased queue.get() timeouts 2018-06-17 10:07:06 -04:00
Simon
e6175c84c9 Re-added timeout that was accidentally deleted 2018-06-16 22:20:15 -04:00
Simon
1283cc9599 Should fix memory usage problem when crawling (part three) 2018-06-16 20:32:50 -04:00
Simon
86144935e3 Attempt to fix Unicode errors part two 2018-06-16 15:30:44 -04:00
Simon
c309aa25c8 Attempt to fix unicode decode errors 2018-06-16 15:20:23 -04:00
Simon
9d0a0a8b42 Should fix memory usage problem when crawling (part two) 2018-06-16 14:53:48 -04:00
Simon
adb94cf326 Should fix memory usage problem when crawling 2018-06-14 23:36:54 -04:00
Simon
9aed18c2d2 Should fix timeout error when indexing 2018-06-14 20:07:50 -04:00
Simon
81fde6cc30 Bug fixes with html parsing 2018-06-14 20:02:06 -04:00
Simon
f3c7b551d2 Some adjustments to make it work on Stretch server 2018-06-14 17:09:05 -04:00
Simon
dffd032659 Indexing after crawling is a bit more efficient 2018-06-14 16:41:43 -04:00
Simon
83ca579ec7 Started working on post-crawl callbacks and basic auth for crawl servers 2018-06-14 15:05:56 -04:00
Simon
1bd58468eb Bug fixes for FTP crawler 2018-06-13 15:54:45 -04:00
Simon
9bde8cb629 uWSGI config and bugfix with file extensions 2018-06-13 14:11:27 -04:00
Simon
e91572a06f Homepage stats now work with elasticsearch 2018-06-12 23:19:57 -04:00
Simon
2fe81e4b06 Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart 2018-06-12 22:28:36 -04:00
Simon
24ef493245 Websites being indexed now show up on the homepage 2018-06-12 21:51:02 -04:00
Simon
bccb1d0dfd Website link list works with elasticsearch 2018-06-12 21:26:44 -04:00