simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-04-20 19:06:42 +00:00
Code Issues Projects Releases Wiki Activity
193 Commits 2 Branches 0 Tags
Commit Graph

48 Commits

Author SHA1 Message Date
Simon
ffeed4192e Refresh index before reddit comment callback 2018-08-09 16:19:21 -04:00
Simon
8ffd9179d2 Increased stats timeout value 2018-08-09 14:26:22 -04:00
Simon
1ff1e039f5 Increased stats generation interval 2018-07-25 12:29:14 -04:00
Simon
3d63184287 Increased website scatter size 2018-07-25 11:33:21 -04:00
Simon
fbbe952e4d Stats are generated in background and stored to file instead of on-demand 2018-07-24 20:29:25 -04:00
Simon
004ade8935 Misc bug fixes 2018-07-20 10:35:17 -04:00
Simon
2f1b0c96f1 retry if deleting docs fail 2018-07-14 21:34:25 -04:00
Simon
fe1d29aaea Crawl tasks are now fetched by the crawlers instead of pushed by the server 2018-07-14 17:31:18 -04:00
Simon
d9e9f53f92 Website should stay online even if elasticsearch is down / timing out 2018-07-12 12:06:45 -04:00
Simon
f202caece8 Increased cooldown time for indexing 2018-07-12 11:45:50 -04:00
Simon
c0327fecda Increased delete_docs timeout value 2018-07-12 11:30:16 -04:00
Simon
290322dfa7 Indexing is a bit gentler on server resources and some pages have been memory cached 2018-07-12 11:26:20 -04:00
Simon
1ee1c3c35d safer document deletion 2018-07-10 19:26:06 -04:00
Simon
711e8282ef 'Go to random website' button, and navigation in the website list 2018-07-08 10:42:14 -04:00
Simon
9ff21e7943 Fixed indentation error 2018-06-28 22:47:14 -04:00
Simon
4c9d79fdbf Added filter for large files in stats 2018-06-28 10:40:54 -04:00
Simon
14037c5f21 Added more extension types and adjusted global stats histograms 2018-06-27 11:53:32 -04:00
Simon
6a3d540de2 Added date filter in search options and github banner on homepage 2018-06-27 10:05:33 -04:00
Simon
b570e81bec More search options 2018-06-26 21:38:26 -04:00
Simon
4abd8d12e2 Added size filter 2018-06-26 20:21:24 -04:00
Simon
8ea57967e6 Filter by extension type 2018-06-26 19:02:46 -04:00
Simon
a0bd45c829 Increased ES timeouts 2018-06-26 17:01:17 -04:00
Simon
14d384e366 Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default 2018-06-21 19:44:27 -04:00
Simon
a7e4e3ae1f Fixed stats bug 2018-06-20 18:07:55 -04:00
Simon
cf51bb381c Added top websites scatter graph 2018-06-20 12:21:34 -04:00
Simon
35837463cd Added admin clear & delete buttons for websites 2018-06-20 10:48:51 -04:00
Simon
5f07e7d340 File types color based on type 2018-06-19 22:34:44 -04:00
Simon
c99400994b Modified graph of file types 2018-06-19 20:17:20 -04:00
Simon
76ed03a82e Dates and sizes graphs styling 2018-06-19 19:44:04 -04:00
Simon
8236b04c2e Dates and sizes graphs 2018-06-19 19:04:12 -04:00
Simon
e54609972c Overwrite document on re-index, update website last_modified on task complete, delete website files on index complete 2018-06-19 11:24:28 -04:00
Simon
e5e38a6faf Elasticsearch export to csv 2018-06-19 09:48:44 -04:00
Simon
81d52a4551 Changed UI to fit the-eye.eu 2018-06-18 22:37:05 -04:00
Simon
8768e39f08 Added stats page 2018-06-18 19:56:25 -04:00
Simon
adb94cf326 Should fix memory usage problem when crawling 2018-06-14 23:36:54 -04:00
Simon
9aed18c2d2 Should fix timeout error when indexing 2018-06-14 20:07:50 -04:00
Simon
dffd032659 Indexing after crawling is a bit more efficient 2018-06-14 16:41:43 -04:00
Simon
9bde8cb629 uWSGI config and bugfix with file extensions 2018-06-13 14:11:27 -04:00
Simon
e91572a06f Homepage stats now work with elasticsearch 2018-06-12 23:19:57 -04:00
Simon
2fe81e4b06 Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart 2018-06-12 22:28:36 -04:00
Simon
bccb1d0dfd Website link list works with elasticsearch 2018-06-12 21:26:44 -04:00
Simon
e266a50197 Website stats now works with elasticsearch 2018-06-12 20:17:30 -04:00
Simon
4b60ac62fc Added website url & date in search results & fixed threading problem 2018-06-12 17:48:15 -04:00
Simon
0127b3a51d Basic searching integrated with elasticsearch + highlighting 2018-06-12 16:29:05 -04:00
Simon
af2601ee70 Fixed file duplication problem 2018-06-12 15:55:52 -04:00
Simon
1718bb91ca Files are indexed into ES when task is complete 2018-06-12 15:45:00 -04:00
Simon
011b8455a7 Elasticsearch search engine (search & scroll) 2018-06-11 23:06:41 -04:00
Simon
72495275b0 Elasticsearch search engine (import from json) 2018-06-11 22:35:49 -04:00