simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-04-19 18:36:44 +00:00
Code Issues Projects Releases Wiki Activity
183 Commits 2 Branches 0 Tags
Commit Graph

69 Commits

Author SHA1 Message Date
Simon
fbbe952e4d Stats are generated in background and stored to file instead of on-demand 2018-07-24 20:29:25 -04:00
Simon
df5b01dc83 Bug when directory is empty with new file upload (server side) 2018-07-17 18:28:50 -04:00
Simon
898ffcf410 File upload is made in small chunks 2018-07-17 17:52:17 -04:00
Simon
73afebec28 Added API commands 2018-07-17 13:12:20 -04:00
Simon
cf96d1697d Fixed bug when submitting 2018-07-16 20:34:42 -04:00
Simon
3b3661cae2 Typo 2018-07-15 21:23:49 -04:00
Simon
0227684a53 Added API commands 2018-07-15 21:21:57 -04:00
Simon
8a19fa0ce7 Decreased bulk enqueue limit 2018-07-15 15:31:21 -04:00
Simon
c35491cb15 Multi threading for bulk enqueue 2018-07-15 12:50:26 -04:00
Simon
08c3e119f0 Typo 2018-07-15 10:52:04 -04:00
Simon
112400886e Crawler no longer crashes when website has no files 2018-07-15 10:46:48 -04:00
Simon
fe1d29aaea Crawl tasks are now fetched by the crawlers instead of pushed by the server 2018-07-14 17:31:18 -04:00
Simon
d9e9f53f92 Website should stay online even if elasticsearch is down / timing out 2018-07-12 12:06:45 -04:00
Simon
290322dfa7 Indexing is a bit gentler on server resources and some pages have been memory cached 2018-07-12 11:26:20 -04:00
Simon
1b743e7aba Updated README and catpchas are now toggled from the config 2018-07-10 22:57:52 -04:00
Simon
d138db8f06 Added filter to check if a website can be scanned from its parent directory 2018-07-10 10:14:23 -04:00
Simon
f226b82f5a Increased website count per page in /website/ 2018-07-08 10:44:25 -04:00
Simon
711e8282ef 'Go to random website' button, and navigation in the website list 2018-07-08 10:42:14 -04:00
Simon
4c9d79fdbf Added filter for large files in stats 2018-06-28 10:40:54 -04:00
Simon
2638e47360 Only log searches in es 2018-06-27 15:39:48 -04:00
Simon
5383ad6aea Searches are not saved to database 2018-06-27 15:29:50 -04:00
Simon
10e1afb2e4 Small fix to allow uppercase in extension names 2018-06-27 10:12:59 -04:00
Simon
6a3d540de2 Added date filter in search options and github banner on homepage 2018-06-27 10:05:33 -04:00
Simon
b570e81bec More search options 2018-06-26 21:38:26 -04:00
Simon
4abd8d12e2 Added size filter 2018-06-26 20:21:24 -04:00
Simon
8ea57967e6 Filter by extension type 2018-06-26 19:02:46 -04:00
Simon
5fd00f22af Task logs now stored on main server 2018-06-24 20:32:02 -04:00
Simon
059d9fd366 Added button to queue empty websites 2018-06-24 19:33:15 -04:00
Simon
a6d753c6ee Added redispatch button and fixed typo in load balancing code 2018-06-24 10:07:46 -04:00
Simon
1ac510ff53 Slots can be updated without removing & adding 2018-06-24 09:39:44 -04:00
Simon
e824b2bf3c Updated readme and UI fixes 2018-06-22 13:22:58 -04:00
Simon
14d384e366 Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default 2018-06-21 19:44:27 -04:00
Simon
80aa8933e6 Added rescan button 2018-06-21 13:02:16 -04:00
Simon
cf51bb381c Added top websites scatter graph 2018-06-20 12:21:34 -04:00
Simon
7400bdc2a9 Added admin blacklist control in dashboard 2018-06-20 11:28:06 -04:00
Simon
35837463cd Added admin clear & delete buttons for websites 2018-06-20 10:48:51 -04:00
Simon
8768e39f08 Added stats page 2018-06-18 19:56:25 -04:00
Simon
83f4b8def9 Enhanced search results page 2018-06-18 15:01:49 -04:00
Simon
400abc9a3c Added crawl logs page 2018-06-18 11:41:26 -04:00
Simon
1283cc9599 Should fix memory usage problem when crawling (part three) 2018-06-16 20:32:50 -04:00
Simon
83ca579ec7 Started working on post-crawl callbacks and basic auth for crawl servers 2018-06-14 15:05:56 -04:00
Simon
9bde8cb629 uWSGI config and bugfix with file extensions 2018-06-13 14:11:27 -04:00
Simon
e91572a06f Homepage stats now work with elasticsearch 2018-06-12 23:19:57 -04:00
Simon
2fe81e4b06 Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart 2018-06-12 22:28:36 -04:00
Simon
24ef493245 Websites being indexed now show up on the homepage 2018-06-12 21:51:02 -04:00
Simon
bccb1d0dfd Website link list works with elasticsearch 2018-06-12 21:26:44 -04:00
Simon
e266a50197 Website stats now works with elasticsearch 2018-06-12 20:17:30 -04:00
Simon
4b60ac62fc Added website url & date in search results & fixed threading problem 2018-06-12 17:48:15 -04:00
Simon
0127b3a51d Basic searching integrated with elasticsearch + highlighting 2018-06-12 16:29:05 -04:00
Simon
d61fd75890 Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s) 2018-06-12 13:44:03 -04:00