simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-04-24 12:45:51 +00:00
Code Issues Projects Releases Wiki Activity

Commit Graph

  • 53db765856 Static export file now managed by nginx Simon 2018-09-06 19:38:07 -04:00
  • 8b13de4a6b Re-init curl handle on error Simon 2018-08-25 16:46:53 -04:00
  • faa9ac3ccb Closing curl handle manually just to make sure Simon 2018-08-23 12:48:15 -04:00
  • dff4125c9f Bugfix post-pycurl update pt. 3 (Sorry!) Simon 2018-08-23 12:47:17 -04:00
  • 25e1e58828 Bugfix post-pycurl update pt. 3 (Sorry!) Simon 2018-08-23 12:46:00 -04:00
  • 6ffc43601b Bugfix post-pycurl update pt. 2 Simon 2018-08-23 12:40:13 -04:00
  • 484a0baf9d Bugfix post-pycurl update Simon 2018-08-23 12:37:27 -04:00
  • cadaf14c1b Small bugfix Simon 2018-08-23 12:12:23 -04:00
  • 54b4d2d5b4 removed debug lines Simon 2018-08-23 12:02:07 -04:00
  • d42be56dee More debug info Simon 2018-08-23 11:59:13 -04:00
  • 8dc8627f78
         Update README.md Simon Fortier 2018-08-23 11:51:48 -04:00
  • 85c3aa918d replaced requests by pycurl Simon 2018-08-23 11:47:09 -04:00
  • 8f218f3c9d Bug fix for pages buttons pt.2 Simon 2018-08-16 13:24:00 -04:00
  • a2327bac7c Bug fix for pages buttons Simon 2018-08-16 13:13:34 -04:00
  • 6d27cbca02 xz -> lzma for export Simon 2018-08-15 11:32:36 -04:00
  • bbe8ed07a8 Reset page number on search Simon 2018-08-14 16:20:00 -04:00
  • c92f2f4937 Should fix export problem Simon 2018-08-14 12:21:34 -04:00
  • 5c386707ed Should fix import error Simon 2018-08-13 14:03:22 -04:00
  • edede200f4 Decresed number of indexed documents per second Simon 2018-08-12 14:58:27 -04:00
  • cc4c70f400 Request content is read all at once Simon 2018-08-11 13:05:24 -04:00
  • 78d1b7a5bd Next and previous buttons now works with captcha Simon 2018-08-10 16:30:40 -04:00
  • bab68819df Increased stats generation interval Simon 2018-08-10 15:27:37 -04:00
  • aab1abba54 Fixed websites link Simon 2018-08-10 15:24:43 -04:00
  • c29af180c5 Captcha for searches Simon 2018-08-10 12:46:40 -04:00
  • c94cf5b313 Adjusted timeout values (again) Simon 2018-08-10 11:46:16 -04:00
  • a6b1d9cba3 More help when no search results Simon 2018-08-09 21:43:07 -04:00
  • faeff701de Increased search timeout value Simon 2018-08-09 18:33:35 -04:00
  • 42d858b62a Queue can be emptied more easily pt.2 Simon 2018-08-09 17:14:17 -04:00
  • 5a084cb857 Queue can be emptied more easily Simon 2018-08-09 17:12:43 -04:00
  • ffeed4192e Refresh index before reddit comment callback Simon 2018-08-09 16:19:21 -04:00
  • 8ffd9179d2 Increased stats timeout value Simon 2018-08-09 14:26:22 -04:00
  • f729b462f0 od_util can be used when od-database is a submodule part 2 Simon 2018-08-08 23:31:50 -04:00
  • 88166054ad od_util can be used when od-database is a submodule Simon 2018-08-08 23:07:09 -04:00
  • 89e378ffd9 Reddit comment callback is not an edit instead of a new comment Simon 2018-08-08 22:41:25 -04:00
  • 458641654c Minimal configuration for reddit comment callback Simon 2018-08-08 21:24:55 -04:00
  • 1ff1e039f5 Increased stats generation interval Simon 2018-07-25 12:29:14 -04:00
  • 65738b0e70 Small fix Simon 2018-07-25 12:05:07 -04:00
  • 5ff198b88a Fix for negative sizes Simon 2018-07-25 11:37:12 -04:00
  • 49206af566 Updated requirements Simon 2018-07-25 11:35:41 -04:00
  • 3d63184287 Increased website scatter size Simon 2018-07-25 11:33:21 -04:00
  • 34d1f375a8 Crawler performance improvements Simon 2018-07-25 11:27:50 -04:00
  • fbbe952e4d Stats are generated in background and stored to file instead of on-demand Simon 2018-07-24 20:08:07 -04:00
  • bf82478fee
         Delete blacklist.txt Simon Fortier 2018-07-21 11:23:59 -04:00
  • f12d5d524a exceptions during push_result are logged instead of raised Simon 2018-07-21 10:45:17 -04:00
  • d43cf3b0ce Empty queue timeout increased to avoid that all workers die before the website is dropped Simon 2018-07-20 14:11:17 -04:00
  • d3801adf74 Typo Simon 2018-07-20 13:39:23 -04:00
  • 1df5d194d2 Very slow websites are skipped. Should fix infinite waiting bug Simon 2018-07-20 13:34:40 -04:00
  • 004ade8935 Misc bug fixes Simon 2018-07-20 10:35:17 -04:00
  • df5b01dc83 Bug when directory is empty with new file upload (server side) Simon 2018-07-17 18:28:50 -04:00
  • 8ef1d36c9d Bug when directory is empty with new file upload Simon 2018-07-17 18:24:05 -04:00
  • 898ffcf410 File upload is made in small chunks Simon 2018-07-17 17:52:17 -04:00
  • 73afebec28 Added API commands Simon 2018-07-17 13:12:20 -04:00
  • 55a0fde19d Skip 'Parent directory' links more efficiently Simon 2018-07-17 11:20:58 -04:00
  • 756e331c83 Fixed bug in crawler when file count in a directory is greater than 150 Simon 2018-07-17 11:03:10 -04:00
  • cf96d1697d Fixed bug when submitting Simon 2018-07-16 20:34:42 -04:00
  • a8a658f55b Crawl server names that are numeric now show up in stats page Simon 2018-07-15 21:33:37 -04:00
  • 3b3661cae2 Typo Simon 2018-07-15 21:23:49 -04:00
  • 0227684a53 Added API commands Simon 2018-07-15 21:21:57 -04:00
  • 8a19fa0ce7 Decreased bulk enqueue limit Simon 2018-07-15 15:31:21 -04:00
  • e4cb91376f Fixed typo in readme Simon 2018-07-15 12:54:28 -04:00
  • c35491cb15 Multi threading for bulk enqueue Simon 2018-07-15 12:50:26 -04:00
  • 08c3e119f0 Typo Simon 2018-07-15 10:52:04 -04:00
  • 112400886e Crawler no longer crashes when website has no files Simon 2018-07-15 10:46:48 -04:00
  • e18ded7ac1 Temporarily removed logger in async methods (https://stackoverflow.com/questions/37907350) Simon 2018-07-15 10:35:13 -04:00
  • 152a6f20fb Re-enabled multi threaded file requests for large directories Simon 2018-07-15 08:54:36 -04:00
  • 2f1b0c96f1 retry if deleting docs fail Simon 2018-07-14 21:34:25 -04:00
  • f452d0f8b2 file lists now deleted after indexing Simon 2018-07-14 20:41:20 -04:00
  • 51a47b3628 Removed debug line Simon 2018-07-14 17:36:16 -04:00
  • fe1d29aaea Crawl tasks are now fetched by the crawlers instead of pushed by the server Simon 2018-07-14 17:31:18 -04:00
  • d9e9f53f92 Website should stay online even if elasticsearch is down / timing out Simon 2018-07-12 12:06:45 -04:00
  • f202caece8 Increased cooldown time for indexing Simon 2018-07-12 11:45:50 -04:00
  • c0327fecda Increased delete_docs timeout value Simon 2018-07-12 11:30:16 -04:00
  • 290322dfa7 Indexing is a bit gentler on server resources and some pages have been memory cached Simon 2018-07-12 11:26:20 -04:00
  • 1b743e7aba Updated README and catpchas are now toggled from the config Simon 2018-07-10 22:57:52 -04:00
  • 123f38e65d Updated README with web server information Simon 2018-07-10 22:35:54 -04:00
  • 2b83698292 Changed 'no results' text in search page Simon 2018-07-10 21:09:21 -04:00
  • 1ee1c3c35d safer document deletion Simon 2018-07-10 19:26:06 -04:00
  • 35b40f002d Modified timeout values Simon 2018-07-10 13:07:54 -04:00
  • d138db8f06 Added filter to check if a website can be scanned from its parent directory Simon 2018-07-10 10:14:23 -04:00
  • f226b82f5a Increased website count per page in /website/ Simon 2018-07-08 10:44:25 -04:00
  • 711e8282ef 'Go to random website' button, and navigation in the website list Simon 2018-07-08 10:42:14 -04:00
  • 9ff21e7943 Fixed indentation error Simon 2018-06-28 22:47:14 -04:00
  • 4c9d79fdbf Added filter for large files in stats Simon 2018-06-28 10:40:54 -04:00
  • 2638e47360 Only log searches in es Simon 2018-06-27 15:39:48 -04:00
  • 5383ad6aea Searches are not saved to database Simon 2018-06-27 15:29:50 -04:00
  • 14037c5f21 Added more extension types and adjusted global stats histograms Simon 2018-06-27 11:53:32 -04:00
  • 10e1afb2e4 Small fix to allow uppercase in extension names Simon 2018-06-27 10:12:59 -04:00
  • 6a3d540de2 Added date filter in search options and github banner on homepage Simon 2018-06-27 10:05:33 -04:00
  • b570e81bec More search options Simon 2018-06-26 21:38:26 -04:00
  • b1ad39c204 bugfix when crawl server is timing out Simon 2018-06-26 20:25:28 -04:00
  • 4abd8d12e2 Added size filter Simon 2018-06-26 20:21:24 -04:00
  • f859bd3f8d Removed unmaintained tests Simon 2018-06-26 19:07:14 -04:00
  • 8ea57967e6 Filter by extension type Simon 2018-06-26 19:02:46 -04:00
  • a0bd45c829 Increased ES timeouts Simon 2018-06-26 17:01:17 -04:00
  • e384efd403 Bugfix for http crawler Simon 2018-06-25 20:36:31 -04:00
  • d7ce1670a8 Logging and bugfix for http crawler Simon 2018-06-25 14:36:16 -04:00
  • 5fd00f22af Task logs now stored on main server Simon 2018-06-24 20:32:02 -04:00
  • 059d9fd366 Added button to queue empty websites Simon 2018-06-24 19:33:15 -04:00
  • f6ee338c0f Removed unused statement Simon 2018-06-24 18:24:41 -04:00
  • e11343de23 More FTP crawler bug fixes Simon 2018-06-24 18:05:30 -04:00