simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-12-13 06:49:02 +00:00
Code Issues Projects Releases Wiki Activity

Commit Graph

  • bbe8ed07a8 Reset page number on search Simon 2018-08-14 16:20:00 -04:00
  • c92f2f4937 Should fix export problem Simon 2018-08-14 12:21:34 -04:00
  • 5c386707ed Should fix import error Simon 2018-08-13 14:03:22 -04:00
  • edede200f4 Decresed number of indexed documents per second Simon 2018-08-12 14:58:27 -04:00
  • cc4c70f400 Request content is read all at once Simon 2018-08-11 13:05:24 -04:00
  • 78d1b7a5bd Next and previous buttons now works with captcha Simon 2018-08-10 16:30:40 -04:00
  • bab68819df Increased stats generation interval Simon 2018-08-10 15:27:37 -04:00
  • aab1abba54 Fixed websites link Simon 2018-08-10 15:24:43 -04:00
  • c29af180c5 Captcha for searches Simon 2018-08-10 12:46:40 -04:00
  • c94cf5b313 Adjusted timeout values (again) Simon 2018-08-10 11:46:16 -04:00
  • a6b1d9cba3 More help when no search results Simon 2018-08-09 21:43:07 -04:00
  • faeff701de Increased search timeout value Simon 2018-08-09 18:33:35 -04:00
  • 42d858b62a Queue can be emptied more easily pt.2 Simon 2018-08-09 17:14:17 -04:00
  • 5a084cb857 Queue can be emptied more easily Simon 2018-08-09 17:12:43 -04:00
  • ffeed4192e Refresh index before reddit comment callback Simon 2018-08-09 16:19:21 -04:00
  • 8ffd9179d2 Increased stats timeout value Simon 2018-08-09 14:26:22 -04:00
  • f729b462f0 od_util can be used when od-database is a submodule part 2 Simon 2018-08-08 23:31:50 -04:00
  • 88166054ad od_util can be used when od-database is a submodule Simon 2018-08-08 23:07:09 -04:00
  • 89e378ffd9 Reddit comment callback is not an edit instead of a new comment Simon 2018-08-08 22:41:25 -04:00
  • 458641654c Minimal configuration for reddit comment callback Simon 2018-08-08 21:24:55 -04:00
  • 1ff1e039f5 Increased stats generation interval Simon 2018-07-25 12:29:14 -04:00
  • 65738b0e70 Small fix Simon 2018-07-25 12:05:07 -04:00
  • 5ff198b88a Fix for negative sizes Simon 2018-07-25 11:37:12 -04:00
  • 49206af566 Updated requirements Simon 2018-07-25 11:35:41 -04:00
  • 3d63184287 Increased website scatter size Simon 2018-07-25 11:33:21 -04:00
  • 34d1f375a8 Crawler performance improvements Simon 2018-07-25 11:27:50 -04:00
  • fbbe952e4d Stats are generated in background and stored to file instead of on-demand Simon 2018-07-24 20:08:07 -04:00
  • bf82478fee Delete blacklist.txt Simon Fortier 2018-07-21 11:23:59 -04:00
  • f12d5d524a exceptions during push_result are logged instead of raised Simon 2018-07-21 10:45:17 -04:00
  • d43cf3b0ce Empty queue timeout increased to avoid that all workers die before the website is dropped Simon 2018-07-20 14:11:17 -04:00
  • d3801adf74 Typo Simon 2018-07-20 13:39:23 -04:00
  • 1df5d194d2 Very slow websites are skipped. Should fix infinite waiting bug Simon 2018-07-20 13:34:40 -04:00
  • 004ade8935 Misc bug fixes Simon 2018-07-20 10:35:17 -04:00
  • df5b01dc83 Bug when directory is empty with new file upload (server side) Simon 2018-07-17 18:28:50 -04:00
  • 8ef1d36c9d Bug when directory is empty with new file upload Simon 2018-07-17 18:24:05 -04:00
  • 898ffcf410 File upload is made in small chunks Simon 2018-07-17 17:52:17 -04:00
  • 73afebec28 Added API commands Simon 2018-07-17 13:12:20 -04:00
  • 55a0fde19d Skip 'Parent directory' links more efficiently Simon 2018-07-17 11:20:58 -04:00
  • 756e331c83 Fixed bug in crawler when file count in a directory is greater than 150 Simon 2018-07-17 11:03:10 -04:00
  • cf96d1697d Fixed bug when submitting Simon 2018-07-16 20:34:42 -04:00
  • a8a658f55b Crawl server names that are numeric now show up in stats page Simon 2018-07-15 21:33:37 -04:00
  • 3b3661cae2 Typo Simon 2018-07-15 21:23:49 -04:00
  • 0227684a53 Added API commands Simon 2018-07-15 21:21:57 -04:00
  • 8a19fa0ce7 Decreased bulk enqueue limit Simon 2018-07-15 15:31:21 -04:00
  • e4cb91376f Fixed typo in readme Simon 2018-07-15 12:54:28 -04:00
  • c35491cb15 Multi threading for bulk enqueue Simon 2018-07-15 12:50:26 -04:00
  • 08c3e119f0 Typo Simon 2018-07-15 10:52:04 -04:00
  • 112400886e Crawler no longer crashes when website has no files Simon 2018-07-15 10:46:48 -04:00
  • e18ded7ac1 Temporarily removed logger in async methods (https://stackoverflow.com/questions/37907350) Simon 2018-07-15 10:35:13 -04:00
  • 152a6f20fb Re-enabled multi threaded file requests for large directories Simon 2018-07-15 08:54:36 -04:00
  • 2f1b0c96f1 retry if deleting docs fail Simon 2018-07-14 21:34:25 -04:00
  • f452d0f8b2 file lists now deleted after indexing Simon 2018-07-14 20:41:20 -04:00
  • 51a47b3628 Removed debug line Simon 2018-07-14 17:36:16 -04:00
  • fe1d29aaea Crawl tasks are now fetched by the crawlers instead of pushed by the server Simon 2018-07-14 17:31:18 -04:00
  • d9e9f53f92 Website should stay online even if elasticsearch is down / timing out Simon 2018-07-12 12:06:45 -04:00
  • f202caece8 Increased cooldown time for indexing Simon 2018-07-12 11:45:50 -04:00
  • c0327fecda Increased delete_docs timeout value Simon 2018-07-12 11:30:16 -04:00
  • 290322dfa7 Indexing is a bit gentler on server resources and some pages have been memory cached Simon 2018-07-12 11:26:20 -04:00
  • 1b743e7aba Updated README and catpchas are now toggled from the config Simon 2018-07-10 22:57:52 -04:00
  • 123f38e65d Updated README with web server information Simon 2018-07-10 22:35:54 -04:00
  • 2b83698292 Changed 'no results' text in search page Simon 2018-07-10 21:09:21 -04:00
  • 1ee1c3c35d safer document deletion Simon 2018-07-10 19:26:06 -04:00
  • 35b40f002d Modified timeout values Simon 2018-07-10 13:07:54 -04:00
  • d138db8f06 Added filter to check if a website can be scanned from its parent directory Simon 2018-07-10 10:14:23 -04:00
  • f226b82f5a Increased website count per page in /website/ Simon 2018-07-08 10:44:25 -04:00
  • 711e8282ef 'Go to random website' button, and navigation in the website list Simon 2018-07-08 10:42:14 -04:00
  • 9ff21e7943 Fixed indentation error Simon 2018-06-28 22:47:14 -04:00
  • 4c9d79fdbf Added filter for large files in stats Simon 2018-06-28 10:40:54 -04:00
  • 2638e47360 Only log searches in es Simon 2018-06-27 15:39:48 -04:00
  • 5383ad6aea Searches are not saved to database Simon 2018-06-27 15:29:50 -04:00
  • 14037c5f21 Added more extension types and adjusted global stats histograms Simon 2018-06-27 11:53:32 -04:00
  • 10e1afb2e4 Small fix to allow uppercase in extension names Simon 2018-06-27 10:12:59 -04:00
  • 6a3d540de2 Added date filter in search options and github banner on homepage Simon 2018-06-27 10:05:33 -04:00
  • b570e81bec More search options Simon 2018-06-26 21:38:26 -04:00
  • b1ad39c204 bugfix when crawl server is timing out Simon 2018-06-26 20:25:28 -04:00
  • 4abd8d12e2 Added size filter Simon 2018-06-26 20:21:24 -04:00
  • f859bd3f8d Removed unmaintained tests Simon 2018-06-26 19:07:14 -04:00
  • 8ea57967e6 Filter by extension type Simon 2018-06-26 19:02:46 -04:00
  • a0bd45c829 Increased ES timeouts Simon 2018-06-26 17:01:17 -04:00
  • e384efd403 Bugfix for http crawler Simon 2018-06-25 20:36:31 -04:00
  • d7ce1670a8 Logging and bugfix for http crawler Simon 2018-06-25 14:36:16 -04:00
  • 5fd00f22af Task logs now stored on main server Simon 2018-06-24 20:32:02 -04:00
  • 059d9fd366 Added button to queue empty websites Simon 2018-06-24 19:33:15 -04:00
  • f6ee338c0f Removed unused statement Simon 2018-06-24 18:24:41 -04:00
  • e11343de23 More FTP crawler bug fixes Simon 2018-06-24 18:05:30 -04:00
  • ab35ce96cc FTP crawler bug fixes Simon 2018-06-24 16:44:21 -04:00
  • f603f41754 Updated readme Simon 2018-06-24 14:27:44 -04:00
  • 8e937e69c0 Should fix some FTP errors Simon 2018-06-24 13:50:55 -04:00
  • a6d753c6ee Added redispatch button and fixed typo in load balancing code Simon 2018-06-24 10:07:46 -04:00
  • 1ac510ff53 Slots can be updated without removing & adding Simon 2018-06-24 09:39:44 -04:00
  • 348914aba9 Removing unused module Simon 2018-06-22 17:34:10 -04:00
  • e824b2bf3c Updated readme and UI fixes Simon 2018-06-22 13:22:58 -04:00
  • 9d3fc2d71b typo (again) Simon 2018-06-21 21:26:44 -04:00
  • efd1981e6f typo Simon 2018-06-21 21:00:50 -04:00
  • 7a4432e4d0 More bugfixes for looping directories, some work on task dispatching Simon 2018-06-21 20:50:26 -04:00
  • 14d384e366 Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default Simon 2018-06-21 19:44:27 -04:00
  • 098ad2be72 Should fix unknown encoding errors + removed https warnings Simon 2018-06-21 19:23:01 -04:00
  • 80aa8933e6 Added rescan button Simon 2018-06-21 13:02:16 -04:00
  • 073551df3c Attempt to handle looping directories Simon 2018-06-21 11:54:40 -04:00
  • dd93d40a55 Small bugfix for ftp crawler Simon 2018-06-20 21:56:38 -04:00