simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-04-19 18:36:44 +00:00
Code Issues Projects Releases Wiki Activity
236 Commits 2 Branches 0 Tags
Commit Graph

34 Commits

Author SHA1 Message Date
Simon
d42be56dee More debug info 2018-08-23 11:59:23 -04:00
Simon
f12d5d524a exceptions during push_result are logged instead of raised 2018-07-21 10:45:17 -04:00
Simon
004ade8935 Misc bug fixes 2018-07-20 10:35:17 -04:00
Simon
8ef1d36c9d Bug when directory is empty with new file upload 2018-07-17 18:24:05 -04:00
Simon
898ffcf410 File upload is made in small chunks 2018-07-17 17:52:17 -04:00
Simon
112400886e Crawler no longer crashes when website has no files 2018-07-15 10:46:48 -04:00
Simon
e18ded7ac1 Temporarily removed logger in async methods (https://stackoverflow.com/questions/37907350) 2018-07-15 10:35:13 -04:00
Simon
f452d0f8b2 file lists now deleted after indexing 2018-07-14 20:41:20 -04:00
Simon
51a47b3628 Removed debug line 2018-07-14 17:36:16 -04:00
Simon
fe1d29aaea Crawl tasks are now fetched by the crawlers instead of pushed by the server 2018-07-14 17:31:18 -04:00
Simon
d7ce1670a8 Logging and bugfix for http crawler 2018-06-25 14:36:16 -04:00
Simon
5fd00f22af Task logs now stored on main server 2018-06-24 20:32:02 -04:00
Simon
8e937e69c0 Should fix some FTP errors 2018-06-24 13:50:55 -04:00
Simon
a6d753c6ee Added redispatch button and fixed typo in load balancing code 2018-06-24 10:07:46 -04:00
Simon
e824b2bf3c Updated readme and UI fixes 2018-06-22 13:22:58 -04:00
Simon
8768e39f08 Added stats page 2018-06-18 19:56:25 -04:00
Simon
8a73142ff8 Support for more than just utf-8 and removed some debug info 2018-06-18 13:44:19 -04:00
Simon
400abc9a3c Added crawl logs page 2018-06-18 11:41:26 -04:00
Simon
344e7274d7 Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations 2018-06-17 22:10:46 -04:00
Simon
9d0a0a8b42 Should fix memory usage problem when crawling (part two) 2018-06-16 14:53:48 -04:00
Simon
adb94cf326 Should fix memory usage problem when crawling 2018-06-14 23:36:54 -04:00
Simon
81fde6cc30 Bug fixes with html parsing 2018-06-14 20:02:06 -04:00
Simon
83ca579ec7 Started working on post-crawl callbacks and basic auth for crawl servers 2018-06-14 15:05:56 -04:00
Simon
1bd58468eb Bug fixes for FTP crawler 2018-06-13 15:54:45 -04:00
Simon
2fe81e4b06 Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart 2018-06-12 22:28:36 -04:00
Simon
24ef493245 Websites being indexed now show up on the homepage 2018-06-12 21:51:02 -04:00
Simon
e266a50197 Website stats now works with elasticsearch 2018-06-12 20:17:30 -04:00
Simon
4b60ac62fc Added website url & date in search results & fixed threading problem 2018-06-12 17:48:15 -04:00
Simon
1718bb91ca Files are indexed into ES when task is complete 2018-06-12 15:45:00 -04:00
Simon
d61fd75890 Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s) 2018-06-12 13:44:03 -04:00
Simon
6d48f1f780 Task crawl result now logged in a database 2018-06-12 11:03:45 -04:00
Simon
72495275b0 Elasticsearch search engine (import from json) 2018-06-11 22:35:49 -04:00
Simon
fcfd7d4acc Bug fixes + export to json 2018-06-11 20:02:30 -04:00
Simon
d849227798 barebones crawl_server microservice 2018-06-11 19:00:43 -04:00