simon987/od-database-crawler
1
0
Fork 0
mirror of https://github.com/terorie/od-database-crawler.git synced 2025-12-10 22:18:55 +00:00
Code Issues Projects Releases Wiki Activity

Commit Graph

  • 777d7028b6 Bump github.com/spf13/viper from 1.3.2 to 1.6.1 dependabot/go_modules/github.com/spf13/viper-1.6.1 dependabot-preview[bot] 2019-12-09 04:19:17 +00:00
  • 9223edbfc5 Bump github.com/beeker1121/goque dependabot/go_modules/github.com/beeker1121/goque-2.1.0+incompatible dependabot-preview[bot] 2019-11-05 07:27:14 +00:00
  • 96a90dd23d Bump github.com/valyala/fasthttp from 1.2.0 to 1.6.0 dependabot/go_modules/github.com/valyala/fasthttp-1.6.0 dependabot-preview[bot] 2019-10-29 04:19:39 +00:00
  • 45866260b7 Bump github.com/spf13/cobra from 0.0.3 to 0.0.5 dependabot/go_modules/github.com/spf13/cobra-0.0.5 dependabot-preview[bot] 2019-06-10 04:48:05 +00:00
  • c19edc99b0 Bump github.com/sirupsen/logrus from 1.4.0 to 1.4.2 dependabot/go_modules/github.com/sirupsen/logrus-1.4.2 dependabot[bot] 2019-05-20 04:52:40 +00:00
  • a962c60b82 Don't panic on file upload error task_tracker simon987 2019-04-06 14:56:22 -04:00
  • 24f0bd91f7 Remove debug messages & don't use disk queue by default simon987 2019-04-06 12:11:42 -04:00
  • 84c10e1981 Change default config simon987 2019-04-04 20:49:10 -04:00
  • 88bf634cb6 Update config.yml master Simon Fortier 2019-04-05 09:30:20 -04:00
  • 860fa79327 Jenkins setup simon987 2019-04-04 18:39:57 -04:00
  • 76bc8293d6 minimum viable simon987 2019-03-30 09:02:55 -04:00
  • 796cf6ac23 Bump github.com/spf13/viper from 1.3.1 to 1.3.2 dependabot[bot] 2019-03-14 10:57:44 +01:00
  • defaf54e66 Bump github.com/sirupsen/logrus from 1.3.0 to 1.4.0 Richard Patel 2019-03-12 19:37:06 +01:00
  • 230824c58f Bump github.com/sirupsen/logrus from 1.3.0 to 1.4.0 dependabot[bot] 2019-03-12 04:48:03 +00:00
  • 3470be6086 More work on task_tracker integration simon987 2019-03-09 16:38:04 -05:00
  • d3c199b738 Update README terorie 2019-02-28 23:57:50 +01:00
  • 60471a081e Switch to simon987/task_tracker Richard Patel 2019-02-28 23:51:26 +01:00
  • 0b3f0d87fe Upgrade fasthttp to 1.2.0 dependabot[bot] 2019-02-28 22:42:40 +01:00
  • da9c75e392 Reduce Docker image size terorie 2019-02-22 21:37:04 +01:00
  • 8947e05d0c Fix Dockerfile v1.2.2 Pascal 2019-02-22 20:11:55 +00:00
  • 8c5f99d616 More descriptive error if /task/get returns invalid JSON Richard Patel 2019-02-22 20:17:53 +01:00
  • 206ea0e91d Simplify config Richard Patel 2019-02-22 18:50:35 +01:00
  • 8b9d8bfd17 Fix README.md format Richard Patel 2019-02-22 06:04:10 +01:00
  • c9ff102d80 Fix Dockerfile Richard Patel 2019-02-22 06:00:57 +01:00
  • 88856c1c19 Flag explanation in README.md Richard Patel 2019-02-22 05:59:59 +01:00
  • 9e9b606250 Merge branch 'stable' Richard Patel 2019-02-22 05:37:52 +01:00
  • 326e29e5e4 Reset to stable branch Richard Patel 2019-02-22 05:37:45 +01:00
  • c2acd5463f Restore .travis.yml Richard Patel 2019-02-22 05:16:25 +01:00
  • e4d04e6a5f go.mod: Fix package path Richard Patel 2019-02-22 05:10:37 +01:00
  • 9f1402e841 New Dockerfile and Travis Config (#23) terorie 2019-02-22 05:07:27 +01:00
  • 7c8ab50ee4 Merge stable into master terorie 2019-02-13 15:32:40 +01:00
  • 281d2d17d6 Update config.yml terorie 2019-02-13 15:32:00 +01:00
  • 9bc3455ee0 Fix missing port pipeline Richard Patel 2019-02-09 16:58:25 +01:00
  • c72f4ba475 Fix segfault Richard Patel 2019-02-09 16:50:45 +01:00
  • d69cd4400e Use fasthttp.PipelineClient Richard Patel 2019-02-09 16:46:36 +01:00
  • 45cbd4d535 Disable resume feature Richard Patel 2019-02-05 15:44:59 +01:00
  • 771d49f2dd Fix WaitGroup deadlock Richard Patel 2019-02-03 17:12:45 +01:00
  • dbd787aa81 Fix WaitGroup crash Richard Patel 2019-02-03 17:09:43 +01:00
  • cea6c1658b Bugfix: Don't schedule new tasks during shutdown Richard Patel 2019-02-03 17:02:44 +01:00
  • 885af5bb3b Beta task resuming terorie 2019-02-03 16:50:08 +01:00
  • 24d9d1fd42 Make resume work resume Richard Patel 2019-02-03 16:47:01 +01:00
  • f3be76e001 resume len Richard Patel 2019-02-03 16:41:26 +01:00
  • 4ef4ab13a8 Fix sleeps Richard Patel 2019-02-03 16:34:29 +01:00
  • 25d0b0042c resume: Fix missing gob register Richard Patel 2019-02-03 16:32:01 +01:00
  • ef7d17cad4 Fix too long sleep Richard Patel 2019-02-03 16:28:43 +01:00
  • e919323169 Resume tests Richard Patel 2019-02-03 16:24:18 +01:00
  • a3aebe4ef2 Pause file version Richard Patel 2019-02-03 16:08:42 +01:00
  • acbfd78a5d Save marker Richard Patel 2019-02-03 16:06:52 +01:00
  • fe1e7bf261 Save: queue dir if not yet exists Richard Patel 2019-02-03 15:57:35 +01:00
  • c6d7fad8e8 Resume state saving Richard Patel 2019-02-03 15:54:02 +01:00
  • 0b20823ae1 Resume log messages Richard Patel 2019-02-03 15:09:49 +01:00
  • 8d68bf1bbc Open result files in append-mode Richard Patel 2019-02-03 15:06:52 +01:00
  • a83eb0cfd7 Initial resume implementation Richard Patel 2019-02-03 15:02:07 +01:00
  • b18b70f798 Fix segfault (thanks Pikami) Richard Patel 2019-02-03 14:00:17 +01:00
  • 9d5f549774 Better server User-Agent string Richard Patel 2019-02-03 12:23:21 +01:00
  • 5239af08f7 Bump version to v1.2.1 v1.2.1 Richard Patel 2019-02-03 03:36:39 +01:00
  • 46c0e0bd32 Smarter HTTP error handling Richard Patel 2019-02-03 03:35:09 +01:00
  • 0ca6deede8 Fix --config flag Richard Patel 2019-02-03 03:17:22 +01:00
  • 120c026983 Bump version to v1.2.0 v1.2.0 Richard Patel 2019-02-03 02:55:21 +01:00
  • 527e8895ec Support configuration without config file Richard Patel 2019-02-03 02:54:52 +01:00
  • 108fff0503 Add Travis CI badge Richard Patel 2019-02-03 02:09:06 +01:00
  • e5746baa5b Switch to spf13/cobra Richard Patel 2019-02-03 02:02:23 +01:00
  • 17ba5583c9 Add .travis.yml Richard Patel 2019-02-02 23:18:03 +01:00
  • 92a8c07f4a Add go.mod Richard Patel 2019-02-02 23:15:52 +01:00
  • 43f96c6988 Benchmark: Reference parser Richard Patel 2018-12-18 15:39:41 +01:00
  • b244cdae80 Minor cleanup Richard Patel 2018-12-18 15:31:33 +01:00
  • 4b8275c7bf Add parser tests Richard Patel 2018-12-18 15:31:09 +01:00
  • f90bf94a44 Bump version to v1.1.1 v1.1.1 Richard Patel 2018-11-27 19:47:52 +01:00
  • e82768ff80 Wait time control in config Richard Patel 2018-11-27 19:47:30 +01:00
  • b1bf59adef Add The Eye DB to README.md Richard Patel 2018-11-27 17:40:12 +01:00
  • a2df2972f4 Bump the upload retry interval up to 30s Richard Patel 2018-11-20 04:13:20 +01:00
  • 3fc8837dd7 Add output files to .gitignore Richard Patel 2018-11-20 03:51:42 +01:00
  • f9a0d6bffe Bump to v1.1.0 v1.1.0 Richard Patel 2018-11-20 03:46:36 +01:00
  • 4dbe2aef2b Add job buffer size parameter Richard Patel 2018-11-20 03:42:32 +01:00
  • 86ec78cae1 Add TCP timeout option Richard Patel 2018-11-20 03:29:10 +01:00
  • b846498030 Delete URL queues after crawling Richard Patel 2018-11-20 03:05:43 +01:00
  • 4f3140a39f Fix queue_count in log Richard Patel 2018-11-20 02:49:03 +01:00
  • 85d2aac9d4 Performance patch Richard Patel 2018-11-20 02:33:50 +01:00
  • b6c0a45900 Job queue disk offloading Richard Patel 2018-11-20 02:03:10 +01:00
  • d332f06659 Limit retries to 10 Richard Patel 2018-11-18 21:05:26 +01:00
  • 1625d6c888 Bump to v1.0.2 v1.0.2 Richard Patel 2018-11-18 18:53:57 +01:00
  • 03a487f393 Fix crawl loop Richard Patel 2018-11-18 18:45:06 +01:00
  • ac8221b109 Retry /task/upload Richard Patel 2018-11-18 18:33:26 +01:00
  • 8ed2cf3b93 Bump to v1.0.1 v1.0.1 Richard Patel 2018-11-18 14:49:07 +01:00
  • f3620262fc Add log file support Richard Patel 2018-11-18 14:46:52 +01:00
  • dc4e4212a0 Add freebsd to release.sh Richard Patel 2018-11-18 14:38:18 +01:00
  • 6e6a4edd27 Ignore all HTTP errors Richard Patel 2018-11-18 14:25:06 +01:00
  • a71157b4d8 Add User-Agent parameter Richard Patel 2018-11-18 14:24:04 +01:00
  • 6dbec8c789 Add release script v1.0 Richard Patel 2018-11-18 02:36:22 +01:00
  • 605f6db5a5 Don't call /task/upload for websites with no results Richard Patel 2018-11-18 01:42:57 +01:00
  • d593ba2d0b Bump to 1.0 Richard Patel 2018-11-18 00:54:58 +01:00
  • 6793086c22 Ignore HTTPS errors Richard Patel 2018-11-18 00:37:30 +01:00
  • 4464f34779 Add recheck and timeout parameters Richard Patel 2018-11-18 00:29:29 +01:00
  • 339175220d Refactor uploading & chunk size parameter Richard Patel 2018-11-18 00:15:08 +01:00
  • 1e6687c519 Upload result ignoring errors Richard Patel 2018-11-17 15:04:20 +01:00
  • 8060556089 Fix: make crawled dir Richard Patel 2018-11-17 13:36:35 +01:00
  • 73ba848e17 Grammar Richard Patel 2018-11-17 13:35:29 +01:00
  • 115983f70e Silent HTTP errors Richard Patel 2018-11-17 13:18:08 +01:00
  • 9210996b4c Fix multiple part file upload Richard Patel 2018-11-17 12:51:30 +01:00
  • 7b29da9340 Fix file uploads Richard Patel 2018-11-17 12:47:16 +01:00