simon987/od-database-crawler
1
0
Fork 0
mirror of https://github.com/terorie/od-database-crawler.git synced 2025-04-04 06:52:59 +00:00
Code Issues Projects Releases Wiki Activity
140 Commits 12 Branches 8 Tags
Commit Graph

140 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Simon Fortier
88bf634cb6
Update config.yml 2019-04-05 09:30:20 -04:00
dependabot[bot]
796cf6ac23 Bump github.com/spf13/viper from 1.3.1 to 1.3.2 2019-03-14 10:57:44 +01:00
Richard Patel
defaf54e66
Bump github.com/sirupsen/logrus from 1.3.0 to 1.4.0 2019-03-12 19:37:06 +01:00
dependabot[bot]
230824c58f
Bump github.com/sirupsen/logrus from 1.3.0 to 1.4.0 
Bumps [github.com/sirupsen/logrus](https://github.com/sirupsen/logrus) from 1.3.0 to 1.4.0.
- [Release notes](https://github.com/sirupsen/logrus/releases)
- [Changelog](https://github.com/sirupsen/logrus/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sirupsen/logrus/compare/v1.3.0...v1.4.0)

Signed-off-by: dependabot[bot] <support@dependabot.com>
 2019-03-12 04:48:03 +00:00
terorie
d3c199b738
Update README 
Add some badges and update description
 2019-02-28 23:57:50 +01:00
dependabot[bot]
0b3f0d87fe Upgrade fasthttp to 1.2.0 
Bumps [github.com/valyala/fasthttp](https://github.com/valyala/fasthttp) from 1.1.0 to 1.2.0.
- [Release notes](https://github.com/valyala/fasthttp/releases)
- [Commits](https://github.com/valyala/fasthttp/compare/v1.1.0...v1.2.0)

Thanks bot

Signed-off-by: dependabot[bot] <support@dependabot.com>
 2019-02-28 22:42:40 +01:00
terorie
da9c75e392
Reduce Docker image size 2019-02-22 21:37:04 +01:00
Pascal
8947e05d0c Fix Dockerfile 
Fixes #22
Credit to @pascaldulieu
v1.2.2 		2019-02-22 21:11:55 +01:00
Richard Patel
8c5f99d616
More descriptive error if /task/get returns invalid JSON 2019-02-22 20:17:59 +01:00
Richard Patel
206ea0e91d
Simplify config 2019-02-22 18:50:35 +01:00
Richard Patel
8b9d8bfd17
Fix README.md format 2019-02-22 06:04:10 +01:00
Richard Patel
c9ff102d80
Fix Dockerfile 2019-02-22 06:00:57 +01:00
Richard Patel
88856c1c19
Flag explanation in README.md 2019-02-22 05:59:59 +01:00
Richard Patel
9e9b606250
Merge branch 'stable' 2019-02-22 05:37:52 +01:00
Richard Patel
326e29e5e4
Reset to stable branch 2019-02-22 05:37:45 +01:00
Richard Patel
c2acd5463f
Restore .travis.yml 
Now handling auto-build over Docker Hub directly
 2019-02-22 05:16:25 +01:00
Richard Patel
e4d04e6a5f
go.mod: Fix package path 
lol
 2019-02-22 05:10:43 +01:00
terorie
9f1402e841
New Dockerfile and Travis Config (#23) 2019-02-22 05:07:27 +01:00
terorie
7c8ab50ee4
Merge stable into master 2019-02-13 15:32:40 +01:00
terorie
281d2d17d6
Update config.yml 2019-02-13 15:32:00 +01:00
Richard Patel
45cbd4d535
Disable resume feature 2019-02-05 15:44:59 +01:00
Richard Patel
771d49f2dd
Fix WaitGroup deadlock 2019-02-03 17:14:20 +01:00
Richard Patel
dbd787aa81
Fix WaitGroup crash 2019-02-03 17:09:43 +01:00
Richard Patel
cea6c1658b
Bugfix: Don't schedule new tasks during shutdown 2019-02-03 17:02:44 +01:00
terorie
885af5bb3b
Beta task resuming 2019-02-03 16:50:08 +01:00
Richard Patel
b18b70f798
Fix segfault (thanks Pikami) 2019-02-03 14:00:17 +01:00
Richard Patel
9d5f549774
Better server User-Agent string 2019-02-03 12:23:21 +01:00
Richard Patel
5239af08f7
Bump version to v1.2.1 v1.2.1 2019-02-03 03:36:39 +01:00
Richard Patel
46c0e0bd32
Smarter HTTP error handling 2019-02-03 03:35:09 +01:00
Richard Patel
0ca6deede8
Fix --config flag 2019-02-03 03:26:48 +01:00
Richard Patel
120c026983
Bump version to v1.2.0 v1.2.0 2019-02-03 02:55:21 +01:00
Richard Patel
527e8895ec
Support configuration without config file 2019-02-03 02:54:52 +01:00
Richard Patel
108fff0503
Add Travis CI badge 2019-02-03 02:09:06 +01:00
Richard Patel
e5746baa5b
Switch to spf13/cobra 
lul
 2019-02-03 02:02:23 +01:00
Richard Patel
17ba5583c9 Add .travis.yml 2019-02-02 23:18:03 +01:00
Richard Patel
92a8c07f4a
Add go.mod 2019-02-02 23:15:52 +01:00
Richard Patel
43f96c6988
Benchmark: Reference parser 2018-12-18 15:39:41 +01:00
Richard Patel
b244cdae80
Minor cleanup 2018-12-18 15:31:33 +01:00
Richard Patel
4b8275c7bf
Add parser tests 2018-12-18 15:31:09 +01:00
Richard Patel
f90bf94a44
Bump version to v1.1.1 v1.1.1 2018-11-27 22:11:57 +01:00
Richard Patel
e82768ff80
Wait time control in config 2018-11-27 22:11:57 +01:00
Richard Patel
b1bf59adef
Add The Eye DB to README.md 2018-11-27 17:40:12 +01:00
Richard Patel
a2df2972f4
Bump the upload retry interval up to 30s 2018-11-20 04:13:20 +01:00
Richard Patel
3fc8837dd7
Add output files to .gitignore 2018-11-20 03:51:42 +01:00
Richard Patel
f9a0d6bffe
Bump to v1.1.0 v1.1.0 2018-11-20 03:46:36 +01:00
Richard Patel
4dbe2aef2b
Add job buffer size parameter 2018-11-20 03:42:32 +01:00
Richard Patel
86ec78cae1
Add TCP timeout option 2018-11-20 03:29:10 +01:00
Richard Patel
b846498030
Delete URL queues after crawling 2018-11-20 03:05:43 +01:00
Richard Patel
4f3140a39f
Fix queue_count in log 2018-11-20 02:49:03 +01:00
Richard Patel
85d2aac9d4
Performance patch 2018-11-20 02:33:50 +01:00