122 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
terorie	7c8ab50ee4	Merge stable into master	2019-02-13 15:32:40 +01:00
terorie	281d2d17d6	Update config.yml	2019-02-13 15:32:00 +01:00
Richard Patel	45cbd4d535	Disable resume feature	2019-02-05 15:44:59 +01:00
Richard Patel	771d49f2dd	Fix WaitGroup deadlock	2019-02-03 17:14:20 +01:00
Richard Patel	dbd787aa81	Fix WaitGroup crash	2019-02-03 17:09:43 +01:00
Richard Patel	cea6c1658b	Bugfix: Don't schedule new tasks during shutdown	2019-02-03 17:02:44 +01:00
terorie	885af5bb3b	Beta task resuming	2019-02-03 16:50:08 +01:00
Richard Patel	b18b70f798	Fix segfault (thanks Pikami)	2019-02-03 14:00:17 +01:00
Richard Patel	9d5f549774	Better server User-Agent string	2019-02-03 12:23:21 +01:00
Richard Patel	5239af08f7	Bump version to v1.2.1 v1.2.1	2019-02-03 03:36:39 +01:00
Richard Patel	46c0e0bd32	Smarter HTTP error handling	2019-02-03 03:35:09 +01:00
Richard Patel	0ca6deede8	Fix --config flag	2019-02-03 03:26:48 +01:00
Richard Patel	120c026983	Bump version to v1.2.0 v1.2.0	2019-02-03 02:55:21 +01:00
Richard Patel	527e8895ec	Support configuration without config file	2019-02-03 02:54:52 +01:00
Richard Patel	108fff0503	Add Travis CI badge	2019-02-03 02:09:06 +01:00
Richard Patel	e5746baa5b	Switch to spf13/cobra ... lul	2019-02-03 02:02:23 +01:00
Richard Patel	17ba5583c9	Add .travis.yml	2019-02-02 23:18:03 +01:00
Richard Patel	92a8c07f4a	Add go.mod	2019-02-02 23:15:52 +01:00
Richard Patel	43f96c6988	Benchmark: Reference parser	2018-12-18 15:39:41 +01:00
Richard Patel	b244cdae80	Minor cleanup	2018-12-18 15:31:33 +01:00
Richard Patel	4b8275c7bf	Add parser tests	2018-12-18 15:31:09 +01:00
Richard Patel	f90bf94a44	Bump version to v1.1.1 v1.1.1	2018-11-27 22:11:57 +01:00
Richard Patel	e82768ff80	Wait time control in config	2018-11-27 22:11:57 +01:00
Richard Patel	b1bf59adef	Add The Eye DB to README.md	2018-11-27 17:40:12 +01:00
Richard Patel	a2df2972f4	Bump the upload retry interval up to 30s	2018-11-20 04:13:20 +01:00
Richard Patel	3fc8837dd7	Add output files to .gitignore	2018-11-20 03:51:42 +01:00
Richard Patel	f9a0d6bffe	Bump to v1.1.0 v1.1.0	2018-11-20 03:46:36 +01:00
Richard Patel	4dbe2aef2b	Add job buffer size parameter	2018-11-20 03:42:32 +01:00
Richard Patel	86ec78cae1	Add TCP timeout option	2018-11-20 03:29:10 +01:00
Richard Patel	b846498030	Delete URL queues after crawling	2018-11-20 03:05:43 +01:00
Richard Patel	4f3140a39f	Fix queue_count in log	2018-11-20 02:49:03 +01:00
Richard Patel	85d2aac9d4	Performance patch	2018-11-20 02:33:50 +01:00
Richard Patel	b6c0a45900	Job queue disk offloading	2018-11-20 02:03:10 +01:00
Richard Patel	d332f06659	Limit retries to 10	2018-11-18 21:05:26 +01:00
Richard Patel	1625d6c888	Bump to v1.0.2 v1.0.2	2018-11-18 18:53:57 +01:00
Richard Patel	03a487f393	Fix crawl loop	2018-11-18 18:45:06 +01:00
Richard Patel	ac8221b109	Retry /task/upload	2018-11-18 18:33:26 +01:00
Richard Patel	8ed2cf3b93	Bump to v1.0.1 v1.0.1	2018-11-18 14:49:07 +01:00
Richard Patel	f3620262fc	Add log file support	2018-11-18 14:46:52 +01:00
Richard Patel	dc4e4212a0	Add freebsd to release.sh	2018-11-18 14:38:18 +01:00
Richard Patel	6e6a4edd27	Ignore all HTTP errors	2018-11-18 14:25:06 +01:00
Richard Patel	a71157b4d8	Add User-Agent parameter	2018-11-18 14:24:04 +01:00
Richard Patel	6dbec8c789	Add release script v1.0	2018-11-18 02:36:22 +01:00
Richard Patel	605f6db5a5	Don't call /task/upload for websites with no results	2018-11-18 01:42:57 +01:00
Richard Patel	d593ba2d0b	Bump to 1.0	2018-11-18 00:54:58 +01:00
Richard Patel	6793086c22	Ignore HTTPS errors	2018-11-18 00:37:30 +01:00
Richard Patel	4464f34779	Add recheck and timeout parameters	2018-11-18 00:29:29 +01:00
Richard Patel	339175220d	Refactor uploading & chunk size parameter	2018-11-18 00:19:43 +01:00
Richard Patel	1e6687c519	Upload result ignoring errors	2018-11-17 15:04:20 +01:00
Richard Patel	8060556089	Fix: make crawled dir	2018-11-17 13:36:35 +01:00