25 Commits

Author	SHA1	Message	Date
Richard Patel	206ea0e91d	Simplify config	2019-02-22 18:50:35 +01:00
Richard Patel	326e29e5e4	Reset to stable branch	2019-02-22 05:37:45 +01:00
Richard Patel	45cbd4d535	Disable resume feature	2019-02-05 15:44:59 +01:00
terorie	885af5bb3b	Beta task resuming	2019-02-03 16:50:08 +01:00
Richard Patel	0ca6deede8	Fix --config flag	2019-02-03 03:26:48 +01:00
Richard Patel	527e8895ec	Support configuration without config file	2019-02-03 02:54:52 +01:00
Richard Patel	b244cdae80	Minor cleanup	2018-12-18 15:31:33 +01:00
Richard Patel	e82768ff80	Wait time control in config	2018-11-27 22:11:57 +01:00
Richard Patel	4dbe2aef2b	Add job buffer size parameter	2018-11-20 03:42:32 +01:00
Richard Patel	86ec78cae1	Add TCP timeout option	2018-11-20 03:29:10 +01:00
Richard Patel	f3620262fc	Add log file support	2018-11-18 14:46:52 +01:00
Richard Patel	a71157b4d8	Add User-Agent parameter	2018-11-18 14:24:04 +01:00
Richard Patel	6793086c22	Ignore HTTPS errors	2018-11-18 00:37:30 +01:00
Richard Patel	4464f34779	Add recheck and timeout parameters	2018-11-18 00:29:29 +01:00
Richard Patel	339175220d	Refactor uploading & chunk size parameter	2018-11-18 00:19:43 +01:00
Richard Patel	115983f70e	Silent HTTP errors	2018-11-17 13:22:46 +01:00
Richard Patel	3f85cf679b	Getting tasks	2018-11-16 04:47:08 +01:00
Richard Patel	ffde1a9e5d	Timeout and results saving	2018-11-15 20:14:31 +01:00
Richard Patel	add6581804	Add resource stats logging	2018-11-05 22:41:17 +01:00
Richard Patel	bfd7302be8	Add urfave/cli app	2018-10-28 17:59:46 +01:00
Richard Patel	b1c40767e0	Remember scanned URLs	2018-10-28 17:07:30 +01:00
Richard Patel	c196b6f20d	Better config	2018-10-28 14:19:09 +01:00
Richard Patel	1c33346f45	Fix crawl descent	2018-10-28 03:06:18 +01:00
Richard Patel	a507110787	Add stats interval parameter	2018-10-28 02:47:20 +02:00
Richard Patel	79f540bf29	Scheduler	2018-10-28 02:40:12 +02:00