40 Commits

Author	SHA1	Message	Date
Richard Patel	206ea0e91d	Simplify config	2019-02-22 18:50:35 +01:00
Richard Patel	326e29e5e4	Reset to stable branch	2019-02-22 05:37:45 +01:00
Richard Patel	dbd787aa81	Fix WaitGroup crash	2019-02-03 17:09:43 +01:00
terorie	885af5bb3b	Beta task resuming	2019-02-03 16:50:08 +01:00
Richard Patel	5239af08f7	Bump version to v1.2.1	2019-02-03 03:36:39 +01:00
Richard Patel	120c026983	Bump version to v1.2.0	2019-02-03 02:55:21 +01:00
Richard Patel	527e8895ec	Support configuration without config file	2019-02-03 02:54:52 +01:00
Richard Patel	e5746baa5b	Switch to spf13/cobra ... lul	2019-02-03 02:02:23 +01:00
Richard Patel	f90bf94a44	Bump version to v1.1.1	2018-11-27 22:11:57 +01:00
Richard Patel	e82768ff80	Wait time control in config	2018-11-27 22:11:57 +01:00
Richard Patel	f9a0d6bffe	Bump to v1.1.0	2018-11-20 03:46:36 +01:00
Richard Patel	b6c0a45900	Job queue disk offloading	2018-11-20 02:03:10 +01:00
Richard Patel	1625d6c888	Bump to v1.0.2	2018-11-18 18:53:57 +01:00
Richard Patel	8ed2cf3b93	Bump to v1.0.1	2018-11-18 14:49:07 +01:00
Richard Patel	f3620262fc	Add log file support	2018-11-18 14:46:52 +01:00
Richard Patel	d593ba2d0b	Bump to 1.0	2018-11-18 00:54:58 +01:00
Richard Patel	6793086c22	Ignore HTTPS errors	2018-11-18 00:37:30 +01:00
Richard Patel	4464f34779	Add recheck and timeout parameters	2018-11-18 00:29:29 +01:00
Richard Patel	339175220d	Refactor uploading & chunk size parameter	2018-11-18 00:19:43 +01:00
Richard Patel	73ba848e17	Grammar	2018-11-17 13:35:29 +01:00
Richard Patel	24ee6fcba2	Quickfix: Revert FTP give back	2018-11-17 12:43:30 +01:00
Richard Patel	f4054441ab	Return FTP tasks	2018-11-17 05:07:52 +01:00
Richard Patel	f8d2bf386d	Fix FTP error ignore	2018-11-17 04:57:19 +01:00
Richard Patel	f41198b00c	Ignore FTP URLs	2018-11-17 04:50:59 +01:00
Richard Patel	d596882b40	Fix ton of bugs	2018-11-17 04:18:22 +01:00
Richard Patel	718f9d7fbc	Rename project	2018-11-17 01:33:15 +01:00
Richard Patel	145d37f84a	Fix wait, add back crawl command	2018-11-17 00:49:09 +01:00
Richard Patel	3f85cf679b	Getting tasks	2018-11-16 04:47:08 +01:00
Richard Patel	1db8ff43bb	Bump version	2018-11-16 00:25:11 +01:00
Richard Patel	ffde1a9e5d	Timeout and results saving	2018-11-15 20:14:31 +01:00
Richard Patel	ed5e35f005	Performance improvements	2018-11-06 00:34:22 +01:00
Richard Patel	395a6f30b2	Fix pprof	2018-11-05 21:55:07 +01:00
Richard Patel	e39565377e	Add pprof debug server	2018-11-05 21:39:15 +01:00
Richard Patel	77cb45dbec	Detect directory symlinks	2018-10-28 18:37:18 +01:00
Richard Patel	bfd7302be8	Add urfave/cli app	2018-10-28 17:59:46 +01:00
Richard Patel	b1c40767e0	Remember scanned URLs	2018-10-28 17:07:30 +01:00
Richard Patel	faad19f121	more stuff	2018-10-28 03:41:16 +01:00
Richard Patel	1c33346f45	Fix crawl descent	2018-10-28 03:06:18 +01:00
Richard Patel	79f540bf29	Scheduler	2018-10-28 02:40:12 +02:00
Richard Patel	abf069f946	Bits of ODDB API	2018-10-27 04:10:08 +02:00