97 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Richard Patel	3fc8837dd7	Add output files to .gitignore	2018-11-20 03:51:42 +01:00
Richard Patel	f9a0d6bffe	Bump to v1.1.0 v1.1.0	2018-11-20 03:46:36 +01:00
Richard Patel	4dbe2aef2b	Add job buffer size parameter	2018-11-20 03:42:32 +01:00
Richard Patel	86ec78cae1	Add TCP timeout option	2018-11-20 03:29:10 +01:00
Richard Patel	b846498030	Delete URL queues after crawling	2018-11-20 03:05:43 +01:00
Richard Patel	4f3140a39f	Fix queue_count in log	2018-11-20 02:49:03 +01:00
Richard Patel	85d2aac9d4	Performance patch	2018-11-20 02:33:50 +01:00
Richard Patel	b6c0a45900	Job queue disk offloading	2018-11-20 02:03:10 +01:00
Richard Patel	d332f06659	Limit retries to 10	2018-11-18 21:05:26 +01:00
Richard Patel	1625d6c888	Bump to v1.0.2 v1.0.2	2018-11-18 18:53:57 +01:00
Richard Patel	03a487f393	Fix crawl loop	2018-11-18 18:45:06 +01:00
Richard Patel	ac8221b109	Retry /task/upload	2018-11-18 18:33:26 +01:00
Richard Patel	8ed2cf3b93	Bump to v1.0.1 v1.0.1	2018-11-18 14:49:07 +01:00
Richard Patel	f3620262fc	Add log file support	2018-11-18 14:46:52 +01:00
Richard Patel	dc4e4212a0	Add freebsd to release.sh	2018-11-18 14:38:18 +01:00
Richard Patel	6e6a4edd27	Ignore all HTTP errors	2018-11-18 14:25:06 +01:00
Richard Patel	a71157b4d8	Add User-Agent parameter	2018-11-18 14:24:04 +01:00
Richard Patel	6dbec8c789	Add release script v1.0	2018-11-18 02:36:22 +01:00
Richard Patel	605f6db5a5	Don't call /task/upload for websites with no results	2018-11-18 01:42:57 +01:00
Richard Patel	d593ba2d0b	Bump to 1.0	2018-11-18 00:54:58 +01:00
Richard Patel	6793086c22	Ignore HTTPS errors	2018-11-18 00:37:30 +01:00
Richard Patel	4464f34779	Add recheck and timeout parameters	2018-11-18 00:29:29 +01:00
Richard Patel	339175220d	Refactor uploading & chunk size parameter	2018-11-18 00:19:43 +01:00
Richard Patel	1e6687c519	Upload result ignoring errors	2018-11-17 15:04:20 +01:00
Richard Patel	8060556089	Fix: make crawled dir	2018-11-17 13:36:35 +01:00
Richard Patel	73ba848e17	Grammar	2018-11-17 13:35:29 +01:00
Richard Patel	115983f70e	Silent HTTP errors	2018-11-17 13:22:46 +01:00
Richard Patel	9210996b4c	Fix multiple part file upload	2018-11-17 12:52:24 +01:00
Richard Patel	7b29da9340	Fix file uploads	2018-11-17 12:47:16 +01:00
Richard Patel	24ee6fcba2	Quickfix: Revert FTP give back	2018-11-17 12:43:30 +01:00
Richard Patel	bfb18d62b2	mini fix	2018-11-17 05:27:09 +01:00
Richard Patel	f4054441ab	Return FTP tasks	2018-11-17 05:07:52 +01:00
Richard Patel	f8d2bf386d	Fix FTP error ignore	2018-11-17 04:57:19 +01:00
Richard Patel	f41198b00c	Ignore FTP URLs	2018-11-17 04:50:59 +01:00
Richard Patel	7fdffff58f	Update config.yml	2018-11-17 04:19:04 +01:00
Richard Patel	d596882b40	Fix ton of bugs	2018-11-17 04:18:22 +01:00
Richard Patel	0fe97a8058	Update README.md	2018-11-17 01:36:07 +01:00
Richard Patel	718f9d7fbc	Rename project	2018-11-17 01:33:15 +01:00
Richard Patel	f1687679ab	Unescape results & don't recrawl 404	2018-11-17 01:21:20 +01:00
Richard Patel	145d37f84a	Fix wait, add back crawl command	2018-11-17 00:49:09 +01:00
Richard Patel	cc777bcaeb	redblackhash: Use bytes.Compare	2018-11-16 21:17:39 +01:00
Simon	1e78cea7e7	Saved path should not contain file name	2018-11-16 13:58:12 -05:00
Richard Patel	3f85cf679b	Getting tasks	2018-11-16 04:47:08 +01:00
Richard Patel	3c39f0d621	Random hacks	2018-11-16 03:22:51 +01:00
Richard Patel	50952791c5	Almost done	2018-11-16 03:12:26 +01:00
Richard Patel	30bf98ad34	Fix tests	2018-11-16 03:02:10 +01:00
Richard Patel	ccaf758e90	Remove URL.Opaque	2018-11-16 01:53:16 +01:00
Richard Patel	f668365edb	Add tests	2018-11-16 01:51:34 +01:00
Richard Patel	1db8ff43bb	Bump version	2018-11-16 00:25:11 +01:00
Richard Patel	82234f949e	Less tokenizer allocations	2018-11-16 00:22:40 +01:00