9 Commits

Author	SHA1	Message	Date
Simon	1b5e6bb7f4	Micro optimization	2018-11-05 17:08:54 -05:00
Richard Patel	77cb45dbec	Detect directory symlinks	2018-10-28 18:37:18 +01:00
Richard Patel	b1c40767e0	Remember scanned URLs	2018-10-28 17:07:30 +01:00
Richard Patel	ab5874129f	Don't retry on 401/403	2018-10-28 03:47:29 +01:00
Richard Patel	faad19f121	more stuff	2018-10-28 03:41:16 +01:00
Richard Patel	4ea5f8a410	Handle HTTP statuses	2018-10-28 03:22:25 +01:00
Richard Patel	1c33346f45	Fix crawl descent	2018-10-28 03:06:18 +01:00
Richard Patel	a507110787	Add stats interval parameter	2018-10-28 02:47:20 +02:00
Richard Patel	79f540bf29	Scheduler	2018-10-28 02:40:12 +02:00