Simon 7a4432e4d0 More bugfixes for looping directories, some work on task dispatching

2018-06-21 20:50:26 -04:00

..

crawled

Should fix memory usage problem when crawling

2018-06-14 23:36:54 -04:00

__init__.py

barebones crawl_server microservice

2018-06-11 19:00:43 -04:00

callbacks.py

Started working on post-crawl callbacks and basic auth for crawl servers

2018-06-14 15:05:56 -04:00

crawler.py

More bugfixes for looping directories, some work on task dispatching

2018-06-21 20:50:26 -04:00

database.py

Added crawl logs page

2018-06-18 11:41:26 -04:00

reddit_bot.py

Started working on post-crawl callbacks and basic auth for crawl servers

2018-06-14 15:05:56 -04:00

remote_ftp.py

Attempt to handle looping directories

2018-06-21 11:54:40 -04:00

remote_http.py

More bugfixes for looping directories, some work on task dispatching

2018-06-21 20:50:26 -04:00

server.py

Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default

2018-06-21 19:44:27 -04:00

task_db_init.sql

Added crawl logs page

2018-06-18 11:41:26 -04:00

task_manager.py

Added stats page

2018-06-18 19:56:25 -04:00