Simon 99d64b658b Disabled thread pool for headers requests in listing

2018-06-18 10:33:33 -04:00

..

crawled

Should fix memory usage problem when crawling

2018-06-14 23:36:54 -04:00

__init__.py

barebones crawl_server microservice

2018-06-11 19:00:43 -04:00

callbacks.py

Started working on post-crawl callbacks and basic auth for crawl servers

2018-06-14 15:05:56 -04:00

crawler.py

Disabled thread pool for headers requests in listing

2018-06-18 10:33:33 -04:00

database.py

Started working on post-crawl callbacks and basic auth for crawl servers

2018-06-14 15:05:56 -04:00

reddit_bot.py

Started working on post-crawl callbacks and basic auth for crawl servers

2018-06-14 15:05:56 -04:00

remote_ftp.py

Bug fixes for FTP crawler

2018-06-13 15:54:45 -04:00

remote_http.py

Disabled thread pool for headers requests in listing

2018-06-18 10:33:33 -04:00

server.py

Should fix memory usage problem when crawling

2018-06-14 23:36:54 -04:00

task_db_init.sql

Files are indexed into ES when task is complete

2018-06-12 15:45:00 -04:00

task_manager.py

Simplified url joining and splitting, switched from lxml to html.parser, various memory usage optimizations

2018-06-17 22:10:46 -04:00