Simon 2fe81e4b06 Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart

2018-06-12 22:28:36 -04:00

..

__init__.py

barebones crawl_server microservice

2018-06-11 19:00:43 -04:00

crawler.py

Fixed file duplication problem

2018-06-12 15:55:52 -04:00

database.py

Files are indexed into ES when task is complete

2018-06-12 15:45:00 -04:00

remote_ftp.py

Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)

2018-06-12 13:44:03 -04:00

remote_http.py

Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s)

2018-06-12 13:44:03 -04:00

server.py

Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart

2018-06-12 22:28:36 -04:00

task_db_init.sql

Files are indexed into ES when task is complete

2018-06-12 15:45:00 -04:00

task_manager.py

Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart

2018-06-12 22:28:36 -04:00