simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-12-20 01:45:57 +00:00
Code Issues Projects Releases Wiki Activity

Crawl server now holds at most max_workers + 1 tasks in pool to minimize waiting time and to avoid loss of too many tasks in case of crash/restart

Browse Source
This commit is contained in:
Simon
2018-06-12 22:28:36 -04:00
parent 24ef493245
commit 2fe81e4b06
5 changed files with 73 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
crawl_server/server.py
View File

@@ -4,7 +4,7 @@ from crawl_server.task_manager import TaskManager, Task, TaskResult
import os
app = Flask(__name__)
tm = TaskManager("tm_db.sqlite3")
tm = TaskManager("tm_db.sqlite3", 2)
@app.route("/task/")