simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-12-13 14:59:02 +00:00
Code Issues Projects Releases Wiki Activity

Slowly losing my sanity part 1: Removed scrapy dependency and moved to custom solution. Added multi-threaded ftp crawler

Browse Source
This commit is contained in:
Simon
2018-06-11 15:46:55 -04:00
parent b649b82854
commit 7f496ce7a8
10 changed files with 338 additions and 284 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
od_util.py
View File

@@ -75,7 +75,7 @@ def is_od(url):
ftp.close()
return True
else:
r = requests.get(url, timeout=15, allow_redirects=False)
r = requests.get(url, timeout=30, allow_redirects=False)
if r.status_code != 200:
print("No redirects allowed!")
return False