mirror of https://github.com/simon987/od-database.git synced 2025-04-16 08:56:44 +00:00

Go to file

Simon 14d384e366 Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default

2018-06-21 19:44:27 -04:00

crawl_server

Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default

2018-06-21 19:44:27 -04:00

Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default

2018-06-21 19:44:27 -04:00

static

Fixed stats bug

2018-06-20 18:07:55 -04:00

templates

Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default

2018-06-21 19:44:27 -04:00

test

Files are indexed into ES when task is complete

2018-06-12 15:45:00 -04:00

__init__.py

Initial commit

2018-05-28 20:35:04 -04:00

.gitignore

Updated link

2018-06-06 13:23:23 -04:00

app.py

Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default

2018-06-21 19:44:27 -04:00

blacklist.txt

Added admin clear & delete buttons for websites

2018-06-20 10:48:51 -04:00

database.py

Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default

2018-06-21 19:44:27 -04:00

debug_put.py

fixed navbar icon for mobile

2018-06-19 21:13:36 -04:00

export.py

Typo in csv export

2018-06-19 10:17:15 -04:00

init_script.sql

Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default

2018-06-21 19:44:27 -04:00

LICENSE

Create LICENSE

2018-06-07 10:36:26 -04:00

od_util.py

Fixes #2

2018-06-20 21:52:45 -04:00

od-database.ini

uWSGI config and bugfix with file extensions

2018-06-13 14:11:27 -04:00

queue_reddit_links.py

Started working on post-crawl callbacks and basic auth for crawl servers

2018-06-14 15:05:56 -04:00

README.md

Refactoring on http crawler

2018-06-11 16:06:56 -04:00

requirements.txt

Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default

2018-06-21 19:44:27 -04:00

scrapy.cfg

Renamed package (again) and removed unused files

2018-05-31 08:30:08 -04:00

startWSGI.sh

uWSGI config and bugfix with file extensions

2018-06-13 14:11:27 -04:00

stress_test.py

fixed navbar icon for mobile

2018-06-19 21:13:36 -04:00

task.py

Decentralised crawling should work in theory + temporary fix for going further than the maximum 10k results elasticsearch allows by default

2018-06-21 19:44:27 -04:00

uwsgi.py

uWSGI config and bugfix with file extensions

2018-06-13 14:11:27 -04:00

README.md

OD-Database

Suggestions/concerns/PRs are welcome

Installation

Assuming you have Python 3 and git installed:

git clone https://github.com/simon987/od-database
cd od-database
pip3 install -r requirements.txt

Create /config.py and fill out the parameters. Empty config:

CAPTCHA_SITE_KEY = ""
CAPTCHA_SECRET_KEY = ""
FLASK_SECRET = ""
USE_SSL = True
RESULTS_PER_PAGE = (25, 50, 100, 250, 500, 1000)
HEADERS = {}

Running

python3 app.py

You should be able to connect with your browser at https://localhost:12345

*Note: To use SSL you have to put the appropriate certificates in /certificates/