simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-04-18 18:06:44 +00:00
Code Issues Projects Releases Wiki Activity
48 Commits 2 Branches 0 Tags
Commit Graph

48 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Simon
d61fd75890 Tasks can now be queued from the web interface. Tasks are dispatched to the crawl server(s) 2018-06-12 13:44:03 -04:00
Simon
6d48f1f780 Task crawl result now logged in a database 2018-06-12 11:03:45 -04:00
Simon
011b8455a7 Elasticsearch search engine (search & scroll) 2018-06-11 23:06:41 -04:00
Simon
72495275b0 Elasticsearch search engine (import from json) 2018-06-11 22:35:49 -04:00
Simon
fcfd7d4acc Bug fixes + export to json 2018-06-11 20:02:30 -04:00
Simon
d849227798 barebones crawl_server microservice 2018-06-11 19:00:43 -04:00
Simon
8421cc0885 Refactoring on http crawler 2018-06-11 16:06:56 -04:00
Simon
7f496ce7a8 Slowly losing my sanity part 1: Removed scrapy dependency and moved to custom solution. Added multi-threaded ftp crawler 2018-06-11 15:46:55 -04:00
Simon
b649b82854 Cleanup of custom crawler 2018-06-10 21:32:08 -04:00
Simon
f2d914060b Removed unsuitable scrapy spider and implemented custom crawler 2018-06-10 20:08:59 -04:00
Simon
d8c16d53e6 FTP url validation 2018-06-10 14:32:19 -04:00
Simon
0304c98a31 Added basic ftp spider for scrapy 2018-06-10 14:12:55 -04:00
Simon
f1e8183cdf Bulk insert captcha 2018-06-10 07:21:44 -04:00
Simon
4523a4335c Added bulk insert feature 2018-06-10 07:20:58 -04:00
Simon
1bd8a5fc22 Designed form for bulk insert 2018-06-10 07:03:59 -04:00
Simon
a25976d24a Generate and delete API tokens 2018-06-09 12:41:28 -04:00
Simon
de717d3992 Blacklisted skyarchive.info 2018-06-09 11:12:03 -04:00
Simon
20d0f97ffb Logout button 2018-06-08 11:48:11 -04:00
Simon
dc0cde61a0 Basic admin page 2018-06-08 11:40:54 -04:00
Simon
537228444b Duplicate website w/ reddit post + refactor 2018-06-08 10:40:58 -04:00
Simon
7f1e12cc3c Blacklisted https://oss.jfrog.org 2018-06-07 18:18:58 -04:00
Simon
b79b0ca58c Results per page now configurable 2018-06-07 13:49:08 -04:00
Simon
306b0ed0fe Added option to choose results per page 2018-06-07 13:19:41 -04:00
Simon
ab25d821c6 Export job no longer let the user download incomplete archives 2018-06-07 11:35:56 -04:00
Simon
06d3a09e11 Quick hack for search order options 2018-06-07 11:22:35 -04:00
Simon
221a16697b Changed user agent 2018-06-07 10:44:43 -04:00
Simon Fortier
92a91606ea
Create LICENSE 2018-06-07 10:36:26 -04:00
Simon
4f6d7f32ad Option to turn off SSL, moved secret keys to config.py, switched to sqlite WAL mode to avoid locked database problems 2018-06-07 10:33:35 -04:00
Simon Fortier
460357f183
Create README.md 2018-06-07 10:01:33 -04:00
Simon
a2835bbbcf Should fix problem when comment/post with subdir doesn't exist pt. 2 2018-06-06 19:40:08 -04:00
Simon
f396d326b9 Should fix problem when comment/post with subdir doesn't exist 2018-06-06 19:23:41 -04:00
Simon
204cf6ca92 Added space in comment 2018-06-06 16:54:47 -04:00
Simon
4b1fce309c Mark reddit post as crawled even if too small to comment 2018-06-06 16:01:07 -04:00
Simon
a69b7b305c /r/test -> /r/opendirectories 2018-06-06 15:31:35 -04:00
Simon
6cdc4b87d0 Minor adjustments 2018-06-06 14:15:06 -04:00
Simon
5fdf35ab56 Updated link 2018-06-06 13:23:23 -04:00
Simon
a92d8fe44e SSL support pt.2 2018-06-06 11:35:55 -04:00
Simon
abe5935847 SSL support 2018-06-06 11:34:07 -04:00
Simon
8db688b64e File size format for export 2018-06-06 11:28:44 -04:00
Simon
820caa171b Download link for export 2018-06-06 11:27:23 -04:00
Simon
7ac7972fd6 Exports are downloadable 2018-06-06 11:12:24 -04:00
Simon
0b1d76f478 Added blacklist feature (untested) 2018-06-06 10:17:30 -04:00
Simon
cfa6a9f02f Added requirement 2018-06-03 10:49:11 -04:00
Simon
270ab1335a Added reply to comments option, fixed some bugs 2018-06-02 17:26:15 -04:00
Simon
bb872a9248 Changed from mime to extension for graph and added script to clear invalid websites 2018-05-31 10:51:59 -04:00
Simon
819e2fbddb Renamed package (again) and removed unused files 2018-05-31 08:30:08 -04:00
Simon
ca651278d0 Renamed package 2018-05-28 20:39:34 -04:00
Simon
ad645490f6 Initial commit 2018-05-28 20:35:04 -04:00