simon987/od-database
1
0
Fork 0
mirror of https://github.com/simon987/od-database.git synced 2025-10-26 12:06:52 +00:00
Code Issues Projects Releases Wiki Activity
294 Commits 2 Branches 0 Tags
Commit Graph

294 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Simon
fcfd7d4acc Bug fixes + export to json 2018-06-11 20:02:30 -04:00
Simon
d849227798 barebones crawl_server microservice 2018-06-11 19:00:43 -04:00
Simon
8421cc0885 Refactoring on http crawler 2018-06-11 16:06:56 -04:00
Simon
7f496ce7a8 Slowly losing my sanity part 1: Removed scrapy dependency and moved to custom solution. Added multi-threaded ftp crawler 2018-06-11 15:46:55 -04:00
Simon
b649b82854 Cleanup of custom crawler 2018-06-10 21:32:08 -04:00
Simon
f2d914060b Removed unsuitable scrapy spider and implemented custom crawler 2018-06-10 20:08:59 -04:00
Simon
d8c16d53e6 FTP url validation 2018-06-10 14:32:19 -04:00
Simon
0304c98a31 Added basic ftp spider for scrapy 2018-06-10 14:12:55 -04:00
Simon
f1e8183cdf Bulk insert captcha 2018-06-10 07:21:44 -04:00
Simon
4523a4335c Added bulk insert feature 2018-06-10 07:20:58 -04:00
Simon
1bd8a5fc22 Designed form for bulk insert 2018-06-10 07:03:59 -04:00
Simon
a25976d24a Generate and delete API tokens 2018-06-09 12:41:28 -04:00
Simon
de717d3992 Blacklisted skyarchive.info 2018-06-09 11:12:03 -04:00
Simon
20d0f97ffb Logout button 2018-06-08 11:48:11 -04:00
Simon
dc0cde61a0 Basic admin page 2018-06-08 11:40:54 -04:00
Simon
537228444b Duplicate website w/ reddit post + refactor 2018-06-08 10:40:58 -04:00
Simon
7f1e12cc3c Blacklisted https://oss.jfrog.org 2018-06-07 18:18:58 -04:00
Simon
b79b0ca58c Results per page now configurable 2018-06-07 13:49:08 -04:00
Simon
306b0ed0fe Added option to choose results per page 2018-06-07 13:19:41 -04:00
Simon
ab25d821c6 Export job no longer let the user download incomplete archives 2018-06-07 11:35:56 -04:00
Simon
06d3a09e11 Quick hack for search order options 2018-06-07 11:22:35 -04:00
Simon
221a16697b Changed user agent 2018-06-07 10:44:43 -04:00
Simon Fortier
92a91606ea
Create LICENSE 2018-06-07 10:36:26 -04:00
Simon
4f6d7f32ad Option to turn off SSL, moved secret keys to config.py, switched to sqlite WAL mode to avoid locked database problems 2018-06-07 10:33:35 -04:00
Simon Fortier
460357f183
Create README.md 2018-06-07 10:01:33 -04:00
Simon
a2835bbbcf Should fix problem when comment/post with subdir doesn't exist pt. 2 2018-06-06 19:40:08 -04:00
Simon
f396d326b9 Should fix problem when comment/post with subdir doesn't exist 2018-06-06 19:23:41 -04:00
Simon
204cf6ca92 Added space in comment 2018-06-06 16:54:47 -04:00
Simon
4b1fce309c Mark reddit post as crawled even if too small to comment 2018-06-06 16:01:07 -04:00
Simon
a69b7b305c /r/test -> /r/opendirectories 2018-06-06 15:31:35 -04:00
Simon
6cdc4b87d0 Minor adjustments 2018-06-06 14:15:06 -04:00
Simon
5fdf35ab56 Updated link 2018-06-06 13:23:23 -04:00
Simon
a92d8fe44e SSL support pt.2 2018-06-06 11:35:55 -04:00
Simon
abe5935847 SSL support 2018-06-06 11:34:07 -04:00
Simon
8db688b64e File size format for export 2018-06-06 11:28:44 -04:00
Simon
820caa171b Download link for export 2018-06-06 11:27:23 -04:00
Simon
7ac7972fd6 Exports are downloadable 2018-06-06 11:12:24 -04:00
Simon
0b1d76f478 Added blacklist feature (untested) 2018-06-06 10:17:30 -04:00
Simon
cfa6a9f02f Added requirement 2018-06-03 10:49:11 -04:00
Simon
270ab1335a Added reply to comments option, fixed some bugs 2018-06-02 17:26:15 -04:00
Simon
bb872a9248 Changed from mime to extension for graph and added script to clear invalid websites 2018-05-31 10:51:59 -04:00
Simon
819e2fbddb Renamed package (again) and removed unused files 2018-05-31 08:30:08 -04:00
Simon
ca651278d0 Renamed package 2018-05-28 20:39:34 -04:00
Simon
ad645490f6 Initial commit 2018-05-28 20:35:04 -04:00