diff --git a/content/posts/pt_1.md b/content/posts/pt_1.md new file mode 100644 index 0000000..9a2adef --- /dev/null +++ b/content/posts/pt_1.md @@ -0,0 +1,140 @@ +--- +title: "Overview Of Private Trackers" +date: 2019-05-12T21:52:00-04:00 +draft: false +tags: ["torrents"] +--- + +# Private BitTorrent trackers + +A BitTorrent tracker is the software in charge of orchestrating the communication between peers that are +using the protocol. +It keeps track of statistics and, in the case of private trackers, manages download quotas +and various other restrictions. + + +
+

BitTorrent Tracker software

+You can find a short list of open-source torrent tracker platforms +here. +
+ + + +It’s not immediately obvious as to why private trackers would be worth the effort to +join. +For one, private torrent trackers usually have a much longer retention rate, have more organised + content and tend to be safer than public trackers (e.g. ThePirateBay, rutracker, +rarbg). +More importantly, some private trackers focus on old/rare content that is no longer +obtainable legally. +In the next section, we will explore the general registration process of private trackers. + +# Getting into private trackers + +Most "top-tier" or specialized torrent trackers are *invite-only*, meaning that one has to +be invited by a current member that has reached the user class required to invite other +members. In most cases, the invite giver is responsible for the invitee’s actions. When a +user is caught trading or selling invites, his entire invite tree will be banned. Therefore, +members are very careful with their invites. This is especially true for trackers that are +deeper in the Invite map. Such an environment makes it quite unlikely for a stranger to +get invited to these trackers. Staff members also have various other countermeasures to +fight invite trading/buying (more on that in the following section). It is, however, very +possible for a person completely foreign to the torrenting world to join any tracker given +enough time and energy. + +Before attempting to join a private tracker, make sure that you are familiar with the +BitTorrent protocol and the concept of sharing ratio. +Having a good upload⁄download ratio +is essential to keep your account at a given tracker and to climb up the user classes. +Network bandwidth is really not as important as most people think. +Dedicated or otherwise constantly running hardware and plenty of disk space – and it should not be +a surprise for seasoned data archivists – are much more valuable in the long term. +The second step is to join "open" trackers. Keep an eye on discussion boards for open-signup + or application timeframes. Some trackers (notably, *RED*[^1] and *MAM* ) have IRC interviews. + +
+

Staying Informed

+You can find miscellaneous information on Reddit (/r/trackers, /r/OpenSignups/) +and 4chan (/ptg/). However do not expect much help for anything beyond technical +problems coming from either forum, as most of their members are highly +paranoid. As we will see in the next section, you are mostly on your own when it comes +to information about specific trackers. +
+ + +# Invite map + +{{< figure src="/pt/map.png" title="Private trackers invite map">}} + + +The tracker invite map is an interactive visualization of the official invite routes of +a subset of the private trackers. Most private trackers have official recruiters with an +infinite number of invites available for members with the required user class (see figure below) +Data was gathered manually from tracker invite forums. An arrow pointing to another +tracker indicates that one can be invited to the tracker via an official invite thread. +Node size indicates the approximate number of enabled users (This metric is not always +made available). Of course this is not a complete map, partly because this information +is voluntarily made hard to find. Trackers with no invite threads from and to other +trackers are not shown on this map. + +{{< figure src="/pt/invite_thread.png" title="Official recruiter in an invite forum">}} + +To view the most up-to-date graph, raw data and source code of this project, you can +visit https://dataarchivist.net/trackermap/. +If you carefully review the tracker map, you will notice that almost all of them are +accessible through a handful of important trackers. Most notably, the music tracker +*Redacted* (RED), which one can join directly via the interview process, grants you access +to almost all other music trackers and many more. +Once you joined the "lower tier" trackers, your goal is to reach the user class that gives +you access to the invite forums. This process is different for every tracker but it always +involves either reaching a particular amount of data uploaded while maintaining a good +ratio, uploading a number of new torrents or staying a member for a certain length of +time. There are various guides for specific trackers linked in the Additional Reading +section. + +# Rules, "Marking" and security + +## Marking + +It is generally frowned upon to share information relating to a tracker in a public forum. +As such, staff members, are supposedly on the lookout for data that could be linked to +a specific member – a screenshot of a forum which would indicate the date at which a +certain thread was last visited, an exact ratio or a specific number of bonus points, etc. +– and would in turn flag them. Flagged, or */marked/* members are more likely to get +banned for minor offenses. + +{{< figure src="/pt/4chan_marked.png" title="Links posted in /ptg/ are seen as marking attempts by staff.">}} + +## Security + +Although you are a lot less likely to receive copyright notices from your ISP while using a +private tracker, it is generally a good idea to assume that the private information related +to your accounts (your email, username password and IP address) is **not** safe. It is good +practice to use different usernames and try to avoid making them associable (avoid +sending screenshots of your other accounts[^2]). Always use strong, unique passwords for +all your accounts. Do not share any personal information (or any information at all, if +possible) in the forums or IRC channels, and do not bother with trackers that are asking +for personal information during the registration process. + +# Additionnal reading + +Sinderalla database leak + +* [4chan message #1](https://web.archive.org/web/20190101190608/https://rbt.asia/g/thread/66466024/#q66471906) +* [4chan message #2](https://web.archive.org/web/20190101190608/https://rbt.asia/g/thread/66466024/#q66472549) +* [4chan message #3](https://web.archive.org/web/20190101190608/https://rbt.asia/g/thread/66466024/#q66485228) +* [Reddit discussion](https://www.reddit.com/r/trackers/comments/8tg5pt/sinderella_database_being_leaked/) + +IPTorrents owner dox (read the Why should I care section) + +* [https://iptorrentsdox.com/](https://web.archive.org/web/20141026015031/https://iptorrentsdox.com/) + +/ptg/ + +* Wiki [https://wiki.installgentoo.com/index.php/Private_trackers](https://wiki.installgentoo.com/index.php/Private_trackers) +* FAQ [https://pastebin.com/thLgSkNE](https://web.archive.org/web/20190504103634/https://pastebin.com/thLgSkNE) + +[^1]: You can find a walkthrough of the interview process [here](https://web.archive.org/web/20181230024615/https://wiki.installgentoo.com/index.php/Redacted.ch) +[^2]: It is common practice to ask for ratio proofs when invinting other members. + diff --git a/content/posts/ytdl_1.md b/content/posts/ytdl_1.md index bca3c4d..c64f5b3 100644 --- a/content/posts/ytdl_1.md +++ b/content/posts/ytdl_1.md @@ -6,16 +6,14 @@ tags: ["youtube-dl", "automation"] --- Google has been known to terminate entire Youtube channels without -notice in the hope of staying advertiser friendly. **58 million "problematic" videos +notice in the hope of staying advertiser-friendly. **58 million "problematic" videos were deleted from the platform in Q3 2018** [1](#sources). This week we are exploring various Youtube archival solutions that utilizes [youtube-dl](https://github.com/ytdl-org/youtube-dl/), - a command-line program that extracts and downloads videos from web pages. + a command-line program that extracts and download videos from web pages. {{< figure src="/ytdl/1.png" title="Channels removed, by removal reason">}} - - # Installation Install youtube-dl via pip to ensure that you have the latest version. Google often @@ -52,11 +50,11 @@ yt_dl.download("https://www.youtube.com/watch?v=XXXXXXXXXXXX") ## Command line arguments & Scripting This document is not a replacement for youtube-dl’s documentation, you can find the -updated list of command line arguments on its [Github page](https://github.com/ytdl-org/youtube-dl/). +updated list of command-line arguments on its [Github page](https://github.com/ytdl-org/youtube-dl/). Below is a bash script that will download everything specified in `list.txt`, a text file with a youtube channel or video on each line. The script will save the URLs of the videos -in archive.txt as it downloads them to speed-up the subsequent executions. +in archive.txt as it downloads them to speedup the subsequent executions. The `--write-info-json` and `--write-thumbnail` options ensures that we also download the video metadata such as the description and the title. @@ -84,15 +82,15 @@ crontab -e ## Live streams -While a cron job will download all videos uploaded by a channel (if the uploader does +While a cron job downloads all videos uploaded by a channel (if the uploader does not delete the video between executions), it does not handle live streams. Youtube-dl allows you to download live streams with the same command but you obviously have to start the execution during the stream. Below is a different approach that takes advantage of the Youtube email notification -feature. This simple Python script reads your last 3 emails and searches for a youtube +feature. This simple Python script reads your last 3 emails and searches for a Youtube link in the email body. It will immediately start downloading the video using the youtube-dl Python library. You can use this method to download uploaded videos as well as live -streams. If you are using a gmail account, you will need to genrate an [App Password](https://security.google.com/settings/security/apppasswords) +streams. If you are using a Gmail account, you will need to generate an [App Password](https://security.google.com/settings/security/apppasswords) to allow the script to login. {{}} @@ -102,7 +100,7 @@ import imaplib import re import youtube_dl -# Initalize the youtube-dl downloader, nooverwrites param will +# Initialize the youtube-dl downloader, nooverwrites param will # skip videos that are already downloaded or # currently being downloaded yt_dl = youtube_dl.YoutubeDL(params={ @@ -111,7 +109,7 @@ yt_dl = youtube_dl.YoutubeDL(params={ "format": "bestvideo[ext=webm]+bestaudio[ext=webm]/bestvideo[ext=mp4]+bestaudio[ext=m4a]/bestvideo+bestaudio/best" }) -# This regex pattern matches youtube video links +# This regex pattern matches Youtube video links YT_LINK = re.compile("Fv%3D([^%]*)%") mail = imaplib.IMAP4_SSL("imap.gmail.com") @@ -138,14 +136,14 @@ for num in last_emails: ## Automatically upload to rclone remote To take advantage of cloud storage, you can setup [ytdlrc](http://github.com/bardisty/ytdlrc) to automatically move -videos to an rclone remote as they are downloaded. This simple script is completely interchangable with -youtube-dl and can be setup on a machine with low disk space. +videos to an rclone remote as they are downloaded. This simple script is completely interchangeable with +youtube-dl and can be set up on a machine with low disk space. The script uses your existing youtube-dl and rclone configuration and is ideal for setting up automatic Youtube archival on a cheap VPS. ## Archiving Metadata -If you wish to save a video’s metadata without downloading the actual video, there are command line utilities dedicated to this task. +If you wish to save a video’s metadata without downloading the actual video, there are command-line utilities dedicated to this task. * [Youtube-MA](https://github.com/CorentinB/YouTube-MA) * [yt-mango](https://github.com/terorie/yt-mango) diff --git a/layouts/partials/css/custom.css b/layouts/partials/css/custom.css index e21e174..9ab9cdf 100644 --- a/layouts/partials/css/custom.css +++ b/layouts/partials/css/custom.css @@ -271,3 +271,21 @@ code, kbd, pre, samp { display: block; } } + +.box { + border: 1px solid black; + border-radius: 2px; + box-shadow: 2px 2px 1px rgba(0,0,0,0.4); + padding: 4px; + margin-right: 3px; +} + +.box .box-title { + font-weight: bold; + margin: 0.3em 0 0.6em; +} + +figcaption h4 { + margin-top: 0.3em; + font-size: inherit !important; +} diff --git a/static/pt/4chan_marked.png b/static/pt/4chan_marked.png new file mode 100644 index 0000000..35750e1 Binary files /dev/null and b/static/pt/4chan_marked.png differ diff --git a/static/pt/invite_thread.png b/static/pt/invite_thread.png new file mode 100644 index 0000000..6b049c7 Binary files /dev/null and b/static/pt/invite_thread.png differ diff --git a/static/pt/map.png b/static/pt/map.png new file mode 100644 index 0000000..4774cdd Binary files /dev/null and b/static/pt/map.png differ