PT post

content/posts/pt_1.md

@@ -0,0 +1,140 @@
+---
+title: "Overview Of Private Trackers"
+date: 2019-05-12T21:52:00-04:00
+draft: false
+tags: ["torrents"]
+---
+
+# Private BitTorrent trackers
+
+A BitTorrent tracker is the software in charge of orchestrating the communication between peers that are 
+using the protocol.
+It keeps track of statistics and, in the case of private trackers, manages download quotas
+and various other restrictions.
+
+
+<div class="box">
+<p class="box-title">BitTorrent Tracker software</p>
+You can find a short list of open-source torrent tracker platforms 
+<a href="https://web.archive.org/web/https://github.com/HDVinnie/Torrent-Tracker-Platforms">here</a>.
+</div>
+
+
+
+It’s not immediately obvious as to why private trackers would be worth the effort to
+join.
+For one, private torrent trackers usually have a much longer retention rate, have more organised
+ content and tend to be safer than public trackers (e.g. ThePirateBay, rutracker,
+rarbg).
+More importantly, some private trackers focus on old/rare content that is no longer
+obtainable legally.
+In the next section, we will explore the general registration process of private trackers.
+
+# Getting into private trackers
+
+Most "top-tier" or specialized torrent trackers are *invite-only*, meaning that one has to
+be invited by a current member that has reached the user class required to invite other
+members. In most cases, the invite giver is responsible for the invitee’s actions. When a
+user is caught trading or selling invites, his entire invite tree will be banned. Therefore,
+members are very careful with their invites. This is especially true for trackers that are
+deeper in the Invite map. Such an environment makes it quite unlikely for a stranger to
+get invited to these trackers. Staff members also have various other countermeasures to
+fight invite trading/buying (more on that in the following section). It is, however, very
+possible for a person completely foreign to the torrenting world to join any tracker given
+enough time and energy.
+
+Before attempting to join a private tracker, make sure that you are familiar with the
+BitTorrent protocol and the concept of sharing ratio.
+Having a good <sup>upload</sup>&frasl;<sub>download</sub> ratio
+is essential to keep your account at a given tracker and to climb up the user classes.
+Network bandwidth is really not as important as most people think.
+Dedicated or otherwise constantly running hardware and plenty of disk space – and it should not be
+a surprise for seasoned data archivists – are much more valuable in the long term.
+The second step is to join "open" trackers. Keep an eye on discussion boards for open-signup
+ or application timeframes. Some trackers (notably, *RED*[^1] and *MAM* ) have IRC interviews.
+
+<div class="box">
+<p class="box-title">Staying Informed</p>
+You can find miscellaneous information on Reddit (/r/trackers, /r/OpenSignups/)
+and 4chan (/ptg/). However do not expect much help for anything beyond technical
+problems coming from either forum, as most of their members are highly
+paranoid. As we will see in the next section, you are mostly on your own when it comes 
+to information about specific trackers.
+</div>
+
+
+# Invite map 
+
+{{< figure src="/pt/map.png" title="Private trackers invite map">}}
+
+ 
+The tracker invite map is an interactive visualization of the official invite routes of 
+a subset of the private trackers. Most private trackers have official recruiters with an 
+infinite number of invites available for members with the required user class (see figure below) 
+Data was gathered manually from tracker invite forums. An arrow pointing to another 
+tracker indicates that one can be invited to the tracker via an official invite thread. 
+Node size indicates the approximate number of enabled users (This metric is not always 
+made available). Of course this is not a complete map, partly because this information 
+is voluntarily made hard to find. Trackers with no invite threads from and to other 
+trackers are not shown on this map.
+
+{{< figure src="/pt/invite_thread.png" title="Official recruiter in an invite forum">}}
+
+To view the most up-to-date graph, raw data and source code of this project, you can
+visit https://dataarchivist.net/trackermap/.
+If you carefully review the tracker map, you will notice that almost all of them are
+accessible through a handful of important trackers. Most notably, the music tracker
+*Redacted* (RED), which one can join directly via the interview process, grants you access
+to almost all other music trackers and many more.
+Once you joined the "lower tier" trackers, your goal is to reach the user class that gives
+you access to the invite forums. This process is different for every tracker but it always
+involves either reaching a particular amount of data uploaded while maintaining a good
+ratio, uploading a number of new torrents or staying a member for a certain length of
+time. There are various guides for specific trackers linked in the Additional Reading
+section.
+
+# Rules, "Marking" and security
+
+## Marking
+
+It is generally frowned upon to share information relating to a tracker in a public forum.
+As such, staff members, are supposedly on the lookout for data that could be linked to
+a specific member – a screenshot of a forum which would indicate the date at which a
+certain thread was last visited, an exact ratio or a specific number of bonus points, etc.
+– and would in turn flag them. Flagged, or */marked/* members are more likely to get
+banned for minor offenses.
+
+{{< figure src="/pt/4chan_marked.png" title="Links posted in /ptg/ are seen as marking attempts by staff.">}}
+
+## Security
+
+Although you are a lot less likely to receive copyright notices from your ISP while using a
+private tracker, it is generally a good idea to assume that the private information related
+to your accounts (your email, username password and IP address) is **not** safe. It is good
+practice to use different usernames and try to avoid making them associable (avoid
+sending screenshots of your other accounts[^2]). Always use strong, unique passwords for
+all your accounts. Do not share any personal information (or any information at all, if
+possible) in the forums or IRC channels, and do not bother with trackers that are asking
+for personal information during the registration process.
+
+# Additionnal reading
+
+Sinderalla database leak
+
+* [4chan message #1](https://web.archive.org/web/20190101190608/https://rbt.asia/g/thread/66466024/#q66471906)
+* [4chan message #2](https://web.archive.org/web/20190101190608/https://rbt.asia/g/thread/66466024/#q66472549)
+* [4chan message #3](https://web.archive.org/web/20190101190608/https://rbt.asia/g/thread/66466024/#q66485228)
+* [Reddit discussion](https://www.reddit.com/r/trackers/comments/8tg5pt/sinderella_database_being_leaked/)
+
+IPTorrents owner dox (read the Why should I care section)
+
+* [https://iptorrentsdox.com/](https://web.archive.org/web/20141026015031/https://iptorrentsdox.com/)
+
+/ptg/ 
+
+* Wiki [https://wiki.installgentoo.com/index.php/Private_trackers](https://wiki.installgentoo.com/index.php/Private_trackers)
+* FAQ [https://pastebin.com/thLgSkNE](https://web.archive.org/web/20190504103634/https://pastebin.com/thLgSkNE)
+
+[^1]: You can find a walkthrough of the interview process [here](https://web.archive.org/web/20181230024615/https://wiki.installgentoo.com/index.php/Redacted.ch)
+[^2]: It is common practice to ask for ratio proofs when invinting other members.
+

content/posts/ytdl_1.md

@@ -6,16 +6,14 @@ tags: ["youtube-dl", "automation"]
 ---
 
 Google has been known to terminate entire Youtube channels without 
-notice in the hope of staying advertiser friendly. **58 million "problematic" videos
+notice in the hope of staying advertiser-friendly. **58 million "problematic" videos
 were deleted from the platform in Q3 2018** <sup>[1](#sources)</sup>.
 This week we are exploring various Youtube archival solutions that utilizes
  [youtube-dl](https://github.com/ytdl-org/youtube-dl/),
- a command-line program that extracts and downloads videos from web pages.
+ a command-line program that extracts and download videos from web pages.
 
 {{< figure src="/ytdl/1.png" title="Channels removed, by removal reason">}}
 
-
-
 # Installation
 
 Install youtube-dl via pip to ensure that you have the latest version. Google often
@@ -52,11 +50,11 @@ yt_dl.download("https://www.youtube.com/watch?v=XXXXXXXXXXXX")
 ## Command line arguments & Scripting
 
 This document is not a replacement for youtube-dl’s documentation, you can find the
-updated list of command line arguments on its [Github page](https://github.com/ytdl-org/youtube-dl/).
+updated list of command-line arguments on its [Github page](https://github.com/ytdl-org/youtube-dl/).
 
 Below is a bash script that will download everything specified in `list.txt`, a text file
 with a youtube channel or video on each line. The script will save the URLs of the videos
-in archive.txt as it downloads them to speed-up the subsequent executions.
+in archive.txt as it downloads them to speedup the subsequent executions.
 The `--write-info-json` and `--write-thumbnail` options ensures that we also download
 the video metadata such as the description and the title.
 
@@ -84,15 +82,15 @@ crontab -e
 
 ## Live streams
 
-While a cron job will download all videos uploaded by a channel (if the uploader does
+While a cron job downloads all videos uploaded by a channel (if the uploader does
 not delete the video between executions), it does not handle live streams. Youtube-dl
 allows you to download live streams with the same command but you obviously have to
 start the execution during the stream.
 Below is a different approach that takes advantage of the Youtube email notification
-feature. This simple Python script reads your last 3 emails and searches for a youtube
+feature. This simple Python script reads your last 3 emails and searches for a Youtube
 link in the email body. It will immediately start downloading the video using the youtube-dl
 Python library. You can use this method to download uploaded videos as well as live
-streams. If you are using a gmail account, you will need to genrate an [App Password](https://security.google.com/settings/security/apppasswords)
+streams. If you are using a Gmail account, you will need to generate an [App Password](https://security.google.com/settings/security/apppasswords)
  to allow the script to login.
 
 {{<highlight python "linenos=table">}}
@@ -102,7 +100,7 @@ import imaplib
 import re
 import youtube_dl
 
-# Initalize the youtube-dl downloader, nooverwrites param will
+# Initialize the youtube-dl downloader, nooverwrites param will
 # skip videos that are already downloaded or 
 # currently being downloaded
 yt_dl = youtube_dl.YoutubeDL(params={
@@ -111,7 +109,7 @@ yt_dl = youtube_dl.YoutubeDL(params={
     "format": "bestvideo[ext=webm]+bestaudio[ext=webm]/bestvideo[ext=mp4]+bestaudio[ext=m4a]/bestvideo+bestaudio/best"
     })
 
-# This regex pattern matches youtube video links
+# This regex pattern matches Youtube video links
 YT_LINK = re.compile("Fv%3D([^%]*)%")
 
 mail = imaplib.IMAP4_SSL("imap.gmail.com")
@@ -138,14 +136,14 @@ for num in last_emails:
 ## Automatically upload to rclone remote
 
 To take advantage of cloud storage, you can setup [ytdlrc](http://github.com/bardisty/ytdlrc) to automatically move
-videos to an rclone remote as they are downloaded. This simple script is completely interchangable with
+videos to an rclone remote as they are downloaded. This simple script is completely interchangeable with
-youtube-dl and can be setup on a machine with low disk space.
+youtube-dl and can be set up on a machine with low disk space.
 The script uses your existing youtube-dl and rclone configuration and is ideal for
 setting up automatic Youtube archival on a cheap VPS.
 
 ## Archiving Metadata
 
-If you wish to save a video’s metadata without downloading the actual video, there are command line utilities dedicated to this task.
+If you wish to save a video’s metadata without downloading the actual video, there are command-line utilities dedicated to this task.
 
 * [Youtube-MA](https://github.com/CorentinB/YouTube-MA)
 * [yt-mango](https://github.com/terorie/yt-mango)

layouts/partials/css/custom.css

@@ -271,3 +271,21 @@ code, kbd, pre, samp {
 		display: block;
 	}
 }
+
+.box {
+	border: 1px solid black;
+	border-radius: 2px;
+	box-shadow: 2px 2px 1px rgba(0,0,0,0.4);
+	padding: 4px;
+	margin-right: 3px;
+}
+
+.box .box-title {
+	font-weight: bold;
+	margin: 0.3em 0 0.6em;
+}
+
+figcaption h4 {
+	margin-top: 0.3em;
+	font-size: inherit !important;
+}