From 1c2683201bd06a803012218a926af99c4b16f689 Mon Sep 17 00:00:00 2001 From: Simon Fortier Date: Mon, 15 Oct 2018 20:38:06 -0400 Subject: [PATCH] First draft --- README.md | 54 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 53 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f5da9c3..131e0c8 100644 --- a/README.md +++ b/README.md @@ -1 +1,53 @@ -# awesome-datahoarding \ No newline at end of file + +### Download/Scraping utilities + +* [Rclone](https://github.com/ncw/rclone): A command line program to sync files and directories to and from various cloud storage providers +* [Youtube-DL](https://github.com/rg3/youtube-dl): A command-line program to download videos from YouTube and a few hundred more sites +* [annie](https://github.com/iawia002/annie): Youtube-DL alternative writtent in Golang +* [wikiteam](https://github.com/WikiTeam/wikiteam): set of tools for archiving wikis +* [FicSave](https://github.com/waylaidwanderer/FicSave): online fanfiction downloader +* [yt-mango](https://github.com/terorie/yt-mango): Youtube metadata archiver +* [Youtube-MA](https://github.com/CorentinB/YouTube-MA): Youtube metadata archiver +* [CrowLeer](https://github.com/ERap320/CrowLeer): Powerful C++ web crawler based on libcurl +* [floatplane_ripper](https://gist.github.com/simon987/0756c378ca2dfb0003931e26ff7fe270): Script to rip all videos from https://floatplane.rip/ +* [grab-site](https://github.com/ludios/grab-site): The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns +* [dzi-dl](https://github.com/ryanfb/dzi-dl): Deep Zoom Image Downloader +* [iiif-dl](https://github.com/ryanfb/iiif-dl): Command-line tile downloader/assembler for IIIF endpoints/manifests +* [ChanThreadWatch](https://github.com/SuperGouge/ChanThreadWatch): Saves threads from \*chan-style boards and checks for updates until the thread dies + +### Compression +* [KGB Archiver](https://github.com/RandallFlagg/kgbarchiver): compression tool with unbelievable high compression rate +* [peazip](http://www.peazip.org/): File archiver utility + +### Network +* [NetLimiter](https://www.netlimiter.com/): Internet traffic control and monitoring tool for Windows + +### File systems +* [NTFS drivers for MacOS](https://www.seagate.com/ca/en/support/downloads/item/ntfs-driver-for-mac-os-master-dl/) +* [httpdirfs](https://github.com/fangfufu/httpdirfs/): A filesystem which allows you to mount HTTP directory listings + +### File conversion +* [AAXtoMP3](https://github.com/KrumpetPirate/AAXtoMP3): convert AAX files to common MP3, M4A, M4B, flac and ogg formats through a basic bash script frontend to FFMPEG + +### Utility Scripts +* [rclone_dirsize](https://gist.github.com/simon987/7aff5ca3e9ae6c755055ca7b350ef9f8): Get size of http directory listing with rclone +* [youtube-dl_soundcloud](https://gist.github.com/simon987/2dd7c57d65a741c93f5791bc984b97d1): snippet for using youtube-dl to download soundcloud playlists +* [rm_empty_subdir](https://gist.github.com/simon987/f5c2cd7602898615ac9bc8c762d9fe1d): Remove empty sub-directories on Windows +* [void-cat-uploader](https://github.com/takky1154/void-cat-uploader): This script automatically uploads all files inside a directory to https://void.cat. + +### Content sharing +* [opds](https://opds.io/): Easy to use, Open & Decentralized Content Distribution +* [ipfs](https://ipfs.io/): Protocol and network designed to create a content-addressable, peer-to-peer method of storing and sharing hypermedia in a distributed file system +* [h5ai](https://github.com/lrsjng/h5ai): HTTP web server index for Apache httpd, lighttpd, nginx and Cherokee + +### Data curation +* [DeepSort](https://github.com/CorentinB/DeepSort/): AI powered image tagger backed by DeepDetect +* [diskover](https://github.com/shirosaidev/diskover): File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch +* [fucking-weeb](https://github.com/cosarara/fucking-weeb): A library manager for animu (and TV shows, and whatever). +* [Everything](https://www.voidtools.com/): Locate files and folders by name instantly (Windows) +* [beets](https://github.com/beetbox/beets): music library manager and MusicBrainz tagger + + +### APIs & Online tools +* [thetvdb](https://www.thetvdb.com/): TV shows metadata (used by plex) +* [iqdb](https://iqdb.org/): Multi-service reverse image search