mirror of
https://github.com/simon987/awesome-datahoarding
synced 2025-04-10 14:16:48 +00:00
Splitted Download utilities into more useful sections
This commit is contained in:
Simon Fortier
GitHub
parent
d6867c0e31
commit
601c4f86ed
90
README.md
90
README.md
@ -1,68 +1,78 @@
|
||||
Note: This is only a first draft/brainstorm. I will try to organize the list with more useful sections in the future
|
||||
Note: This is only a first draft/brainstorm. I will try to organize the list with more useful sections in the future
|
||||
Feel free to contribute!
|
||||
|
||||
## Download utilities
|
||||
|
||||
### Download/Scraping utilities
|
||||
|
||||
* [annie](https://github.com/iawia002/annie): Youtube-DL alternative writtent in Golang
|
||||
* [aria2](https://aria2.github.io/): A lightweight multi-protocol & multi-source command-line download utility
|
||||
* [bazarr](https://github.com/morpheus65535/bazarr): Companion application to Sonarr and Radarr for downloading subtitles
|
||||
* [ChanThreadWatch](https://github.com/SuperGouge/ChanThreadWatch): Saves threads from \*chan-style boards and checks for updates until the thread dies
|
||||
### Web Archiving
|
||||
* [wail](https://github.com/machawk1/wail): Web Archiving Integration Layer: One-Click User Instigated Preservation
|
||||
* [Collect](https://github.com/xarantolus/Collect): A server to collect & archive websites that also supports video downloads
|
||||
* [CrowLeer](https://github.com/ERap320/CrowLeer): Powerful C++ web crawler based on libcurl
|
||||
* [curl](https://github.com/curl/curl): Tool and library for transferring data with URL syntax, supporting many protocols
|
||||
* [dzi-dl](https://github.com/ryanfb/dzi-dl): Deep Zoom Image Downloader
|
||||
* [FanFicFare](https://github.com/JimmXinu/FanFicFare): Tool for making eBooks from stories on fanfiction and other web sites
|
||||
* [FicSave](https://github.com/waylaidwanderer/FicSave): Online fanfiction downloader
|
||||
* [FlexGet](https://github.com/Flexget/Flexget): Multipurpose automation tool for content like torrents, nzbs, podcasts, comics, series, movies, etc
|
||||
* [floatplane_ripper](https://gist.github.com/simon987/0756c378ca2dfb0003931e26ff7fe270): Script to rip all videos from https://floatplane.rip/
|
||||
* [gallery-dl](https://github.com/mikf/gallery-dl): Fownload image galleries and collections from pixiv, exhentai, danbooru and more
|
||||
* [grab-site](https://github.com/ludios/grab-site): The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
|
||||
* [Heritrix](https://github.com/internetarchive/heritrix3): Extensible, web-scale, archival-quality web crawler
|
||||
* [HTTrack](https://www.httrack.com/): Download a website from the Internet to a local directory
|
||||
* [iiif-dl](https://github.com/ryanfb/iiif-dl): Command-line tile downloader/assembler for IIIF endpoints/manifests
|
||||
* [wikiteam](https://github.com/WikiTeam/wikiteam): set of tools for archiving wikis
|
||||
|
||||
### General
|
||||
* [annie](https://github.com/iawia002/annie): Youtube-DL alternative writtent in Golang
|
||||
* [aria2](https://aria2.github.io/): A lightweight multi-protocol & multi-source command-line download utility
|
||||
* [CrowLeer](https://github.com/ERap320/CrowLeer): Powerful C++ web crawler based on libcurl
|
||||
* [curl](https://github.com/curl/curl): Tool and library for transferring data with URL syntax, supporting many protocols
|
||||
* [Youtube-DL](https://github.com/rg3/youtube-dl): A command-line program to download videos from YouTube and a few hundred more sites
|
||||
* [Rclone](https://github.com/ncw/rclone): A command line program to sync files and directories to and from various cloud storage providers
|
||||
* [Plowshare](https://github.com/mcrapet/plowshare): Command-line tool to manage file-sharing site
|
||||
* [wget](https://savannah.gnu.org/git/?group=wget): Utility for non-interactive download of files from
|
||||
* [you-get](https://github.com/soimort/you-get): Dumb downloader that scrapes the web
|
||||
|
||||
### Application-specific
|
||||
|
||||
* [ChanThreadWatch](https://github.com/SuperGouge/ChanThreadWatch): Saves threads from \*chan-style boards and checks for updates until the thread dies
|
||||
* [floatplane_ripper](https://gist.github.com/simon987/0756c378ca2dfb0003931e26ff7fe270): Script to rip all videos from https://floatplane.rip/
|
||||
* [gallery-dl](https://github.com/mikf/gallery-dl): Fownload image galleries and collections from pixiv, exhentai, danbooru and more
|
||||
* [dzi-dl](https://github.com/ryanfb/dzi-dl): Deep Zoom Image Downloader
|
||||
* [FanFicFare](https://github.com/JimmXinu/FanFicFare): Tool for making eBooks from stories on fanfiction and other web sites
|
||||
* [FicSave](https://github.com/waylaidwanderer/FicSave): Online fanfiction downloader
|
||||
* [Google Images Download](https://github.com/hardikvasa/google-images-download): Python script for downloading images
|
||||
* [Instagram Scraper](https://github.com/dankmemes/instagram-scraper): Instagram-scraper is a command-line application written in Python that scrapes and downloads an instagram user's photos and videos. Use responsibly.
|
||||
* [RipMe](https://github.com/RipMeApp/ripme): RipMe is an album ripper for various websites. Runs on your computer. Requires Java 8.
|
||||
* [yt-mango](https://github.com/terorie/yt-mango): Youtube metadata archiver
|
||||
the Web (HTTP & FTP)
|
||||
* [Youtube-MA](https://github.com/CorentinB/YouTube-MA): Youtube metadata archiver
|
||||
* [PyInstaLive](https://github.com/notcammy/PyInstaLive): Instagram live stream downloader.
|
||||
* [RedditDownloader](https://github.com/shadowmoose/RedditDownloader): Scrapes Reddit to download media of your choice
|
||||
* [Scribd-Downloader](https://github.com/ritiek/scribd-downloader): Allows downloading of Scribd documents
|
||||
* [iiif-dl](https://github.com/ryanfb/iiif-dl): Command-line tile downloader/assembler for IIIF endpoints/manifests
|
||||
|
||||
### Download automation
|
||||
* [bazarr](https://github.com/morpheus65535/bazarr): Companion application to Sonarr and Radarr for downloading subtitles
|
||||
* [FlexGet](https://github.com/Flexget/Flexget): Multipurpose automation tool for content like torrents, nzbs, podcasts, comics, series, movies, etc
|
||||
* [Jackett](https://github.com/Jackett/Jackett): API support for torrent trackers (works with Sonarr, Radarr and others)
|
||||
* [Lidarr](https://github.com/lidarr/Lidarr): Music collection manager for Usenet and BitTorrent users
|
||||
* [Mylar](https://github.com/evilhero/mylar): An automated Comic Book downloader (cbr/cbz) for use with SABnzbd, NZBGet and torrents
|
||||
* [Plowshare](https://github.com/mcrapet/plowshare): Command-line tool to manage file-sharing site
|
||||
* [PyInstaLive](https://github.com/notcammy/PyInstaLive): Instagram live stream downloader.
|
||||
* [Radarr](https://github.com/Radarr/Radarr): A fork of Sonarr to work with movies à la Couchpotato
|
||||
* [Rclone](https://github.com/ncw/rclone): A command line program to sync files and directories to and from various cloud storage providers
|
||||
* [RedditDownloader](https://github.com/shadowmoose/RedditDownloader): Scrapes Reddit to download media of your choice
|
||||
* [RipMe](https://github.com/RipMeApp/ripme): RipMe is an album ripper for various websites. Runs on your computer. Requires Java 8.
|
||||
* [Sick-Beard](https://github.com/midgetspy/Sick-Beard): PVR for newsgroup users (with limited torrent support)
|
||||
* [Radarr](https://github.com/Radarr/Radarr): A fork of Sonarr to work with movies à la Couchpotato
|
||||
* [Sonarr](https://github.com/Sonarr/Sonarr): PVR for Usenet and BitTorrent users
|
||||
* [wail](https://github.com/machawk1/wail): Web Archiving Integration Layer: One-Click User Instigated Preservation
|
||||
* [Collect](https://github.com/xarantolus/Collect): A server to collect & archive websites that also supports video downloads
|
||||
* [wget](https://savannah.gnu.org/git/?group=wget): Utility for non-interactive download of files from
|
||||
* [wikiteam](https://github.com/WikiTeam/wikiteam): set of tools for archiving wikis
|
||||
* [you-get](https://github.com/soimort/you-get): Dumb downloader that scrapes the web
|
||||
* [Youtube-DL](https://github.com/rg3/youtube-dl): A command-line program to download videos from YouTube and a few hundred more sites
|
||||
* [Youtube-MA](https://github.com/CorentinB/YouTube-MA): Youtube metadata archiver
|
||||
* [yt-mango](https://github.com/terorie/yt-mango): Youtube metadata archiver
|
||||
the Web (HTTP & FTP)
|
||||
|
||||
### Compression
|
||||
|
||||
## Compression
|
||||
* [KGB Archiver](https://github.com/RandallFlagg/kgbarchiver): compression tool with unbelievable high compression rate
|
||||
* [peazip](http://www.peazip.org/): File archiver utility
|
||||
* [PIGZ](https://zlib.net/pigz/): Multi-threaded gzip
|
||||
* [WinRAR](https://www.rarlab.com/download.htm): Can decompress RAR and zip files.
|
||||
|
||||
### Network
|
||||
|
||||
## Network
|
||||
* [NetLimiter](https://www.netlimiter.com/): Internet traffic control and monitoring tool for Windows
|
||||
|
||||
### File systems
|
||||
## File systems
|
||||
* [httpdirfs](https://github.com/fangfufu/httpdirfs/): A filesystem which allows you to mount HTTP directory listings
|
||||
* [mergerfs](https://github.com/trapexit/mergerfs): a featureful union filesystem
|
||||
* [NTFS drivers for MacOS](https://www.seagate.com/ca/en/support/downloads/item/ntfs-driver-for-mac-os-master-dl/)
|
||||
|
||||
### File conversion
|
||||
## File conversion
|
||||
* [AAXtoMP3](https://github.com/KrumpetPirate/AAXtoMP3): convert AAX files to common MP3, M4A, M4B, flac and ogg formats through a basic bash script frontend to FFMPEG
|
||||
* [html2warc](https://github.com/steffenfritz/html2warc): Convert web resources to a single warc file
|
||||
|
||||
|
||||
### Utility Scripts
|
||||
## Utility Scripts
|
||||
* [Backblaze B2 sync backup script](https://gist.github.com/AlexanderProd/cb645cf858fd5c89780e7df267226b80): Script to sync mutliple directories with Backblaze B2
|
||||
* [Misc download scripts](https://github.com/simon987/Misc-Download-Scripts): Scripts for downloading content from various websites
|
||||
* [rclone_dirsize](https://gist.github.com/simon987/7aff5ca3e9ae6c755055ca7b350ef9f8): Get size of http directory listing with rclone
|
||||
@ -70,12 +80,12 @@ the Web (HTTP & FTP)
|
||||
* [void-cat-uploader](https://github.com/takky1154/void-cat-uploader): This script automatically uploads all files inside a directory to https://void.cat.
|
||||
* [youtube-dl_soundcloud](https://gist.github.com/simon987/2dd7c57d65a741c93f5791bc984b97d1): snippet for using youtube-dl to download soundcloud playlists
|
||||
|
||||
### Content sharing
|
||||
## Content sharing
|
||||
* [h5ai](https://github.com/lrsjng/h5ai): HTTP web server index for Apache httpd, lighttpd, nginx and Cherokee
|
||||
* [ipfs](https://ipfs.io/): Protocol and network designed to create a content-addressable, peer-to-peer method of storing and sharing hypermedia in a distributed file system
|
||||
* [opds](https://opds.io/): Easy to use, Open & Decentralized Content Distribution
|
||||
|
||||
### Data curation
|
||||
## Data curation
|
||||
* [baobab](https://github.com/GNOME/baobab): Graphical disk usage analyzer
|
||||
* [beets](https://github.com/beetbox/beets): Music library manager and MusicBrainz tagger
|
||||
* [Calibre](https://github.com/kovidgoyal/calibre): Ebook manager
|
||||
@ -95,11 +105,11 @@ the Web (HTTP & FTP)
|
||||
* [tree](http://mama.indstate.edu/users/ice/tree/): 'tree' command for linux
|
||||
* [WinDirStat](https://windirstat.net/): Disk usage statistics viewer and cleanup tool for Windows
|
||||
|
||||
### APIs & Online tools
|
||||
## APIs & Online tools
|
||||
* [iqdb](https://iqdb.org/): Multi-service reverse image search
|
||||
* [thetvdb](https://www.thetvdb.com/): TV shows metadata (used by plex)
|
||||
|
||||
### Hardware / Monitoring
|
||||
## Hardware / Monitoring
|
||||
* [CrystalDiskInfo](https://crystalmark.info/en/software/crystaldiskinfo/): A HDD/SSD utility software which supports a part of USB, Intel RAID and NVMe.
|
||||
* [Hard Drive Sentinel](https://www.hdsentinel.com/): Multi-OS SSD and HDD monitoring and analysis software
|
||||
* [smartmontools](https://www.smartmontools.org/): Control and monitor storage systems using the (SMART) built into most modern ATA/SATA, SCSI/SAS and NVMe disks
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user