Noice write up.
My goto's for searches (should work with most engines)
Movie Search
intext:"*" (avi|mkv|mov|mp4|mpg|wmv) -inurl:(jsp|pl|php|html|aspx|htm|cf|shtml) intitle:"index.of./"
Image
intext:"*" (bmp|gif|jpg|png|psd|tif|tiff) -inurl:(jsp|pl|php|html|aspx|htm|cf|shtml) intitle:"index.of./" /wp-content/uploads/
Audio
intext:"*" (ac3|flac|m4a|mp3|ogg|wav|wma) -inurl:(jsp|pl|php|html|aspx|htm|cf|shtml) intitle:"index.of./"
Replace asterisk with search term.
On linux I use wget, axel and aria2 and if you're unsure of cli you can use uget which is basically a gui for those 3.
If you're using jdownloader make sure you install the adware free version.
ripme is gud for pics, image galleries and insta etc.
and youtube-dl will get you most vids off vid sites/tv sites you'll ever want.
So I usually save a lot of posts from this sub, the other 2meirl42meirl and depression. Recently I lost all of them but I've managed to get most of them using ripme.
I'm still missing one which had some nice comments. It was a picture with some text suggesting they want to cuddle? I can't remember which sub exactly it was from, if someone finds that post link me pls.
PS: I was too shy to ask from my main account
RipMe is my preferred tool. Also supports imgur, GFY, and other popular image/video hosting sites. Jdownloader 2 works as well, but may be overkill for this purpose.
If you're not aware of the tool ripme yet, I can highly recommend getting it. It supports erome.
The download link is a bit below, but here's a link to the latest release.
Download the ripme.jar file and just run that. It's pretty self explanatory from there.
http://i.imgur.com/XHBa71T.jpg
I just tried it and it doesn't appear to function, you'd probably have to request it, but somehow I don't think it's going to be a priority.
I love the concept, and thank you for sharing this!
I use something similar for batches of images called, RipMe:
https://github.com/RipMeApp/ripme
RipMe is a Java (.jar) app that supports clipboard monitoring. You should take a look at the code for ideas on URL validation. It seems to me that you are on your way toward making a video downloading version of that app.
Note: RipMe CAN download videos, but it does not give you the control that youtube-dl does. A similar app (particularly with clipboard monitoring) that was specific to youtube-dl would be awesome. I'm no Java dev, but I'd be happy to help you out with ideas and testing.
>go to the release page
>download and run the .jar file under the latest release
You should be able to double click it in windows once java's installed, if not the command line is
java -jar /path/to/where/you/saved/the/jar/file/ripme.jar
once it's open it's fairly straight-forward - paste the url (it can even catch urls from the clipboard), set the destination directory and click "Rip". There are bells an whistles, take some time to have a look at the settings.
And enjoy your tame no-nipples tumblr pr0n!
You can use the config option history.end_rip_after_already_seen = X
. It will cause ripme to end the rip after it finds X urls that where already downloaded
Use this: https://github.com/RipMeApp/ripme
Download albums from: * imgur
https://github.com/RipMeApp/ripme
Download the latest jar from "releases".
There's no view of the progress of 1 file downloading, so when it lists the files it's downloading and then says "Downloading next page" and it seems to be just sitting there, you just need to wait. It's downloading those files in the background.
It's still uploading, but here's a link to the Google Drive folder (should be done in an hour or so).
Alternatively, you can use the wonderful RipMe utility.
EDIT: It's finished uploading, so Google Drive is an alternative now.
Im a noob. So i just have to translate the words in https://github.com/RipMeApp/ripme/blob/translation/src/main/resources/LabelsBundle.properties ?
Edit: I dont have Github but I translated the file as good as I could
Sometimes there's a download button at the bottom of the album off a little '...' menu. If that doesn't work, ripme is a java app that does the job nicely. There's a wiki on how to run it.
There's this tool made by a member of this subreddit, but it is unfinished and yet to be released. I don't know if there's an existing tool that can save text posts. Someone else will probably know if there is one.
https://www.reddit.com/r/DataHoarder/comments/6t5jss/release_ripreddit_semiautomatic_discord_based/
If you want to download only the images posted on a subreddit then there's ripme
RipMe is a Java app used to rip media from social media websites. I don't do anything in Java myself, but the code seems to be quite straightforward from what I've seen.
Maybe RipMe? I use it a lot for downloading all the posts from a given user, it works great with Reddit. I've never tried it on my "saved" page before, but it's definitely worth a shot.
Can also use Ripme to rip every post from her insta and watch them all that way. The tool was created as a way to watch/collect content that has awful video player controls or is locked behind something silly when inspect element
can free it.
Hm. You're right. Seems like web.stagram.com changed their html. I'll look into it but I'm busy atm so it might take longer than usual.
If you want to download that profile right now and you have a PC then try RipMe - it's an open source downloader for many sites and also supports instagram.
I've added support for this in there PR, it should be merged in for ripme 1.7.64, however because of how ripmes subalbum ripping works you'll have to download every page seperately
IE: ripping https://m.xhamster.com/users/USER/photos will only rip the first page, to rip all the pages you'd have to enter https://m.xhamster.com/users/USER/photos/{1-LASTPAGE}
I don't rip from thousands of tumblers but maybe about 10 large ones. The program i use is ripme for images and videos. its nice because i can queue up all my old jobs in two clicks and it checks for existing files before downloading again. The one problem i have and i'm sure you will have is there is a limit on how many API requests you can send to tumbr (1000 per hour/ 5000 per day). You can just use a new API key to get around that but i am not aware of a ripping program that can do this automatically while resuming the rip.
The more i think about it, ripme is not the solution for you but maybe it is a starting point or can help someone else will a smaller request amount.
> Im a noob. So i just have to translate the words in https://github.com/RipMeApp/ripme/blob/translation/src/main/resources/LabelsBundle.properties ?
Yup
> Edit: I dont have Github but I translated the file as good as I could
Thanks!
Get RipMe, make a link list containing the users and subs you want to download, save it as list
in the same directory as ripme.jar
then do cat list | parallel -j32 'java -jar ripme.jar -t 8 -u {}'
to do the initial download, then you can run that command again as often as you want or add it to cron, etc.
There's a tool called ripme that has a graphical interface (if that's something you're into) and works for multiple sites. I use it to download entire subreddits pretty often, it's nice since it nests albums into folders
Install Python, save the code as a .py
file, then double click on that file.
Though if you're just trying to download posts from a subreddit and don't actually intend on learning Python, you'll be better off using an existing tool like RipMe or gallery-dl to do it. That's way easier than trying to do it yourself.
Really high?
Like it depends on what they are using to scrape the subreddit the two I ran across yesterday ripme and Reddit Media Downloader both only run when called so it would be highly unlikely those would pick it up because they'd have to be called within the few seconds it was available.
However anything built to pull constantly from reddit or pushshift's API could easily archive something that was only visible for 20 seconds.
For example your post here https://www.reddit.com/r/pushshift/comments/q783nh/does_pushshift_or_any_other_archiver_save_any was collected by pushshift only 11 seconds after it was posted.
I use RipMe. Its a GUI program, at least on Windows, don't have a Mac which is what your asking for. Easy to use aswell. None of the sites you specific seem to be on the supported list but I'm not sure if its been updated as far as i know, Best id say is just to test it out.
Just to clarify : Hentoid does NOT reprocess / compress any picture when downloading. What you get is exactly the same picture you would get when downloading from any desktop browser.
However, I can understand viewing it through Nox isn't ideal, and the emulation certainly renders a lower quality image than what you would get by viewing it directly on your disk.
A good place to start for a PC "equivalent" would be https://github.com/RipMeApp/ripme
Just download
​
https://github.com/ripmeapp/ripme/releases
Install java
open the Jar file
Copy and paste the pornhub / erome link in the tool --> download
​
#Enjoy
p.s. PC non-mobile should work on MacOS aswell as its java.
youtube-dl is for downloading videos. If you want a similar tool for images, then try gallery-dl. Or if you want a simpler GUI maybe try ripme.jar:
You don't need to be computer savvy. You just need to be able to read the directions for downloading Java, Maven, and this https://github.com/ripmeapp/ripme
There are other apps that do the same thing. Someone with a few decent computers could run it on each and rack up a shit ton of pictures very quickly. It would still probably take days to a week though. My 8 year old laptop that runs too hot and is connected through a VPN can download tens of thousands of photos in a day. If they were all smaller files I imagine over 150,000 would be pretty easy.
Photo size varies wildly but I would estimate most photos taking about .5 seconds to 1 second. If I remember when I get home I'll run a few tests on file size.
Hi, I've got a program called Timesearch, but it doesn't download media. I leave the media for other tools, maybe RipeMe can do what you need.
A quick search on github returns this:
https://github.com/RipMeApp/ripme
Which seems to have support for downloading from instagram. Not sure if it does exactly what you want.
Instagram has an API, so in principle it should be possible to write exacly what you want as a small script. Like
https://github.com/LevPasha/Instagram-API-python
Should let you get access to the API from python, and save you some hassle if you have a python dev available. You might have to apply for a API key somehow from instagram, which should be straight forward if you have an instagram account, and I don't sadly.
You should also ask r/socialistprogrammers there there might be people who could whip something up.
I hope my title is true. I downloaded all pictures with this: https://github.com/ripmeapp/ripme
With exception of the videos and those eight pictures only showing question marks I should have all posts right?
It seems like RipMe supports the site, but you might need to test that. Just give it the link from your browser's address bar and try it out, it might even download whole accounts
I monitor quite a few accounts and subreddits for a variety of reasons, and here is how I do it:
On a Raspberry Pi using Raspbian Lite, I installed Java and RipMe. RipMe can pull lists of urls, so I made various lists with different requirements, such as once daily, once weekly, etc.
Next, I made cronjobs to run the app using the lists, and saving to network drives. This would ensure every X amount of time, it would run the scrapes again.
Doing it separately on a Pi offloaded the scraping tasks so my main machines wouldn't have to deal with it, and saving to network drives backed everything up.
I also use scheduled wget tasks using cronjobs to scrape pages. RipMe is great for images, gifs, and videos, and wget grabs everything else using the same setup.
ty i also just found that. but how do you rip the images there to put into zip/rar ? I use cdexviewer to read the manga.
the site isn't listed for FMD app https://github.com/fmd-project-team/FMD/releases
Nor is it supported by ripme https://github.com/RipMeApp/ripme/releases/
https://github.com/RipMeApp/ripme
Get this and aim it at a subreddit, downloaded about 200 images per minute on my connection. Images average out at 1MB each.
15GB = ~15,000MB / 200 = ~75 minutes to hit your target. This will of course be quantity over quality. But who's going to scrutinise 15,000 memes?
If you mean individual photos, if you have javascript disabled, then you can simply click on a image and there's an option to download it in the bottom right. If you mean the entire album, then you can save it using a program like ripme: https://github.com/ripmeapp/ripme
You would have to log into your reddit account using ripme, but this isn't possible yet:
https://github.com/RipMeApp/ripme/issues/1245
https://github.com/RipMeApp/ripme/issues/1273
There also is no cookie support yet, so logging in isn't possible at all.
You’ve got a few options, worth trying some of the ones below:
https://github.com/RipMeApp/ripme
> When modifying the rip.properties file to change "log.save" to true I can't find where the log is being written
It should be in the dir ripme was started from
> So where does the log get written?
No idea, but if you find out leave a comment and I'll update the wiki
> Where is the canonical list of all the available options?
Also worth checking out some of the purpose built tools for this. They have a lot more functionality and can be automated:
https://github.com/RipMeApp/ripme
For ripme, you'd have to write your own code for those unsupported sites, see: https://github.com/ripmeapp/ripme/wiki/How-To-Create-A-Ripper-for-HTML-websites Rather than use a program or browser extension, I would be inclined to write everything in my own code as it would be more efficient and easier to manage downloads to how I would want them.
> I was unsure if it was being updated with each release. Does the docker container get updated with each release?
It should be but it does sometimes get forgotten. It's easy enough to do by hand via editing the docker file
> Is this what I should be using, or is there another docker container that is kept more up to date?
The official docker is https://github.com/RipMeApp/ripme-docker
> Also how would I go about changing the properties file permanently for each container?
I;ve not tried it but docker exec -it <container> bash
should start bash in the container. From there just edit ripmes config file
ripme and TumblThree are both safe bets - I prefer TumblThree because it's a lot more user friendly (and lets you get around profiles that are NSFW-locked or password locked (in some cases).
Thanks for the reply. I was planning on doing something like that anyways so it should work out for me.
A quick question about the wiki, did ripme used to rip the files from URLs directly in the past, but that was a feature that was taken out? I am asking because one of the flags for using ripme says that you can rip URLs from a file
usage: java -jar ripme.jar [OPTIONS] -4,--skip404 Don't retry after a 404 (not found) error -d,--saveorder Save the order of images in album -D,--nosaveorder Don't save order of images -f,--urls-file <arg> Rip URLs from a file. <---
Pushshift.io apparently ingests almost a terabyte of Reddit data every month, according to its creator, and I think that's purely Reddit data, and not external images/videos/link. I don't know how they're set up behind the scenes, but it just shows that it's definitely possible.
You could spread it over many different hosts in different locations, and use many different accounts to access the API.
I "casually" hoard images/gifs (no metadata or comments, just the media) from just around 700-750 subreddits with the application Ripme (and some custom scripts for managing Ripme), some subreddits ripped twice daily and some every 2 days, and I don't get blocked by Reddit in any way, even though this is all running from one server. I don't think Ripme uses the official API, but I don't know for sure. That might have something to do with it. Currently I'm at a little bit more than 12TB, but that's from the last 4-5 years, so this is not a super comprehensive archive.
I mainly use youtube-dl for videos. It just works.
Collect as personal web site/video archive. I stopped developing it some time ago and some stuff still doesn't work great, but it's ok most of the time.
RipMe, an image/media downloader (for sites like reddit, twitter...).
For some other archiving projects I use custom scrapers that only download specific content and metadata. Some part of that is automated.
Inspired by the post today, “Can we make this sub just selfies again and not soft porn?! It’s a great sub but it’s becoming NSFW and makes me not even want to post! You don’t have to wear underwear to get attention ladies!”
http://sankeymatic.com/manual/ - Used to Create Chart
https://github.com/ripmeapp/ripme - Used to rip images
After ripping the sub (I have no idea how “good” of a job it did), I randomized the pictures and manually data mined 1000 of them while watching 1.5 episodes of House MD.
Honestly, I think there are inherent problems with this data. I am led to believe there is a lot of deleting that goes on in this sub. These deletions may lean towards the risqué posts? I honestly thought there would be a much lower percentage of “normal” selfies.
Inspired by the post today, “Can we make this sub just selfies again and not soft porn?! It’s a great sub but it’s becoming NSFW and makes me not even want to post! You don’t have to wear underwear to get attention ladies!” http://sankeymatic.com/manual/ - Used to Create Chart https://github.com/ripmeapp/ripme - Used to rip images After ripping the sub (I have no idea how “good” of a job it did), I randomized the pictures and manually data mined 1000 of them while watching 1.5 episodes House MD. Honestly, I think there are inherent problems with this data. I am led to believe there is a lot of delating that goes on in this sub. These deletions may lean towards the risqué posts? I honestly though there would be a much lower percentage of “normal” selfies. You guys are all beautiful. Keep doing your own thing, whatever it is that makes you happy!
Not sure about any chrome add ons, but have a look at these tools if you just want rip some image subreddits:
https://github.com/RipMeApp/ripme
Yup.
This is the issue right here. https://github.com/RipMeApp/ripme/issues/1093#issuecomment-460057619
It just creates a folder and says you've already downloaded the album but it doesn't. I've seen the 4pr0n RipMe but never used it.
What I suggest you do is you download ripme.jar you'll also need Java installed if you don't already have it. It's an image ripping tool, where you can paste links of websites and it'll download every image found. Anyway, what you do next is you copy the link of any celeb subreddit and it'll download all of its content.
> After a user has been ripped, another script calls jdupes to look for duplicate files leaving symlinks in place so ripme does not re-download them
Why not just use the remember.url_history
feature? It will remember all the file urls ripme downloads and skip them in future rips
Anyhow it is currently no possible to change this behavior but there should be. I've filed an issue here and will get on fixing this for the 1.7.62 release
If you're downloading image albums, The RipMe app is pretty nice, and in active development.
It started off as a tool to download imgur albums and scrape gonewild posts, but it's had a lot added to it. Mostly porn sites, but also a number of others.
It'll also download videos from a handful of sites, which was totally unexpected to me. The only one I know of is xhamster, but there could be more.
If you're not aware of the tool ripme yet, I can highly recommend getting it. It supports erome.
The download link is a bit below, but here's a link to the latest release.
Download the ripme.jar file and just run that. It's pretty self explanatory from there.
you're welcome, btw you can use this similar strategy to get higher quality versions of instagram pictures. go to that page info like in my screenshot, right click the img url from that and then enter the url and you'll get the full size, or to make it way way simpler and faster, use "Ripme" https://github.com/ripmeapp/ripme/releases lets you download everything on an instagram or twitter page automatically super fast in the full res.
> I will check out ripme because I’ve never heard of it, and see what actually happens
I'm one of ripme current maintainers, if you have any concerns feel free to PM me/make a github issue at https://github.com/RipMeApp/ripme/issues
I'm one of the ripme devs, ripme ought not be doing that. Each image/video should be saved in one folder named after the user/tag/post that's being ripped
Could you make a bug report at https://github.com/RipMeApp/ripme/issues
Want to harvest Instagram too? Use ripme. Rip Instagram and other websites with one click.
https://github.com/ripmeapp/ripme :)
It is not the same thing as harvesting but I wanted to post this here.
My statement isn't wrong, just weird phrased. TumblThree is api key free, RipMe is not.
For RipMe, you either you use the default key, or you have to make one for yourself if you don't want to share the request with the other users. Yet, you still can run into the rate limit with your (or the default) key as there is no rate limiter.
Sure, you can make several of it, but its still more cumbersome than not having to make any at all.
> How do I download/use it? I'm not familiar with GitHub
Download ripme.jar from here:
https://github.com/ripmeapp/ripme/releases
Requires JAVA installed from here:
Have you tried any virtualization on it yet? I'm curious as to whether you could do virt-manager on there,(vt-d/x Permitting) and as for OS choices, I'd be leaning more towards fedora workstation, it's as stable as red hat and docker is already in the yum repo.
I also thought this might be useful for acquiring all that anime.
This is damn near a crime against humanity. Photobucket literally erased over a decades worth of laboriously crafted graphics and illustrations, MANY of which were created to help others with research, development, guides, tutorials, documentation, etc. Now all gone.
I had about 300 graphic files I created for about a dozen tutorials I posted throughout the years (pre-imgur), totaling 8MB. Photobucket wants me to pay $400 per year for that!?!? THEY'RE FUCKING INSANE!
If they're losing money, they should have just gone after those that eating a shit-ton of bandwidth, and left the small accounts alone that are barely consuming any costs.
Oh well, I ripped all my albums and just saved them, mainly for nostalgia sake.
If you need to rip your albums, you might find Photobuckets "download album" feature is broke (surprise surprise).
I used a program called RipMe to download my albums. First you need to make your album public, then you can add the album URL into RipMe and it will fetch all the images.
Here's a link to RipMe's GitHub, scroll down for the java file download link which can be run on linux or windows. https://github.com/4pr0n/ripme
Here's the wiki page that explains how to run RipMe https://github.com/ripmeapp/ripme/wiki/How-To-Run-RipMe
u/SENDME-YOUR-NUDES I gotchu.
https://github.com/4pr0n/ripme
Steps(make sure Java is up to date for this):
Scroll down halfway until you see "Downloads" Click on that.
Grab the latest release (1.5.7) by clicking "ripme.jar"
Make sure you setup the .jar file to open with Java Platform SE Binary (right click>Open With>Choose another app) Sometimes it'll be there if you mouse over the "Open With" arrow.
Once opened, it will automatically generate two files: rip.properties and history.json (including a rips folder). This is normal.
Go to the "Configuration" tab and checkbox "Autorip from clipboard" (if not checked by default). Copy the desired link of choice (supported links can be seen here).
After it's done, you'll see your saved goodies in the "rips" folder (same location as ripme.jar) -OR- by clicking "Open ./rips/ripped folder goes here" after the rip is done.
I've tested the link beforehand to make sure it works. Happy ripping. :)