Not OP, but as someone with a similarly large hand-picked collection I originally used VisiPics and Awesome Duplicate Photo Finder (ADPF). Both are good in their own way - VisiPics has adjustable filtering and can find images with slight color shifts, but ADPF is better at photos that are cropped differently. VisiPics is also getting on in age and unusual color spaces or photo formats often throw it for a loop.
In the end, though, I switched to DupeGuru. There's a bunch of photos that come up every time I scan that are similar enough that they always show up when I add new pictures, and always sorting through those got to be a pain. DupeGuru has ignore lists, customizable autoselect options, and enough other features (hardlinking, not restricted to photos, etc.) to make it worth it.
Haven't used all that many tools, but http://www.visipics.info/ is one that's worked pretty well for me for images. The interface kinda (yeah, okay, definitely) stinks, but the image comparison works extremely well. I notice that DupGuru, (mentioned by u/pogowolf!) also has fuzzy image scanning, but I've not used that tool myself. I'll have to check it out, but the more free options the better, right?
As someone who also manages a fairly large wallpaper collection (11.6k), I suggest you use VisiPics. It scans folders for duplicate images, so you can keep the duplicates out of your collection.
Thanks for these. I collect Huey photos and there were a whole bunch in here that I didn't have. I have the same problem with duplicates, there must be a bit of software that recognizes duplicate photos based on image content, irrespective of size, compression etc? I'll google for one.
Edit: Found visipics, looks hopeful
Now that you have all your photos again one handy tool to download is Visipix. It is a visual photo duplicate detector. Because it is comparing the files visually it will detect images that are the same but different resolutions and other similar duplicate but not exact duplicate images.(although watch out for it trying to delete your touched up versions of photos). So if you have duplicate file names just keep them both and let Visipix take care of them after the fact.
Next I would check out PhotoMove. It will help you sort all those photos by year and month based on the date taken in the exif data(meta data left by the camera itself). This will cut your sorting down by a ton.
After you get everything sorted and deduplicated I would upload the whole thing to Google photos. They don't keep the original version but the high res versions are close enough and it makes a good back up. It also allows you to search them by location.
visipics is very good for finding duplicate images, it will also give you previews of the duplicate images and give you size info at the bottom. Simple as set filter > scan > wait > mark for deletion > delete/move
Edit: You can also get it to move the images to another folder if you want to keep them for some reason.
If there are images that look very similar but are different id suggest setting the filter to strict and going into options and disabling loosen filter automatically
It can also scan for 90° rotations if you have images that are the same but rotated.
This is a fantastic program. And it's free. http://www.visipics.info/
It not only can find identical pictures, but very similar pictures. For instance, different sized versions of the same picture. Or two pictures, one with a border or one with text, or one with a slightly different tint, or two pictures taken a second apart, etc.
That's maybe a bit late as you're already uploading, but if you want to search for duplicate pictures to delete them, you can try visipics
And thanks for this album !
VisiPics is what I use, has a nice amount of configuration I think you might benefit from. Idk of a OSX app to do this, so unless you have access to a Windows PC I guess this is a useless post.
I recently used VisiPics for just such a project. Note that there are better tools for finding exact file matches, but this one does work more like TinEye, and you can set thresholds for similarity.
http://www.visipics.info/index.php?title=Main_Page
I have used this before and it worked well. Not sure if it is still supported but it does a good job analyzing the photos for a percentage match and can auto process.
I used this http://www.duplicate-finder.com/photo.html
and this: http://www.visipics.info/index.php?title=Main_Page
to remove duplicates from my collection.
Edit: It would be awesome if you can take the good collection, zip it and then upload it somewhere (mega.nz?)
VisiPics - Scans through a specified folder and finds identical pictures.
FlashHacker - Allows you to have flash movies playing in fullscreen on one monitor, while working on another monitor.
AMP WinOff - Best freeware program I've found for setting a specific shutdown time.
VisiPics is what I use and it finds duplicate with different resolutions. I love it.
First I want to add, disable the admin account for security, especially if you have your NAS exposed to the internet.
I’d copy from DSM in the browser, the photos from admin account to user1 and then run a duplicate file scanner app on the Moments directory.
I use VisiPics to check and remove duplicates.
VisiPics, on windows at least, it was always best even as a linux user I've ran it in a vm on a samba share against 8 millions images and it performs really well.
There is a program for that on the computer. It’s called Visipics (I made a post on it more than a year ago at r/declutter ) which is free, but not for Macs. My dad uses it and it helped him a lot.
Download them all, get a software that detects duplicates and deletes them, create a new google account and upload all remaining pictures > share albums with your main account. You should download them regularly anyway in case you get locked out of your account for some reason.
Software that detects duplicates:
http://www.visipics.info/index.php?title=Main_Page is free but if you have that many pictures or you need something more professional feel free to spend some money on a more advanced tool (there are tons I don't want to turn this post into advertisement though)
For images I use VisiPics. It's free and finds not only images that are exact duplicates, but also those that are similar, like the same image in different resolutions/states of compression.
So if you're an idiot like me and don't sort your stuff, the program VisiPics will save your ass.
Select a folder and it will scan for images that look alike (so not the exact same, it might pick up your before/after cumshot pics, set the filter slider to strict to surprise make it more strict). When it's done it will show all images it detected to the left and you can select which ones to delete (I always keep the highest resolution, shown at the bottom).
Try Visipics.
VisiPics, it's function is to analyze all your images and find duplicates. It's a really powerful tool i used all the time.
Here's the info on it
What type of operating system do you run?
On Linux FSlint is a hash comparer, it'll only work for identical files. This stackexchange post exists for Linux solutions. Offers a few suggestions for programs more robust than straight up hash comparison.
For Windows VisiPics seems promising.
http://technet.microsoft.com/en-us/library/dn520872.aspx
The file name shouldn't matter, but different exif data -> different file content -> different filehash.
I've used a free app called VisiPics for this purpose before though, and it worked pretty well.
http://www.visipics.info/index.php?title=Main_Page
Find the two folders on your hard drive. Mark the folder you prefer to keep original pictures and then click on the square + (1). Then mark the folder you want to delete the duplicates from and click on square + (1). The folder highest on the list will keep the orginal pictures, and the folder below will be chosen if any duplicates for deletion.
Choose strict if you only want to delete pictures that are accurate duplicates, not images that are similar.
Press the green square to start the progress finding duplicates.
After the progress is done, press "auto-select". This will pick images from the folder you chosed last on step 1.
Delete/move images to complete the progress.
If you have any questions, i'll gladly answer them. Either here or a PM.
img 2329 WTF? also some other pics are completly unreleated (like a 5 :P)
Anyway you should use something like VisiPics to avoid having too much duplicates ( for example single picture of Nonon is like 6+ time duplicated there )
Found few nice pics that weren't in my collection, nice share thanks :)
sweet collection, but you have a lot of duplicates
I had this problem a while back and found Visipics.
It will help get rid of most of the duplicates!
http://www.visipics.info/index.php?title=Main_Page
Visipics. Free! I use it to weed out dupes in my collection. You can even set the threshold of how closely or loosely it matches to find the same image at different resolutions, or even different images from the same set. It's an amazing app.
Great collection! I was happy to see a few fresh faces in there!
For the future, I suggest you consider using VisiPics to remove any duplicates.
Uninstall that malware "cleaner" app before it has a chance to break something or delete files it shouldn't.
Assuming you have a computer with Windows, install VisiPics on your computer, move all the photos to your computer, and use VisiPics to find the duplicates. It's completely free and runs locally without an internet connection.
For exact duplicate files I use DoubleKiller Pro. If you change some of the match options you can also have it do a decent job of finding duplicate pictures that have slightly different metadata (by looking at the last/first x bytes).
For similar images (resizes, rotations, crops, etc) I use VisiPics. With the strict setting I pretty much trust that it's only matching true duplicate. As you start moving the slider it will end up picking up a lot of pictures that were taken in quick succession with maybe a smile changing or something.
In addition to these I use XnView to rename my photos based on the metadata (date/time mostly) which helps with filtering and identifying duplicates.
Disclaimer: I started using these programs years ago and at the time they were the best I found to fit my needs. I haven't spent any time in recent years looking for replacements. Also, I take no responsibility for the integrity of the downloads on the linked sites.
A lot of weird suggestions. I think you're trying to delete photos which are the same, but resized? If so, try VisiPics: http://www.visipics.info/index.php?title=Download
It's not the easiest to use, but it's the only free software I found which scans a folder then lets you bulk auto-select the duplicates. It leaves behind the largest file (in terms of dimensions) while deleting all smaller ones. E.g. it'll leave the 3000x3000 original, but delete all the 150x150 and 400x400 ones.
Without getting into really fancy stuff like heuristics, you could compare at block-level. I can't figure out how to load files by NTFS-block, but I don't think there's a reason you couldn't load any binary file an arbitrary amount at a time, say 1024 or 4096 bytes at a time, hash each block, then compare the number of matching blocks.
Maybe there's something I'm missing, though.
Also, I don't know what kind of files you're comparing, but I think that if a single byte is changed near the front of a compressed file like a JPEG, it might cause a shift in the bit sequence for every block after it (or maybe even before). So, comparing compressed files might be very difficult. I know there's third party tools available for this, though--I've used VisiPics for image deduplication.
Visipics or DigitalVolcano's Duplicate Cleaner can help you detect and delete duplicate image files by visual similarity.
As for organizing photos locally, I haven't found anything better than the old, discontinued (but still working!) version of Google Picasa. IIRC, it detects duplicate photos too.
I feel like a shill for them at times, but visipics is a great option.
You just add the folder you want scanned, move the filter to strict, and hit the play button. It will show a list of all duplicates found, and you can hit auto-select, which selects smaller or lower resolution files to get rid of. Confirm with A/B comparison in the preview, then delete/move them, move the filter slider down two notches scan again, auto-select, confirm, etc.
At "loose" levels of filter it will find pictures with similar backgrounds, and be able to match thumbnails/web sized to originals, etc. I use it all the time on my own reference files I get online where I might snag a lower resolution or slightly color edited version that I already have in my library, or my boss that backed up her phone to separate folders every month for 5 years and wanted to consolidate all her pictures.
I've had some success with Visipics for finding duplicates.
Windows Explorer should allow you to sort by height and width, although I don't know if total pixels is possible...
As far as galleries, unfortunately I don't have much advice.
VisiPics is the best if you have a medium-small amount of photos IMO. It's easy to use and works well. However, it seems to choke past a few tens of thousands of pictures.
That's why I've switched to AntiDupl. It's got a slightly more obtuse GUI, and takes a bit to figure out how to setup, but it's much better once you get it setup. The hotkeys are great, and it'll detect bit-identical, rotated, lower resolution, even cropped depending on your settings. It'll even compare exif, dates, blockiness, blurriness, preferred directories, preferred filetypes, and a dozen other things to make intelligent recommendations on which picture should be deleted.
It's interface and workflow takes a little getting used to though, so for smaller batches I recommend Visipics. It's easier to use, but seems to choke past a few tens of thousands of pictures.
Do note that neither of these will help you with sorting. Only finding duplicated, nearly-identical, or other undesirable photos.
You can do this. You need to download every Steam background in a folder (Search it). And put your artwork within same folder and run app like Visipics on lower settings so that it'll match your artwork to a similar background.
I've recently went through this for photos so I just have advice in this realm.
First you want to sort something like pics>[year]>[year-month-day]. I have adobe lightroom and importing photos into this will do that sorting for you by looking at the 'taken date' metadata. I don't have any freeware recommendations to do this but they do have a 30 day trial. If you look around there's probably free software to pull this off.
Next is dupe cleaning. I haven't found a good app to do all photos at once so my stack is:
Dupeguru for very high level obvious photo copy deletion.
AntiTwin to fine tune this high level deletion
Visipics to look at the actual photos to find visually similar photos.
Bonus: If you store this photo directory in a google drive folder you'll get to use google photos for your whole collection which is great photo browsing and cataloging.
I've had some messes, maybe not as far reaching as yours. There is little to do than decide on the way you want to structure 20 years of data (by type, or date, etc) and dig in. I will tell you I've used Visipics with great success for rooting out duplicate photos and jpg.
A little at a time and progress is its own momentum.
I like VisiPics, which is a free Windows utility that doesn't do it automatically but allows you to do what you've described. It uses an adjustable scale for anything from a loose resemblance to a very close resemblance. http://www.visipics.info/index.php?title=Main_Page
~~I have used 2 different programs to find these automatically. Both work well. You can set the apps to automatically delete the lower quality file or you can review each one and choose what to do with them.~~
~~or~~
Oh, you want something that will dump a text file, xml or similar. My bad, I don't do the reading so good apparently.
Visipics and Awesome duplicate photo finder
I use both of these because sometimes one program will pick up a duplicate that the other one wouldn't. I also know Picasa has a built in duplicate finder feature.
I'm by no means a pro, but Lightroom is just... very very very awesome.
Give the 30-day version a try, and just import photos in their original location, that way you can uninstall if you don't wanna fork out the $$ at the end of the 30 days.
I'll bet that you'll love it, though.
Oh, and for dupes try Visipics. Successfully deduped 10k wallpapers gathered over 15 years in about an hour.
AntiTwin. I have used it on a Windows 8.1 computer. It has an option to compare images. There is also VisiPics but it is slightly more complicated but it can compare and find different quality/sizes of same images.
edit: I just noticed you are on Windows 10. I have not tried either of these yet on Windows 10.
Certainly, and I'll add it to the original post - I thought I had it there originally, but that'll teach me to reddit when basically asleep!
Duplicate File Finder was by far the best - it had the best interface, easiest to work with, and it showed both files right beside one another. But that one, as well as Visipics, Auslogics were, simply put, missing dupes left and right - almost like it was only checking one specific thing.
What I am specifically looking for is an interface that shows you the duplicates, checks for more than just one specific match type, and ideally checks the actual images itself, regardless of different dimensions (but I realize I may be asking for the moon at this point.)
Ok, downloaded it over torrent and checked for duplicated using VisiPics. Got exactly 6229 duplicated pictures, about 2,37GB free after deleting them.
Thank you, OP. =)
Also one other tip. If you end up have several songs that were on compilation type albums. Like "Best of 2008" or other such things and that is why your trying to separate the doubles or triples, I suggest always saving the highest bit rate first and well as the longest time. That way the best quality is saved first and the rest get scrapped.
I had to do any doubles like this manually, but there are tricks you can use that will make it easier on you. One is doing a Windows search with the name of the song, since the name is the same for them all, a simple search would suffice to do a delete on them.
Another tip would be to do a ".MP3" extension search (Assuming .mp3 is your used file extension) then sort the list by NAME. This will pop all the doubles into a list, side by side, and then your can adjust your tags for time/duration/bitrate etc. And do massive ctrl+left mouse click selects to edit those out.
This same type of File sorting can be used on any type of file, including Movies and Pictures. But I use VisiPics app for pics, since it also does a quality scan. Unrelated to music, link for Picture sorting only http://www.visipics.info/index.php?title=Main_Page
Whoops, yes, you're right. I was wondering how it could have gotten all the way up there, but then when I realized the character's name is Kuroko it all made sense. Lumped her in with the rest of the Kurokos by accident. Fixed, thanks!
Glad you like it! As for duplicate pictures, I see Visipics often lauded as a good, free tool that can help you pin down duplicate pics on your computer. Haven't tried it personally though.
Okay, so I used one that was really good (after searching through quite a few) called VisiPics, download link is on the left. Now, one of the first ones I found was Awesome Photo Finder. It wasn't as good, but it did help fill in gaps that VisiPics missed. I believe both are adjustable, so you can adjust how strict it will be when comparing photos. Good luck!
I just had to do this: Visipics. Open source. Free. It works.
Compares for a number of different (selectable) paramteters. It will not delete the dupes automatically. You must pull the trigger. Other than that, it does what I need.
Edit: Woops. Just re-read your explanation. This works on folders. I'm not sure if it'll work on web-based stuff.
>Are you curious what I'm using this bot for?
Nope, it took about 3 seconds of reading your profile to figure that out.
>Have you made any other bots?
Only a few.
>My other request
Scheduled posting is something I've thought about doing for myself, but I don't know how to make the interface user-friendly. It'll be an SQL database just like the rest of my bots, which means it's either commandline-powered or I'll have to tackle TKinter GUIs again. SQL's can't just be edited with a notepad.
TLDR: I'd like to, but I need to practice that first
The picture comparison would be possible only if visipics supports piping stuff to it through the commandline, which I don't think it does (Look at Weak Points > Tools).
I don't know of any other tools for image matching. If you want to get hacky, we could grab a chunk of data out of the middle of two pictures and see if they match. It's probably an extraordinarily shit way of doing it, and I don't know how that would play with image compression / re-compression. If you have any other ideas, I'm open to hearing. You might do better by just VisiPics'ing them yourself every few days
Now I can't stop thinking about the scheduled poster bot. I think I'll have to give that one a try, and I'll let you know what happens.
Why a visual hash? Why not just MD5 or pHash? A few people are throwing this idea around and suggesting features for a possible local 'search by image' application that would also integrate features of programs such as visipics. Also see comments in this thread regarding duplicate identification and removal.
removing doubles you say? this may come in handy [](/a00 "i'm not just trying to shorten the amount of time till I get more clop to look through. I actually am trying to make your life easier")
Visipics is about the best I've used. It lets you select the level of similarity, then delete by size, resolution or folder. You can choose which folders to delete from first, which to leave intact, delete to recycle bin or another folder just in case (my undoing). Also you see what you are getting rid of. The GUI is a bit old fashioned but it's really fast and does the job.