So, you’ve probably heard this before, but your duplicates are showing… Well, mine are at least. Which reminds me of one of the signs I spotted on our Cross Country Travels.

Managing and backing up large photo files is not particularly easy. But I recently whacked my head on a problem that perplexed me.

To set some background, it may help to know that I use DropBox to BACKUP, SHARE, and DISTRIBUTE my image files both to clients AND among my various computers. I have the 2 Terrabyte Plan at $99 a year.

Let me briefly explain how and why I do it, so what I communicate next won’t look like my crazy is showing.

Image Organization

My file organizational structure looks something like this on all my machines. Most of my work is in Windows, by the way, but I do use a Mac sometimes. I keep the same structure across multiple machines for several reasons including consistency and the ability to work the same on each machine. My folder/file structure looks like this:

• Images
  • 2011
  • 2012 …
  • 2019
    • 2019-05
      • 2019-05-29
        • RAW
        • JPEG
        • Export
        • Processed
        • Video
    • 2019-06
      • 2019-06-03
        • (same as 2019-05-29)
    • Android
      • Phone
      • Card
    • IPhone
  • Work In Progress
• Audio (e.g. Itunes and purchased music)
• Business Records
• Business Public
• LightRoom Catalog(s) (Good luck, Lightroom doesn’t much like shared drives or volumes, so I’ve had to trick Lightroom into thinking it’s writing to a regular volume)

That’s important because you’ll notice that I do not really have a good organization system for Android and Iphone photos and videos … just a year by year copy. I also break down my images by year, month, and year-month-day. Again, not ideal, but that’s the pattern Lightroom follows and it mostly works for me.

Lightroom does not know how to separate JPEG and RAW images into separate folders. I usually shoot BOTH image types. The fast load time of the JPEGs allows me to breeze through them first, and I sometimes use the small JPEGS (which are still plenty big) for web purposes. Usually once I’m done with my photo processing sessions of the RAW images, I no longer have a need for the JPEGs – or even the RAWs. But of course I do back them up. All of them in fact except the absolute stinkers.

My Exported images are Processed images that have been downsized and watermarked ready for publication on e.g. Flickr or Facebook, etc. Processed images are those where I’ve done more than just Camera Raw adjustments or Lightroom adjustments. That is, composites or extensive masking and adjusting that I have done in Photoshop. By the way, I always create side-car files (the .xmp files) that keep track of the changes you make with Lightroom or Adobe Camera Raw. I keep the Export(ed) and Processed images locally until I need space at which point I remove them from my LOCAL computer and just leave them in DropBox should I need them again.

Since Lightroom does not import things the way I want, you might be interested to know that I do the importing with a primitive Windows Batch file. I copy all the content into a pre-created directory and then run this in that directory to move the files to the respective sub-folders.

 @ECHO OFF 
 REM Can optionally specify the source folder name e.g. .\OtherDirectory
 REM cd %USERPROFILE%\Pictures\ImportFolder
 REM mkdir NEWIMPORT
 REM cd NEWIMPORT
 REM COPY all files from the card to this folder

 mkdir RAW
 mkdir Processed
 mkdir JPEG
 mkdir Video
 mkdir Export
 @echo ON

 move %1.J* JPEG
 move %1.NEF RAW 
 move %1.CR2 RAW
 @REM This should copy MOV and MP4 ... but might copy something you don't expect
 move %1.M* VIDEO
 move %1.

DropBox “Smart Sync” is nifty: and so is “Selective Sync”

One relatively new feature of DropBox is the SmartSync option which behaves a bit like Lightroom – you can have a file that is indexed, but not local. There are three ways to use Dropbox for managing backups:

A. Use selective sync to synchronize specific directories (e.g. 2019-05-29) on a computer by computer basis. Be careful, if you do it wrong, you can wipe out a whole directory at once including your backup! That is, do NOT delete a local copy of a DropBox folder until after you turn OFF selective sync. Indeed, until Smart Sync, if you turned off selective sync, DropBox will delete the whole folder LOCALLY.

B. Use the new Smart Sync feature to distinguish Local and Online copies. Local copies the same as the synced version that is in DropBox. While “Online” means DropBox preserves the file structure and names on your local computer, but doesn’t download the content unless you try to open the file. This behavior doesn’t play with with all programs, mind you.

C. Use both – that is what I do. On my under-sized laptop drive, I selectively synchronize a small handful of things (Business Records, Work In Progress and perhaps a few days worth of photos). While on my workhorse desktop machine, I keep many more photos local on a huge hard drive.

If you’re wondering why I don’t just pay Adobe for cloud storage… never mind, you’ve probably already figured out why I don’t want to pay twice, and MORE for what I already am using.

iPhone and Android folder structures are not consistent – Why?

The organizational structure on both the iPhone and Android are a bit messy, mysterious, and … change over time, so I have not had much success trying to duplicate the structure I use for my DSLR real photography in the cell phone space. And yet, like many of you, my cell phones are doing more and more of the lifting for illustrative things. One way my Android gets twisted up is that it has both phone memory and card memory. The problem is that there are lots of places where pictures occur, on say the Android. For example:

Galaxy S9+\Phone\DCIM\Camera
Galaxy S9+\Phone\Pictures (which has many subfolders)
Galaxy S9+\Card\DCIM\Camera
Galaxy S9+\Phone\WDisk\Photos
and more!

Trying to extricate files from those many locations sometimes means I goof and recreate the same structure multiple times. To make matters more complicated, I also record images and videos with a Mavic drone.

Another Organizational Failure

I may copy May 29 files into May 28th for my big camera images. But I never intentionally duplicate images even if it sometimes happens by accident.

Sometimes I get ahead of myself. I simultaneously copy e.g. Android stuff on one machine while copying iPhone stuff on another. When I do that, Dropbox can get confused. And that’s on top of the pain that Lightroom imparts… so I find myself using less and less of Lightroom for importing photos (too slow and cumbersome), and instead I get the structure in place and then tell Lightroom where to look. Indeed, I admit I now prefer to use Adobe Bridge, or some much simpler programs like Image Glass for quickly viewing photos. Image Glass is very quick to load and use, something like Windows Photos, and Image Photo Viewer, but more nimble (and less powerful).

After the cross country trip, I found myself with apparently MANY copies of the same cellphone and camera images… but with thousands of images it is not easy to identify which are really duplicates. Some duplicates are the result of doing in-phone photo editing. Obviously those I want to keep. However the accidental duplicate copies I do NOT want to keep.

And the duplicate situation differs by cell phone, too. Android often tags -001 to files you edit on phone, while iOS creates an .aae file.

Deduplication To the Rescue

Is there software that will do a good job cleaning up duplicates on a Windows machine? Yes and No. Yes there is, but WHAT IS A DUPLICATE? I mean I know when what I look at is a duplicate, but that doesn’t mean software is particularly good at it. I was grievously frustrated when trying to trial some of the photo/file de-duplication tools that exist. I thought, CCleaner did a pretty good job, until I realized IT DOESN’T understand images very well.

I then went on to try Easy Duplicate Finder, Duplicate Photo Cleaner, and a few others. Unfortunately my tests were against a real directory with thousands of files and these “tools” would only let you do 10 removals unless you paid the $30, $40 or $50 to activate them. Well that’s NO TEST! And more, none of them bother to tell you up front that the many minutes you spend reviewing its scan is a waste of your time – or what the charge will be to license the software. One of the tools begged to not be uninstalled and offered a significant pricing break, which brings me to a tip:

When trialing software, it’s always a good idea to try to uninstall it before you even think about purchasing it. Crafty programs (and marketers) realize that offering a discount is their only defense against losing a sale to you. The discount may be substantial!

Duplicate Photo Cleaner Irked Me

Not only did DPC NOT tell you what the charge was, it asked you to register (you give it your email address) before it ever mentions the price ($39.90). It does seem to have some smarts to pick photos by appearance but it lumped together three different screenshots from the phone as duplicates that were NOTHING alike. Suspiciously, Duplicate Photo Cleaner and Easy Duplicate Finder are both Webmind products and use the same shameful sales tactics.

I suppose there may be instances where you want it to figure out if one image is merely a smaller version (or only “slightly different” from another), but the inspection, took a long time and decided that not only were the screenshots identical, but two quite different sunset shots were marked as duplicates. If your goal is to toss out all but the first image of your daughter, the default behavior of this tool might be what you want. You *can* adjust the settings, but the slimy marketing and the out of the box behavior had me saying No Thanks in a jiffy.

Duplicates Cleaner by kaeros < Works!

The good news is that those badly behaved trials made me try Duplicates Cleaner from the Windows Store. I normally avoid the Windows Store because it has a lot of rubbish, and it’s Microsoft. Duplicates Cleaner is FREE. And guess what, it worked very well – got rid of 4.5 Gb of redundancy in my Android directory alone. Duplicates Cleaner has a $6 Pro version which I may well upgrade to. But the free version does the trick and includes a small ad for one of the developers products. Duplicates Cleaner doesn’t try to inspect your image to see if they are visually similar – it looks at the content (or name) to make its decision. Content is a good way to go, two files with different names and identical content ARE duplicates. You can see it also found redundancies in the organization of my “big camera” directories. (Usually because I’m not always careful about where I tell Photoshop to stick things).