How to move your data around among disks. You could move images from your primary storage to your backup storage simply by dragging changed files over. I don’t recommend this; you want to use a method that keeps your backup data current automatically. You don’t want to have to remember which files you changed, and you probably don’t want to wait the time required to replace all files on the backup device every time you edit a few images. Most of all, you don’t want to have to think about backup during your normal working day; it should just happen.
There is a kind of backup program that clones entire disk partitions. Symantec makes one called Ghost. Acronis has one called True Image, and the one I currently favor is called ShadowProtect. These programs are really good at getting you back up quickly if the disk that holds your operating system fails. I recommend that you use one of these programs, but not for your images. Data is usually stored in a proprietary format, and it’s sometimes difficult to restore to different hardware than what was running when the backup was made.
The kind of backup program you want for your images is a file and folder synchronization program; an Internet search on “file sync” will yield a slew of candidates. The kind of synchronization you will be performing is one-way synchronization; that is copying data from your primary image store to the backup image store. You can set up a file synchronization program to copy files from your primary disk whenever a file changes, or after waiting set period of time. I recommend the latter choice, since it allows you to avoid having your machine bog down performing the backup while you’re working on your images. Some sync programs let you set different criteria for different file folders. Most of these programs also offer options that let you save the last few versions of each file, which can be useful if you overwrite a file with an edit that you later regret.
Here’s a sure-fire way to ruin a day. You’ve been backing up for years, but have never done a restore. You have a disk failure. No problem you think, as you find and mount the backup. The backup proves to be corrupted. You find the previous backup. It’s bad too. Turns out all your backups are bad. The moral: whatever backup software you use, you need to make sure that it works, and that it keeps working. This is one of the reasons I’m down on the backup programs that generate a monolithic file with all of your images rolled up into it. The only way to make sure you can get at your backup data is to do a trial restoration. I know lots of people who resist trial restorations, because they’re a pain. You may not have that space available. Doing a trial restoration over your primary data is a move that makes most people a little queasy. Restoring a lot of data also takes a while. You can use virtualization and restore to another virtual machine; if you’re comfortable doing that, you are more skilled than the target audience for this little essay, but remember, if you’ve got a terabyte of data in your backup, you need a terabyte of free disk space to do the restoration.
The file-sync approach avoids all that. Your data is stored right out in the open, and you can open a few files and make sure they are uncorrupted. You can also use the file compare option of most file sync programs to compare all of your backup files to the primary ones. By the way, one of the nice things about most of the NAS systems is that you can have them send you an email if there’s a problem, plus a regular status report. That way you don’t have to remember to go out and check up on your storage; after a few weeks of looking at an email from your NAS box every morning, you’ll notice if it’s missing.
A word on compression. Avoid it, unless it’s part of the image file format. Lossless compression (the kind that doesn’t affect image quality) doesn’t work well on photographic images. It won’t damage the images, but you will find that the losslessly compressed files are only slightly smaller than their uncompressed versions. In addition, compression offers opportunity for new adventures in obsolescence.
How to organize your images into folders. I wish I knew more about this. I would welcome an article on the subject from anyone with a well-reasoned point of view. I do have one recommendation. Keep raw and finished files in completely separate folder trees, since the kind of scripts you will want to run against the two kinds of files will be different.
How to get at the image you want. If you are incredibly organized, you may be able to create a directory tree that will let you find what you’re looking for just by sorting through folders. For the rest of us, some kind of image organizer is essential by the time the number of images in your collection hits quadruple digits. Image organizers let you assign keywords to images and retrieve them by searches. They let you group images together independently of the folders where the files reside. They let you rank and flag images. There are many organizers around, and there are several programs, like Lightroom and Aperture, that include organizers and do much more.
I have a perspective on choosing an organizer. Take the long view. Many organizers work by creating a database in proprietary format with the information that you enter about your images. You will invest a great deal of time in assigning rankings and keywords to images. You will upgrade computers and operating systems many times over the life of your image collection. Try to find an organizer that will be around for years, and will be updated to run on newer operating systems as they are introduced. If an organizer today runs under both Mac and Windows OSs, that’s a good sign. It also inspires confidence if the company selling the organizer has a track record of shipping quality products, supporting them well, providing transition paths when they introduce new products, and making money. If you can find an organizer that does all its work with IPTC tags rather than propritary database entries you’ll have a chance of transporting your organized image collection to another organizer program, but even that isn’t a slam dunk.
First of all, many thanks for this series of articles! Very well thought out.
I organize my images first under one folder – the Pictures folder. That means that backing up the Pictures folder and subfolders backs up everything. Within that I sort by camera, then by location, then by year then by month. I do all of my DAM (digital asset management) in Lightroom, but I also write out the metadata to the files themselves as a backup measure. The Lightroom catalogs, presets, backups, etc. are also stored under the Pictures folder for easy access and for backups.
I realized this isn’t a post on Lightroom, but LR is very popular today. To that end, more info on how Lightroom works with metadata can be found here: http://bit.ly/LRFmM and my Lightroom Workflow can be found here: http://bit.ly/LRWFlow A list of links to over 200 other sites providing Lightroom tutorials, videos, information, etc. can be found here:
http://bit.ly/LRTips
Mike.
Mike,
Thanks for the comments and the LR links. The material appears to be well thought out. I’ll have to some some time with it to see if I’m in complete agreement. One thing that both of you mention that makes me nervous: keeping all your images in the Pictures folder. If you’re on a Wintel machine, the standard mapping of the Pictures folder is to the boot partition. If you keep your pictures there, you’ll need really big disks to backup your OS partition with Ghost or similar cloners, and upgrading to get more space for images will be complicated. I like to keep my images on a different disk than the OS. That way I can put the OS and all the programs on a solid state disk, and the images on a set of striped 3 TB spinning disks. If you like the idea of the Pictures folder, Windows will let you remap all the Documents, including Pictures, anywhere you like. I just have a completely separate directory called Images.
If you’re on a Mac, the above specifics are wrong for you, but the principle applies.
Jim
Great article, and I’m definitely in agreement about using hard disks for backup. I follow a similar pattern for backup – internal HD, 2 swappable external HDs (using Apple’s Time Machine and cloning periodically to a drive stored in a safe), and an external cloud backup service. Regarding the cloud service, a number of the ones I investigated allow you to have them ship you a hard drive with your data for a fee, otherwise you’d be spending a month downloading the data.
Regarding a folder structure, I prefer to just create one folder for each day. When I import images via Lightroom, I set it to automatically copy images onto a second 2TB drive into folders named by YYYY-MM-DD so that the folder sort order is chronological. Once I have had the chance to go through a folder, trim unwanted photos, tag, and rank the rest, then I rename the folder to be in the format “YYYY-MM-DD – Description”. That way I can scan through my folders and quickly see which ones I haven’t yet processed. I would probably revise that though to use the directory format /Photos/YYYY/YYYY-MM-DD/ so that photos for a particular year are grouped together and you don’t get a list of hundreds of folders. I’m not a big fan of grouping photos by other criteria, in my mind it just makes things harder to find, and in Lightroom you can easily create collections based on metadata such as camera type anyways, making the additional folders unnecessary. I find with my system I can find things quite quickly through either searching on keywords, or on just scrolling to the right date and looking for the relevant description. Also a vote of recommendation for Adobe Lightroom, it has vastly improved my ability to actually use the photos I take.