The Importance of Protecting your data - My Data Management Plan

I have spoken a lot recently about NAS & RAID and allsorts of other technical stuff to do with data storage, but what does it all mean and what is the point? Hopefully, this blog post will help explain why it is important to protect your data and a look at the best ways to do it.

Imagine losing everything…

For many of us Photographers & Videographers, our data is our most important asset, arguably even more important than our camera gear. I still sell prints of images I took four or five years ago so the idea of losing my files fills me with a crippling sense of anxiety. However, it’s not just as simple as losing a few photos or a Lightroom catalog. I have invoices, agreements, personal projects and many other important documents that, if they were to simply disappear, would grind my business to a halt. Yet the vast majority of us only ever decided to protect our data ‘after’ we have had a major catastrophe, so now is the time to get proactive!

What are the options?

Before we delve too deep into some of the methods available, it’s worth taking a step back and looking at the bigger picture. Most of us either work from home or an office but in almost all circumstances, we leave our precious data in one place. I, for instance, work from home and choose not too rent an additional office space as I have everything I need in a dedicated ‘office’ (spare room) at home. However, here is the first issue. All of my data is in one place. If I was to be burgled, flooded or if there were a fire, my data (hard drives) would literally go up in smoke, removing any possibility of recovering them.

With this in mind, it is important that we remember that many of the options that we will look at are not failsafes, they are not legitimate ‘back ups’ of your data. Only when the data is being stored in multiple safe locations (off site away from your home/office) and backed up on a regular basis can it be considered safe. To be honest, even then there are still risks involved.

Cloud Storage

If ‘offsite’ backup is a must, then it would be easy to think that cloud storage is a bit of a no-brainer. Though it certainly has an element safety, knowing that your data is ‘in the cloud’ and not vulnerable to such earthly things as fire and theft, the truth is, it’s not 100% safe. In a world of cyber crime, even large organisations such as the NHS have surcome to data being held at ransom.

The other downside to cloud storage is that most commonly, space is at a premium with even the more generous service only offering around 1tb of storage and there can be restrictions on the file types that can be stored. The other major issue is that depending on your internet connection, it can be painfully slow saving and accessing your data and if, like me, you have multiple terabytes of data, you’ve pretty much had it.

NAS & RAID

He we go with the technical stuff again. I will attempt to describe these in as basic terms of possible. One of the main threats that we have to our data is the chance of a mechanical error and losing a hard drive to a technical fault. Although there are services out there that can attempt to recover lost data, there is no guarantee. So a clever option is Network Attached Storage or NAS for short. NAS is a centralised storage device that can allow multiple users to access data view a network (or LAN). NAS’s are commonly made up of one or more hard drives which are referred to as a ‘Redundant Array of Independent Disks’ or RAID for short.

The purpose of RAID is to protect against loss of data by spreading or mirroring that data across multiple disks. We will look at a few common RAID options in a future post but see it as a carbon copier for your data. If you lose one disk, the other disks will have your precious data saved, ready to be restored to a new disk. However, the major downside of this is that although it protects against mechanical loss of data, it is still only stored in one place - so we still have the risk of fire/flood/theft etc.

The Blended Option.

I’m hoping I may have coined the phrase but a great solution for protecting against loss of data is to ‘blend’ all of the options available to you. So in essence, you would have a NAS setup in your home or office, where multiple users could access and store data. The NAS would have an appropriate RAID setup to protect against mechanical loss. The most important data would then be copied to a cloud service that automatically updates on a regular basis, to add in an additional layer of protection.

Taking things one step further, for those that are looking for the ultimate in protection, it would be possible to have another RAIDed NAS system kept off site (maybe at a friends home or office) that ‘backs up’ data so if there were a fire in one location - all data should be saved somewhere else. Now, this would be an extremely costly measure and a simpler option may be to simply rotate a group of hard drives and once a month - mirror the backup disks with your working disks and then store them off site. It’s worth noting at this point though that even by going to these extremes, there are still other potential risks such as file corruption, so before investing it really is advisable to do as much research as possible, to find the best option for you.

My Data Management Workflow

I’m not an expert, I just know what works for me, but you may find something useful in the way I manage data. Seeing as the majority of my ‘data’ are image or video files, the first thing I do is make sure I have plenty of fast CF/SD cards and if I’m shooting really important work, I will write to two cards at once, you know, just in case.

When I get the data home, I make sure that I import the RAW files to Lightroom immediately. Before I go out on a new shoot, I always format the memory card in the camera, by importing the images immediately to Lightroom, I ensure that I won’t ‘accidentally’ delete all the files from a previous shoot. I then save the images into a ‘working’ folder structure on either my MacBook or PC. By doing this, I am using the internal hard drive of the computer (both MacBook and PC have SSD drives for improved performance) for editing, as trying to edit direct from the NAS is possible, but not as efficient as using the internal SSD.

Once the edits are complete, I export the completed images to a DropBox folder as medium res Jpeg files and for images I know that I will print, I export the Tiff/PSD files to a ‘print’ folder. At this point, I have a copy of the file on my working hard drive and an output file in DropBox. The next step is moving the working folder in Lightroom from my internal SSD to the NAS. Lightroom does an excellent job of folder management, meaning I don’t need to go into Finder/My Computer and start copy & pasting folders to move them. I have the NAS set up as an ‘Archive’ folder in Lightroom and can simply move the project I am working on from the SSD to the Archive by dragging & dropping.

Once the data is on the NAS, I am using a RAID 1 setup which mirrors the data from one disk to another meaning that if a disk fails - I can still recover from the working disk. The final step (and the most crucial in terms of protecting against fire/theft etc) is taking a regular copy of the most important projects and leaving them on an encrypted external Hard drive, with a friend, so they are stored off site. Now this final step is not the best method as it does not protect all data and there are other inherent risks (losing the disk in transport, finding someone you trust enough etc!) but it is better than nothing!

All of this may seem a little bit excessive but when faced with the prospect of losing years worth of valuable data, it suddenly becomes quite appealing. The truth is, once it’s set up, it’s actually remarkably easy to manage and the biggest difficult is forming the new habits to make sure you are treating your data with the respect it deserves. I would certainly suggest looking at a NAS setup at a minimum but also consider how you can store your data in multiple locations. I’m pretty certain that this won’t be the end of my data management journey but for now, I feel a lot happier knowing that my data is in safe hands.