Safely Storing Digital Multi-Channel Files

With a lot of people having converted their analog quad collection to digital files and playing those files back via hard drive it makes me wonder what a safe way to store audio files really is. I work in the television industry on the creative, not the technical side. I was in a meeting today where we were discussing television shows that had been edited on Avid and stored on digital hard drives. The series we were discussing was on for 6 seasons and the oldest hard drives would be about five years old. I'm hearing from the tech types at my company, "Oh, we lost all those early files because the hard drives are bad. They are dead. Can't retrieve any of the material from them." So it made me wonder, how many people have jettisoned their classic quadraphonic tapes and vinyl after converting them to digital and thinking they were safely storing them on a hard drive. If these hard drives really have a relatively short shelf life of 4, 5 or 6 years, that sounds like a really bad plan.
Is there some other magical way of saving large digital files that can be safely retrieved? Or are we all better off hanging onto our original quad recordings that we've been successfully able to play for 40 plus years now?

If you've got the money, you want to build a hard drive array, or RAID. In simple terms, what this does is build one large storage area out of all the drives, and for every 5 drives in the RAID, one of them is allocated to what's called parity data. This means that if any one of the hard drives fail, you can take out the defective drive, replace it with a blank one and the system can rebuild the data on the missing drive using the parity data and some complicated math.

I worked for one of the bigger video post production companies in London, managing about 550TB of storage so I've had some experience with this stuff - portable hard drives are going to fail, it's only a matter of when and not if. The circuitry and power supplies in those things are anemic to say the least - all it takes is a small blip in the power going to a drive and the circuitry is fried. As soon as you get in to hard drive recovery you're looking at tons of money as the places that do that kind of work charge by the gigabyte and the prices are in dollars rather than cents. The servers at work rain RAID 6 (where the parity data is randomly spread across 20% of the drives rather than having dedicated parity drives) and we did weekly backups to LTO4 tape for disaster recovery.

For my own home use I built a RAID 5 system (you can look up RAID levels on Wikipedia) using a HP N40L microserver and 5 2TB drives and ran UNRAID as my OS, which gave me about 8TB of usable space. There's a bit of a learning curve with that stuff but it's incredibly cheap - I think the Microserver was something like £100 and the drives were just standard 2TB drives, nothing special, and UNRAID is something like 50 bucks US. If you're not quite that tech savvy, companies like Synology and QNAP make ready to go out of the box NAS (network attached storage) systems, although they are quite a bit more expensive.

If you want your data to be truly safe you need to follow the "3-2-1" storage theory, which says you need 3 copies of your data on two different types of media, with one copy of your data in an off-site location. That way not only are you protected against your main storage going down, you're also protected against fire, floods, media failure (as anyone who's backed up exclusively to outdated technologies like zip disks probably knows), and other acts of god. This obviously isn't as easy for the home user, but having your data on at least two sets of hard drives is a start - having extra copies of your data on SD cards, in cloud storage, etc. is another way you can go as tape based backups are pretty much prohibitively expensive.


I think the impending 'data crisis' is something that's going to become more and more apparent in the next decade or two, because people are making vast amounts of data, and it's growing every year and no one has any real idea of how to permanently store any of it. There was an article in one of the audio magazines a few years ago about how the major labels had no strategy for dealing with their digital multitrack masters, and the point was made that even if the files are stored correctly, will you be able to recall a ProTools session done in 2014 thirty years from now? What if the mix references all sorts of circa-2014 plugins? I think that we're going to see a digital version of what happened in the 50's and 60's with TV studios erasing their master tapes, only it's happening in a more passive way because files are being lost, corrupted, or forgotten about rather than actively destroyed. If some label wanted to do a 100.1 mix of some digitally recorded artist from 2014 in 2034 will they be able to find the multitrack files?

dr8track said:

With a lot of people having converted their analog quad collection to digital files and playing those files back via hard drive it makes me wonder what a safe way to store audio files really is. I work in the television industry on the creative, not the technical side. I was in a meeting today where we were discussing television shows that had been edited on Avid and stored on digital hard drives. The series we were discussing was on for 6 seasons and the oldest hard drives would be about five years old. I'm hearing from the tech types at my company, "Oh, we lost all those early files because the hard drives are bad. They are dead. Can't retrieve any of the material from them." So it made me wonder, how many people have jettisoned their classic quadraphonic tapes and vinyl after converting them to digital and thinking they were safely storing them on a hard drive. If these hard drives really have a relatively short shelf life of 4, 5 or 6 years, that sounds like a really bad plan.
Is there some other magical way of saving large digital files that can be safely retrieved? Or are we all better off hanging onto our original quad recordings that we've been successfully able to play for 40 plus years now?

Click to expand...

Yes and yes! - I have all of my content stored on 2, 3TB external hard drives - one is a complete duplicate back-up, having already suffered the slings and arrows of a failed hard drive. I also maintain a library of my original discs down in the dungeon! I suppose another safety valve would be the "cloud" - but even if I were to employ that option, I would still maintain the hard disks.

elmer said:

Yes and yes! - I have all of my content stored on 2, 3TB external hard drives - one is a complete duplicate back-up, having already suffered the slings and arrows of a failed hard drive. I also maintain a library of my original discs down in the dungeon! I suppose another safety valve would be the "cloud" - but even if I were to employ that option, I would still maintain the hard disks.

Click to expand...

A cloud-based back-up service like Carbonite would be worth exploring. Next on my list to check.
Web Site: http://www.carbonite.com/
2 Months Free Offer: carbonite.com, offer code = WINDOWS

bmoura said:

A cloud-based back-up service like Carbonite would be worth exploring. Next on my list to check.
Web Site: http://www.carbonite.com/
2 Months Free Offer: carbonite.com, offer code = WINDOWS

Click to expand...

Thanks. I have sent them an inquiry to see if their home plan can accommodate a really big collection. They say "unlimited" but speak only in terms of GB, not TB.

Here's the response to my query about the multiple TB of music on my NAS drive:
"Carbonite does give you an unlimited backup on a Personal subscription. You would not however be able to back up a NAS drive on a Personal plan. We would backup any internal drives and with our Personal Plus or Personal Prime accounts, you would be able to backup an external drive connected directly to the computer. With the size of the backup that you are talking about, it could take an extremely long time to complete a backup. Carbonite does backup about 3-4 GB of data per day. It does appear that Carbonite may not be the backup option that you are looking for. You are more than welcome to start a trial to see if the service is something that you are interested in. "

steelydave said:

For my own home use I built a RAID 5 system (you can look up RAID levels on Wikipedia) using a HP N40L microserver and 5 2TB drives and ran UNRAID as my OS, which gave me about 8TB of usable space. There's a bit of a learning curve with that stuff but it's incredibly cheap - I think the Microserver was something like £100 and the drives were just standard 2TB drives, nothing special, and UNRAID is something like 50 bucks US. If you're not quite that tech savvy, companies like Synology and QNAP make ready to go out of the box NAS (network attached storage) systems, although they are quite a bit more expensive.

Click to expand...

RAID on NAS + 2 external copy is my way to go. Will like to know more about the Microserver solution you're using, both on HW and SW side; i have a Synology 411j 4-bay* with 4x3Tb drives, the synology software is very user-friendly and can be used and configured very easily, even if it's not so fully adaptable for specific needs (doesn't serve flac+cue); as for final price, isn't that much more than the Hp+Software, so for a non-geek user it can be a valid solution.


*one of the best birthday present i ever had...

Kal Rubinson said:

Here's the response to my query about the multiple TB of music on my NAS drive:
"Carbonite does give you an unlimited backup on a Personal subscription. You would not however be able to back up a NAS drive on a Personal plan. We would backup any internal drives and with our Personal Plus or Personal Prime accounts, you would be able to backup an external drive connected directly to the computer. With the size of the backup that you are talking about, it could take an extremely long time to complete a backup. Carbonite does backup about 3-4 GB of data per day. It does appear that Carbonite may not be the backup option that you are looking for. You are more than welcome to start a trial to see if the service is something that you are interested in. "

Click to expand...

Check out CrashPlan. The price is right, though:

1. Backing up an NAS requires some kind of hack under Windows but no special trick under Linux.

2. I find the upload speed very slow, though the fact that I'm running it in a VM may be relevant.

3. You MUST leave anything you want backed up connected. If you disconnect a drive, CrashPlan assumes you no longer care about that data and will destroy their backup.

I've got something like 5TB backed up with them now and they've never complained, though it took MONTHS to get it uploaded.

Kal Rubinson said:

Thanks. I have sent them an inquiry to see if their home plan can accommodate a really big collection. They say "unlimited" but speak only in terms of GB, not TB.

Here's the response to my query about the multiple TB of music on my NAS drive:
"Carbonite does give you an unlimited backup on a Personal subscription. You would not however be able to back up a NAS drive on a Personal plan. We would backup any internal drives and with our Personal Plus or Personal Prime accounts, you would be able to backup an external drive connected directly to the computer. With the size of the backup that you are talking about, it could take an extremely long time to complete a backup. Carbonite does backup about 3-4 GB of data per day. It does appear that Carbonite may not be the backup option that you are looking for. You are more than welcome to start a trial to see if the service is something that you are interested in. "

Click to expand...

The response I received from Carbonite was that for a NAS, you would need to buy one of their Business grade plans. The Personal plans are for the PC and connected drives - like a USB connected external hard drive.
With the offers for a couple months free, it may be worth a try.

Well all I have to say is thank God the digital age has made things so much easier.

Really, a digital media collection is no different from any other digital file collection, except it tends to be bigger and more cumbersome. Treat it like data that you don't want to lose - and learn from the IT community about how to protect it.

I keep my 4.5TB media collection on 6TB (8TB striped RAID 5) external drive array to protect from hardware (drive) failures. This array is connected to my media server. I also have a NAS to which I synchronize the array contents nightly. This protects me from RAID controller failures (yes, RAID controllers ARE a single point of failure) and human errors (RAID does nothing for you if you delete a file you did not want to delete).

I also routinely (generally monthly) back up the contents of that array (using drive sync software rather than backup software), as well as personal documents and such, to a pair of 3TB Western Digital MyBook USB3 drives, which I then store off-site. These give me an easy way to recover from catastrophic site failure (house fire, for example).

Really, a digital media collection is no different from any other digital file collection, except it tends to be bigger and more cumbersome. Treat it like data that you don't want to lose (which is what it is) - and learn from the IT community about how to protect it.

I keep my 4.5TB media collection on 6TB (8TB striped RAID 5) external drive array to protect from hardware (drive) failures. This array is connected to my media server. I also have a NAS to which I synchronize the array contents nightly. This protects me from RAID controller failures (yes, RAID controllers ARE a single point of failure) and human errors (RAID does nothing for you if you delete a file you did not want to delete).

I also routinely (generally monthly) back up the contents of that array (using drive sync software rather than backup software), as well as personal documents and such, to a pair of 3TB Western Digital MyBook USB3 drives, which I then store off-site. These give me an easy way to recover from catastrophic site failure (house fire, for example).

Also note the cloud-based backup services are great for documents and smaller media collections - and I often recommend them for photo and document collections. But large collections of large files become rather cumbersome unless you have a high UPSTREAM data rate from your home all the way to the cloud vendor who provides the service. Most folks forget that a 50MB connection is about the DOWNSTREAM speed, and the upstream speed on such a connection is generally 10-20% of the downstream speed. Rip a couple of blu-rays, which results in 50-70GB of data, and then back those up to cloud backup over a typical upstream link for even a high-end home internet plan. From a practical standpoint, if backups start to take too long, you will stop doing them...and then you will lose the protection associated with them.

A lot of the business model of these subscription services is to get people to sign up for an auto-renewing subscription when they know that the vast majority of subscribers will seriously underutilize the service.

Laird_Williams said:

From a practical standpoint, if backups start to take too long, you will stop doing them

Click to expand...

I assume CrashPlan is not the only backup service that operates through a proprietary Windows/Mac/Linux client that simply runs any time the computer is on. In those cases, one doesn't really "do backups" in a conscious, time-consuming manner. The backing up is just another task the computer does without the user really thinking about it.

Your point about the upload speed rarely being as good as the download speed is a good one. I'm fortunate in that my situation is usually the other way around: My download speed is slightly lower than claimed while my upload speed is significantly higher than the rating. However, that doesn't help when the service to which I'm connecting can't or won't accept the data as quickly as I can send it.

But any kind of offsite backup that I don't have to explicitly think about is, at least for me, preferable to none.

My situation comes from putting off getting my backup system running after my NAS failed. I had all my music and movies mirrored between the PC and the NAS. Instead of getting a new NAS and rebuilding my backup, I dropped a 2TB drive in the PC and backed up the most important files on a 500GB USB drive. I planned on putting together a better NAS setup later on and the new drive in the PC would buy me time for that. That was a little over a year ago and I still hadn't got around to the new NAS.

Last weekend, that year old hard drive failed with no warning at all. It just died. Hundreds of movies and thousands of audio files gone. It will not be a total loss, but I haven't backed up to the USB drive for a few months so there will be some not on the backup drive.

I've learned my finally learned my lesson and in the process of setting up a 3TB RAID 1 NAS until I can get the next setup together.

I have decided that the cloud-based services are not suitable for me. I have about 5Tb of music files on a 12Tb RAID/NAS and it is duplicated by another identical one in my other home. These are backed up and manually synchronized by a few 4Tb external drives that go back and forth.

I'm using Google Drive storage in the cloud for backup. Pricey but effective.

I looked at Microsoft's free cloud storage offer with Office 365. But their restrictions on file size make it unusable for multichannel download files.

This might not be the time or place to mention such things, but apparently--and according to a report on CBS' 60 MINUTES--our stockpile of ready-and-capable nuclear missiles is still maintained under an ancient system that used those larger floppy discs! That's going way back, and more than a little worrisome that, with all the ways government can waste or misuse funds, they haven't bothered to upgrade even to this day (even the phones are old with fraying cords. Sheesh!)

For myself, I use an old (2001) Dell Dimension L for my archiving of data and photos, but everything is put on external drives, whether on the DL or our laptop. With the way an OS or any computer can just suddenly croak, and with the real pain of retrieval (which is possible but can be costly) I find that the better USB drives work for me--at least with the handful of MC files I want to keep that way. And that's because I know from experience that CD and DVD-R's can eventually started glitching and going bad. Unlike vinyl or well stored tape sources, digital seems to be erratic in longevity.

Worse, digital media continues to be modified and ever-changing in so many ways. Couple that with the compromises in software (let alone hardware) that happen with any mass-produced product, small wonder no one really knows how long any digital media will last without developing possibly irreversible problems. Analog media may be ancient, but often tapes can be 'baked' and copied or transferred yet again, and God knows whatever drawbacks vinyl may have, I can still find really clean stuff that is many decades old, yet plays very well even when it may not look so good. The irony is that vinyl is an inherently fragile medium, yet you can come across some things that will take your breath away, not least because digital versions of the same music don't match it (though to be fair, some surpasses the original vinyl, too).

Truth is, though, for something you really, REALLY LOVE, there is nothing like a second copy, well stored (and where you can recall where you put it!) to give some peace of mind. The nice thing about being a vinyl hound is that most of what you own can, even now, be easily found again. It's the genuinely hard to find and obscure stuff (like what we all dig here) that needs backup, because it comes and goes and may be hard to find again without paying a price you'd rather not.

ED

Kal Rubinson said:

I have decided that the cloud-based services are not suitable for me. I have about 5Tb of music files on a 12Tb RAID/NAS and it is duplicated by another identical one in my other home. These are backed up and manually synchronized by a few 4Tb external drives that go back and forth.

Click to expand...

Um Kal, sounds like you have your own cloud service.

milt said:

Um Kal, sounds like you have your own cloud service.

Click to expand...

Yup. I consider it purpose-designed for music while commercial cloud servers are designed for data.

Kal Rubinson said:

I have decided that the cloud-based services are not suitable for me. I have about 5Tb of music files on a 12Tb RAID/NAS and it is duplicated by another identical one in my other home. These are backed up and manually synchronized by a few 4Tb external drives that go back and forth.

Click to expand...

I ended up making the same decision, but I don't quite have it all set up yet. I've got my first NAS configured as RAID 6 (I went with a 10 drive QNAP NAS) and I'm saving up for my second; which I'll keep at my dad's place. I currently have the NAS redundantly backed up to external USB drives.

I now have a 6 drive NAS but no RAID configured. Just 6 individual drive (4 x 4Tb and 2 x 3Tb). RAID arrays are known to fail. RAID won't protect against fire or theft.

I backup onto 6 backup drives (1:1 with each NAS drive) by copying changed files only. These drives are stored in my detached garage. RAID won't protect against fire or theft.

For my multichannel audio files I actually have a 2nd backup drive (4Tb) in my desktop too. (This is my 'master' disc used to push files to my NAS).

I only ever play the hi-res FLAC files on my NAS. Not only do I have 2 backups of the FLACs - I also have 2 copies of the original ripped ISOs distributed across other NAS drives and backed up!