Hard Disk Clicking and a Time Machine Failure

It had been a long couple of months. We’d been working furiously on the launch of EasyGrouper and were just a few days away from having it go live. As I sat in my home office working on my new MacBook Pro I heard a deeply disturbing sound coming from my older Mac Pro tower a few feet away:

   ZZZZ-click 

   ZZZZ-click

   ZZZZ-click

That’s never a good sign. My Mac Pro is my main home machine and runs a lot of stuff for me. It’s got a 320GB main drive (“BootDisk”) running my apps and working documents, a 1TB drive called “BigDisk” that contains my family photos, videos, music, etc. and another 1TB drive called “Backup” for... you guessed it... backups.

I tapped the keyboard to wake up the screens and could see that the icons for my drives were all on my desktop when suddenly the BigDisk icon vanished and OS X gave me an error message that said I had not ejected the drive properly.

Not good.

My Mac could no longer see my BigDisk internal drive—the drive that contained over 380GB of rather important information. Virtually every digital picture I’ve taken since 1999. I didn’t panic though. I’m a big Time Machine advocate and it’s saved me before. I tried restarting the machine but the ZZZZ-click sound returned and it wasn’t even able to see my drive. I got back to work on EasyGrouper, determined to get this resolved the following Monday when I came up for air after the launch.

Fast Forward to Monday Night(mare)

I sat down Monday night to work on restoring my files. I clicked on the Time Machine icon in the menu bar, selected Enter Time Machine and the space traveling interface popped up with a Finder window.

My Time Machine Display


But there was a problem. A big problem.

I clicked on my machine in the list of items on the left pane and it only showed one drive: BootDisk. Oh yeah, Time Machine had been running for the last few days so it had been backing up my machine without BigDisk in there. I started to use the Time Machine interface to go back in time.

I backed up to the day I heard the clicking sound. No BigDisk. I backed up further. Nothing. The only drive that was being backed up was my BootDisk drive. It’s as though Time Machine didn’t even see the drive.

Now I started to panic. I counted on Time Machine to back this up for me and I never saw an error message or any indication in the menu bar that Time Machine was having a problem. I knew that at some point Time Machine had been backing up that drive because I had previously used it to recover a file I inadvertently modified and saved.

I looked into Console and searched for “backupd”; there were plenty of entries for it backing up my BootDisk, but that was it. No Time Machine error messages like I’d seen in the past. I had an external drive in my fireproof safe that contained a backup of my pictures and home movies, but I hadn’t updated it since the end of 2010. I had become lazy and complacent and it was about to bite me. The last three and a half years of my digital life had just vanished.

Recovery Time

I asked around the office and many people recommended I see if a hard drive salvage specialist could restore the data for me. I found Salvage Data in nearby Herndon, VA—they were super nice and would perform a free diagnostic to see if the drive could actually be recovered. I dropped it off and waited to see if they could help.

The next day I got a call. The drive’s primary head cluster had failed and the drive had logical errors that would need to be corrected. It would take 10-15 days if I didn’t need rush service. The price: $1,740. The price seemed high but Salvage Data had an excellent reputation and the people were extremely attentive and understanding. They described the work that would need to be done in a clean room environment and the parts they would need to acquire and it seemed reasonable given the amount of time it would take. Finally, they assured me that if I did proceed with the recovery I wouldn’t have to pay anything if it turns out they couldn’t get the actual data off.

I talked it over with my wife and we decided that the information we were missing was just too important. I went back to my Mac Pro, dejected that Time Machine had failed me and that I didn't have a more robust backup system in place. I decided on a whim to check it one last time.

I used the right-hand time-line and clicked on the oldest date I could see. Time Machine animated the Finder windows and scrolled rapidly through the days and I noticed something odd. One of the days had an extra drive in it. Through patient scrolling I found something that I had missed earlier: on April 3, a little over a month before my drive had failed, there was a backup of BigDisk. There was no entry for it before or after that day.

I opened the drive entry in Time Machine and lo and behold my photos, movies and music were all there! All of the pictures I took after April 3 were missing but otherwise I had virtually all of my data! I immediately started the process of restoring the data to my external drive.

Time Machine - Not Confidence Inspiring

I’ve always been a big fan of Time Machine. I love that Apple has made backing up as simple as they have. This little experience has given me pause though. Time Machine had failed and I never received an error message. No notice, no warning. Just a lot of missing files.

While I am a huge fan of simple and powerful, I really need to stay informed. If I have a Mac with multiple drives does Time Machine back them up? The only option I have in Time Machine is to tell it what I want to exclude, which implies that everything else is going to be backed up.

In the mean time, I’ve decided that this little experience requires a complete change in the way I store and ultimately back up my digital life. Over my next few posts I’ll be covering how I have added new and more robust storage to my Macs, how I’m handling offsite storage and some of the tools I’ve found to make my information far more secure.

If you’ve got some background on how Time Machine works in a multiple drive system, please share it in the comments below.

Trust me when I tell you this: within the next 24 hours you should verify that you are properly backing up your important files. Pick any hard drive (or SSD for that matter) that you have. If it died now, how would you recover? The cost of prevention is a tiny fraction of the time, money and frustration you'll experience if you don't.

Comments

Unknown said…
This site has lots and lots of great FAQs about Time Machine:

http://pondini.org/TM/32.html
Anonymous said…
You want a backup that includes at least two copies using two different technologies keeping backups in at least two different locations.

For my Macs, that's TM+Crashplan with CP backing up to a Windows box and, for really important files, to Crashplan Central.

For my Windows boxes, it's Windows Backup + Crachplan to a Mac and Crashplan.

The specific technologies are less important that the ideas.

Bill
David Alison said…
@Chris Jones: Really helpful Chris - excellent resource.

@Bill: Agreed on using different technologies. My next blog post will cover that.

Back in the late 80s I was the IT director for a software company. We had this archaic backup system that I inherited and had a catastrophic failure of our primary LAN drive. Though the backup tape system was "working" properly and not generating errors, the ability to restore had never been tested. Turns out, the maker of the backup software had gone out of business and we couldn't restore from tape. That wasn't very fun either.

Scott Gammans said…
I run Windows 7 on my home workstation and backup to one of a pair of external drives each weekend that rotate between home and the office each Monday (a backup does no good if it perishes in a fire or gets stolen). Last year I suffered a drive failure on the workstation and was able to restore from backup without any major problems, but you'd better believe that scare prompted a new procedure: I now have a quarterly recurring reminder in Outlook that reminds me to test the integrity of my backup disks.
Anonymous said…
I second Crashplan plus TM! Just backed up over 600GB of pictures, music and files to CP and though it took a good deal of time, the benefit of having piece of mind is priceless!
David Alison said…
@Scott: I'm hoping the blog post helps put the data scare into a few people and makes them check their backups. The only reason this became a problem for me was because I'd become complacent and life had me too busy to even bother checking my backups.
Unknown said…
My first complete data loss occurred back in the Windows 3.11 days. Lost everything. I haven't always been so proactive, but my general philosophy has been you cant have too many backups.

I run TimeMachine to an external drive always plugged in.

I use SuperDuper! once a week (or so) on a second external drive which is stored that the office - insurance incase of theft or fire.

I have a Crashplan cloud based backup.

I don't check the Time Machine backups much, only during the occasional "oh crap, deleted that by mistake" moments.

SuperDuper verifies during backup.

I haven't checked the Crashplan backups, although I get a daily email of stats
Anonymous said…
David, head over to grc.com and get yourself a copy of SpinRite.

Simon
Anonymous said…
I have had the same scare when trying to restore a disconnected drive from time machine. Took me a while too to navigate past now to see the backed up drive. I also use TM and BackBlaze as my redundant backup options.

Did you figure out why TM stopped backing up the drive on April 9th?
David Alison said…
@Bradley: Nope. Nothing in the logs. No idea why it stopped working. I know the drive was functional because in late April I copied over nearly a hundred pictures from my daughter's prom. Those unfortunately were not recovered.
Anonymous said…
Brock360
I just had a big issue with my hard drive on my MacBook Pro. I keep backups with Time Machine and SuperDuper.
Time Machine, after 36 hours, still showed over 24 hours more estimated time to restore!

Got out of the restore and went with SuperDuper. 2-1/2 hours later I was up and running.

SuperDuper has saved me in the past and I would strongly recommend that this be used in your backup planning.
Anonymous said…
I'm wondering if the problem is related to the fact the disk was starting to fail long before it actually crash. When it happened to me, TimeMachine was taking forever to backup the failling disk. Most days, it would not have time to finish before i shut down the computer for the night (therefore no error message, it would only try to complete or start-over the next day).

Did you try to open the TimeMachine disk directly in the Finder? It gives direct acces to all available backups with direct Finder-like acces to all files. If you're lucky you might see the backup files from the missing disk even if they did not show up in the TimeLine (hopefully more recent than those already recovered). If not, you might have files with the extension ".InProgress". Those are backups that did not finish and won't show up in the TimeMachine TimeLine. I think they can be opened with "show package contents" (right-click)... won't have all files, but might have some of the more recent ones.

Good luck

Dan
David Alison said…
@Dan: Yep, tried looking through the bundles in Finder, though not in any great detail. A cursory inspection didn't reveal much and I didn't think to dive into the bundles themselves.

I was running a SMART utility on the drive but it never picked anything up, and there were no signs on the drive itself that it was having an issue. I had recently dropped over 100 RAW photos on the drive and had done some editing on them and there was no indication of a problem.

If Time Machine had experienced some kind of reading error I would have hoped that it would A) put an asterisk in the Time Machine menu bar icon or B) dropped something in the logs. It did neither that I saw.

I think the core lesson here is while Time Machine is a ridiculously simple tool to use, it's important to verify that it is indeed backing up everything you have because based on this experience, the error reporting is not as robust as it should be.
Anonymous said…
Time machine will only by default back up internal disks, as it appears you have discovered.

External disks are excluded by default, and there is no warning that it is doing so unless you opt to look in the backup 'options' and notice that your external disk is included in the exclusions list.

Time machine is great for entry level users who would never back up if it involved anything more than plugging in a disk and selecting 'yes back up'. But for professional application and advanced users it definitely has it's shortcomings.
David Alison said…
@Anon: Yes, I do know TM only backs up the internal disks by default. But it also claims to back up all of the internal disks. The drive it failed to continue backing up was indeed an internal disk.
Anonymous said…
I have a loaded Mac Pro (all four bays in use) and am backing up via Time Machine to a Synergy NAS. Yes, Time Machine will backup all of the internal drives.

However, I had problems after upgrading to Mountain Lion in that the Time Machine bundle would get marked as "bad" by Time Machine and would have to be recreated. I tried to recover it twice using instructions found on the 'net. It would work for a few days, but then fail again.

I trashed my old Time Machine bundle and created a new one. Also, Synergy released an update to their NAS software. I haven't experienced any issues since those two changes.

John
Neurus said…
I use http://www.haystacksoftware.com/arq/ + time machine

I'm going to start using this soon:

http://www.amazon.com/Synology-DiskStation-Diskless-Network-Attached/dp/B008U69LC4/ref=wl_it_dp_o_pC_nS_nC?ie=UTF8&colid=1B7LTMGOUM003&coliid=I35UDQ5EBEX06A

It was recommended by a co-worker (syncs to S3 too)
Anonymous said…
Correction - for some reason I typed "Synergy" when I meant to type "Synology." Lack of sleep, I guess.

John
David Orriss Jr said…
Hey David, LTNS...

I don't know if you have seen this tool before.. but for data 'spelunking' into your Time Machine backup, Back-in-Time can be pretty useful..

http://www.tri-edre.fr/english/products/backintime.html

David Alison said…
@John: ironically that setup is what I've gone with. A Synology DS213 and Arq. More on that soon.

@David: thanks for the software tip David. I really could have used that a few weeks ago.
Ian McIlvaine said…
Ian
I got the click-click-whirr sound yesterday, then the folder with question mark icon. My hard disk was gone, but luckily it was backed up with Time Machine to a Time Capsule.

I followed the instructions for restoring a Time Machine backup to another computer. It was a bit confusing, but eventually I found the backup for the computer with the dead hard drive, except that it wasn't showing anything after April 19, 2013. I looked, but that was it.

Like your experience, I never got a message that Time Machine was having a problem backing up. My local repair shop, Mac 911 in Santa Monica, CA said that they have seen similar cases with hard drives starting to go bad, and Time Machine not managing to complete a backup, but apparently being too embarrassed to say anything. So, here I am several months later crossing my fingers that the data recovery service will be able to extract my last three months of work.

Our whole world is going/gone digital, but where is that information and how will we access it 1, 3, 5, 10 or 20 years from now? I have a family bible from 1725 that I can still read, but a hard drive that may prove to be less useful than a blank piece of paper.
David Alison said…
@Ian: be sure to manually go through every single snapshot using the TimeMachine interface. It may have done a backup you want see on casual inspection.

Good luck!
Tomac86 said…
Hard Drive Clicking is a case of physical damage and no data recovery software would work on it. This is one nightmare that no one wishes to encounter. However, if your Time Machine hard drive is logically damaged then best practise is to Image it using CCC and restore on a healthy external drive which is formatted correctly.

Popular posts from this blog

Finding Davey: A Father's Search for His Son in the Afterlife

Keyboard vs. Mouse

Some cool Firefox add-ons