Arq and Glacier - Affordable Mac Cloud Storage

After the near miss I had in losing a considerable portion of my personal digital library I decided to do something about it and look into a cloud based solution for keeping my files safe. I’m still using Time Machine locally to back up nearly everything, I just wanted a final line of storage just in case.

I’ve been using DropBox for years for my documents and miscellaneous files. I have several Google Apps for Business accounts that store my emails and shared docs and spreadsheets. The code I write is versioned and stored in GitHub. For the most part I live off the cloud already, the only thing missing was my large collection of family photos and videos, which totaled nearly 140GB.

iCloud is cool and all, and I love the way it keeps my little iPhone photos synced, but at $100 / year for only 55GB, this is a pretty expensive solution. I looked at a variety of different cloud backup solutions and found them to be ill-fitted to my needs. While many of them have plenty of capacity and are pretty affordable, they usually require a rather heavy backup application to be running in the background monitoring changes.

Amazon to the Rescue

I’ve always been a huge fan of Amazon Web Services and specifically Amazon S3 (Simple Storage Service), a cloud based storage model. Amazon makes S3 incredibly resilient, providing a 99.999999999% durability rate. Though I’ve used Amazon S3 extensively myself as a software developer for my online services, I had never used it for storing personal data. Since up to 1TB of storage costs $0.095 / GB / month to store, my 140GB collection would cost me just a little over $13 / month to keep safe. That was a little steeper than I wanted.

Fortunately Amazon introduced another variation of S3 storage called Amazon Glacier. Designed to be just as durable as S3, with Glacier you cannot pull the data out very quickly without incurring some additional costs. This wasn’t an issue for me since this collection was really just a deep archive and backup to my backup. The advantage is Glacier is very affordable at $0.01 / GB / month. This meant my 140GB collection would cost me $1.40 / month on Glacier. Perfect!

Transporting the Files with Arq

The next challenge was getting the files up to Glacier from my Mac Pro. Amazon designs their services as something a programmer or systems administrator would access, not the average end user. I considered writing some scripts that would push my files up to Glacier but that seemed like too much work. Besides, it turns out somebody has already done that: Haystack Software’s Arq.

Arq's Main Window

This $29 download (30 day free trial) makes it really easy to just select what you want pushed up to your Glacier account and let it manage moving it up to the cloud. The interface is very spartan, which is actually great. You just need to have established an Amazon AWS account and signed up for S3, and Arq will take care of the rest.

One of the more reassuring features of Arq is that they have released the restore tool as an open source project on GitHub, providing some peace of mind.

The Restore Tool is available as open source

I set up Arq to run at 3AM, dropped the key photo and video folders on to it and let it run. It took a few days through my Cox Cable connection to get everything up to Glacier. If I add any new photos Arq picks them up on the next sweep and pushes them to Glacier.

Restoring from Glacier Takes Time

If you want the files to be readily accessible, then Glacier probably isn’t your best bet; you should use S3. Restoring even a single file from Glacier can take up to 4 hours before it even starts. This is one of the side effects of Glacier and why it’s best suited for deep, long term archiving, whereas S3 is better for files you need rapid access to.

Glacier Restores require great patience

For $29 up front and $1.40 / month, I’ve now got highly durable cloud copies of all my photos and home videos. Sure beats the panic I went through a few weeks ago.

Got a great cloud based backup / archiving solution people should know about? Drop a line in the comments. I’d love to know how others are handling this.

Upgraded your Mac to an SSD? Enable TRIM

As I wrote last week, the SSD upgrade for my Mac Pro went very smoothly. There were two important things I found out after I started reading the comments from that entry:

1. An SSD is a good performance upgrade but the OWC Mercury Accelsior_E2 provides incredible performance. Though quite a bit more expensive than a standard SSD, if you want the absolute best performance for a Mac Pro you may want to consider it. At 820MB/s it is over 5 times faster than the SSD I just installed. Hat tip to Eytan for pointing this out to me.
2. After adding a 3rd party SSD to your Mac, you need to investigate TRIM support. Derek brought up the issue in the comments and I spent some time investigating it.

What is TRIM?

TRIM is effectively a garbage collection model for SSDs. There is some great information on it on Wikipedia’s TRIM entry. The bottom line is that without TRIM enabled the performance of an SSD will suffer over time. This is something you’ll want to address.

I contacted Crucial technical support to ask if the Crucial 512GB M4 SSD I purchased supported TRIM. Crucial told me I had two options: occasionally "clean" the SSD or enable the operating system to send TRIM commands to the drive. The cleaning process sounds... tiresome. You restart your Mac every couple of weeks, holding down the Option key during the reboot, then let it sit at the boot menu for 24 hours as the firmware cleans the SSD.

You can however turn on TRIM support through a series of Terminal commands and script execution or by using a free utility called Trim Enabler.

Trim Enabler - just turn it on

Running Trim Enabler is pretty simple. Download and install it from the provided DMG and then flip the main toggle button to On. Once this is done you will need to reboot your Mac and TRIM will be enabled. You can confirm this by checking your System Information application and looking at the Serial-ATA entry for your SSD:

TRIM Support is enabled

Based on this information available you will need to restart and run Trim Enabler again if you upgrade OS X to the next point release. If you’re not sure, after an upgrade simply check System Information to ensure TRIM is enabled.

Got a tip for maximizing SSD performance? Drop a note in the comments!

Update: I always get the best comments. In case you don't read them, Hendrik pointed out that not all SSDs are created equal and that Sandforce based SSDs have built in garbage collection baked in. OWC SSDs in particular should not run Trim Enabler. Your mileage may vary. Clearly for the Crucial SSD I installed it's needed but if you install an SSD check with your manufacturer to see if it needs TRIM enabled in the OS.

Upgrading a Mac Pro to SSD

Mac Pro circa 2008

I bought my Mac Pro 5 years ago and other than the hard disk failures I just had with my add-on drives, the machine has been rock solid. I’ve kept it up on every version of OS X that’s been available and it runs Mountain Lion like a champ.

Even though my Mac hasn’t slowed down with age, it feels relatively slow ever since I added my new 15” MacBook Pro with Retina display to the mix. Between the 16GB of RAM, stunning display and ridiculously fast SSD, the new MacBook Pro seemed to run circles around the larger Mac tower.

Upgrading the processors didn’t seem reasonable. It already has 12GB of RAM and for the way I use the machine that provides plenty of headroom. The one area I figured could see dramatic improvement was by swapping out the 320GB boot disk with an SSD drive. This was the route I took.

Finding the Right SSD

I did a lot of reading on SSDs, looking at Amazon reviews and finding guidance from a variety of different sources where I primarily focused on SSD reliability. Since my Mac Pro only has a 3GB/s SATA interface—not the 6GB/s that most of the SSDs support and what comes on a new MacBook Pro—I wasn’t too worried about performance. I knew it would be dramatically faster than the conventional magnetic drive I had humming away in the drive bay.

Icy Dock adaptor and Crucial SSD

I ended up settling on the Crucial M4 512GB drive. One of the fundamental changes I wanted to make was to move my iPhoto Library to the SSD. That meant making room for an 80+ GB file bundle, which is why I went with the bump in capacity.

The first step in upgrading to the new drive was to ensure it would even mount in the drive rails of the Mac Pro. The SSDs are generally delivered as 2.5” laptop drives. Turns out what you need is a 2.5” to 3.5” SATA drive converter. For that I went with the Icy Dock enclosure. It’s a plastic box that is in roughly the same shape as a 3.5” internal drive and fits perfectly with the rails on a Mac Pro. If you do this upgrade be careful - there are lots of 2.5” to 3.5” converters out there but not all of them will line up the SATA interface with the Mac Pro’s drive system.

Installation with the Icy Dock takes just a few minutes

Installing the drive into the Icy Dock takes all of a couple of minutes. Simply open the box, line up the SATA connectors with the SSD and plug it in. Snap the lid on and it’s ready to be mounted to the Mac Pro drive rails. Once mounted—there are 4 #1 phillips screws—the drive slides right in.

Prepping the Drive

With the drive installed I fired up my Mac Pro. The drive is not pre-formatted so I needed to get to work on that. Since this was going to be a direct replacement of an older drive, I decided to try cloning the original to this new SSD. For that I chose Carbon Copy Cloner. CCC is a nice tool for not only creating a complete duplicate of a drive, but also providing incremental updates to keep a cloned drive current. This is critical if you want to be able to recover from a drive failure in minutes, not hours. For now, all I needed to do was clone the drive.

CCC saw the new drive and recommended that I use the Desk Center to create a Recovery Partition on the new SSD, which I did. Once that’s created the cloning process can kick off. For me CCC ended up pushing nearly 205GB of data onto the new drive. That process took 2 hours and 50 minutes to complete.

From there I went into the System Preferences and clicked on Startup Disk. After I changed it to the new SSD drive I clicked Restart and I was now running off the SSD.

Though everything appeared exactly the same on the new SSD, for some reason it didn’t carry over my Dropbox credentials. I had to log back into that tool manually.

Faster? Oh yeah. Faster.

Before doing the install I recorded the time it took to perform certain tasks and load some applications. I also ran the Nova Bench application and let it calculate a score before and after the conversion. Nova Bench upgraded my score from 1042 to 1094 after the SSD was in and reported that my disk throughput went from 52MB/s to 166MB/s. For perspective my MacBook Pro tests out at 323MB/s, roughly double the performance. Given the MacBook Pro is running a 6GB/s SATA interface, that makes sense.

I also tested some normal tasks:

Task	Hard Drive	SSD	Improvement
Power on to Login Page	1m 11.3s	24.6s	65%
Login to Desktop Ready	13.3s	3.8s	69%
Shut Down (no apps running)	1m 9.7s	27.0s	61%

Some decent performance improvement was achieved. Starting up felt much snappier than before. But how about regular use applications? Here’s what I found:

Application	Hard Drive	SSD	Improvement
Mail.app	5.2s	2.4s	54%
Safari.app	6.8s	1.6s	76%
Contact.app	2.1s	1.1s	48%
Pages.app	8.3s	2.7s	67%

I measured the improvement based on that first pass opening the apps. In every day usage this performance was pretty stunning. It felt as though I had upgraded to a brand new Mac. Every aspect of the Mac just seemed snappier.

Back when I was heavily into Windows I found myself upgrading machines every couple of years. It wasn’t really a hardware issue as much as the degradation of a Windows machine after extended use. You pretty much HAD to format the drive, reinstall Windows and start over again. Once you get to that point it was worthwhile just upgrading the hardware at the same time.

My Mac Pro has been different. It’s now 5 years old and has been through Leopard, Snow Leopard, Lion and now Mountain Lion. These were not full reinstalls but upgrades to the existing OS. It really is amazing to me that with this SSD upgrade I’ll likely get at least another 2-3 years out of the machine, if not more. Sure, Macs may cost a little more up front, but the ROI is easily justified if you'll be keeping it long term.

Got some advice for SSD upgrades? Please drop a note in the comments.

Hard Disk Clicking and a Time Machine Failure

It had been a long couple of months. We’d been working furiously on the launch of EasyGrouper and were just a few days away from having it go live. As I sat in my home office working on my new MacBook Pro I heard a deeply disturbing sound coming from my older Mac Pro tower a few feet away:

   ZZZZ-click 

   ZZZZ-click

   ZZZZ-click

That’s never a good sign. My Mac Pro is my main home machine and runs a lot of stuff for me. It’s got a 320GB main drive (“BootDisk”) running my apps and working documents, a 1TB drive called “BigDisk” that contains my family photos, videos, music, etc. and another 1TB drive called “Backup” for... you guessed it... backups.

I tapped the keyboard to wake up the screens and could see that the icons for my drives were all on my desktop when suddenly the BigDisk icon vanished and OS X gave me an error message that said I had not ejected the drive properly.

Not good.

My Mac could no longer see my BigDisk internal drive—the drive that contained over 380GB of rather important information. Virtually every digital picture I’ve taken since 1999. I didn’t panic though. I’m a big Time Machine advocate and it’s saved me before. I tried restarting the machine but the ZZZZ-click sound returned and it wasn’t even able to see my drive. I got back to work on EasyGrouper, determined to get this resolved the following Monday when I came up for air after the launch.

Fast Forward to Monday Night(mare)

I sat down Monday night to work on restoring my files. I clicked on the Time Machine icon in the menu bar, selected Enter Time Machine and the space traveling interface popped up with a Finder window.

My Time Machine Display

But there was a problem. A big problem.

I clicked on my machine in the list of items on the left pane and it only showed one drive: BootDisk. Oh yeah, Time Machine had been running for the last few days so it had been backing up my machine without BigDisk in there. I started to use the Time Machine interface to go back in time.

I backed up to the day I heard the clicking sound. No BigDisk. I backed up further. Nothing. The only drive that was being backed up was my BootDisk drive. It’s as though Time Machine didn’t even see the drive.

Now I started to panic. I counted on Time Machine to back this up for me and I never saw an error message or any indication in the menu bar that Time Machine was having a problem. I knew that at some point Time Machine had been backing up that drive because I had previously used it to recover a file I inadvertently modified and saved.

I looked into Console and searched for “backupd”; there were plenty of entries for it backing up my BootDisk, but that was it. No Time Machine error messages like I’d seen in the past. I had an external drive in my fireproof safe that contained a backup of my pictures and home movies, but I hadn’t updated it since the end of 2010. I had become lazy and complacent and it was about to bite me. The last three and a half years of my digital life had just vanished.

Recovery Time

I asked around the office and many people recommended I see if a hard drive salvage specialist could restore the data for me. I found Salvage Data in nearby Herndon, VA—they were super nice and would perform a free diagnostic to see if the drive could actually be recovered. I dropped it off and waited to see if they could help.

The next day I got a call. The drive’s primary head cluster had failed and the drive had logical errors that would need to be corrected. It would take 10-15 days if I didn’t need rush service. The price: $1,740. The price seemed high but Salvage Data had an excellent reputation and the people were extremely attentive and understanding. They described the work that would need to be done in a clean room environment and the parts they would need to acquire and it seemed reasonable given the amount of time it would take. Finally, they assured me that if I did proceed with the recovery I wouldn’t have to pay anything if it turns out they couldn’t get the actual data off.

I talked it over with my wife and we decided that the information we were missing was just too important. I went back to my Mac Pro, dejected that Time Machine had failed me and that I didn't have a more robust backup system in place. I decided on a whim to check it one last time.

I used the right-hand time-line and clicked on the oldest date I could see. Time Machine animated the Finder windows and scrolled rapidly through the days and I noticed something odd. One of the days had an extra drive in it. Through patient scrolling I found something that I had missed earlier: on April 3, a little over a month before my drive had failed, there was a backup of BigDisk. There was no entry for it before or after that day.

I opened the drive entry in Time Machine and lo and behold my photos, movies and music were all there! All of the pictures I took after April 3 were missing but otherwise I had virtually all of my data! I immediately started the process of restoring the data to my external drive.

Time Machine - Not Confidence Inspiring

I’ve always been a big fan of Time Machine. I love that Apple has made backing up as simple as they have. This little experience has given me pause though. Time Machine had failed and I never received an error message. No notice, no warning. Just a lot of missing files.

While I am a huge fan of simple and powerful, I really need to stay informed. If I have a Mac with multiple drives does Time Machine back them up? The only option I have in Time Machine is to tell it what I want to exclude, which implies that everything else is going to be backed up.

In the mean time, I’ve decided that this little experience requires a complete change in the way I store and ultimately back up my digital life. Over my next few posts I’ll be covering how I have added new and more robust storage to my Macs, how I’m handling offsite storage and some of the tools I’ve found to make my information far more secure.

If you’ve got some background on how Time Machine works in a multiple drive system, please share it in the comments below.

Trust me when I tell you this: within the next 24 hours you should verify that you are properly backing up your important files. Pick any hard drive (or SSD for that matter) that you have. If it died now, how would you recover? The cost of prevention is a tiny fraction of the time, money and frustration you'll experience if you don't.

Employee Lists on your iPhone or Android with EasyGrouper

It’s been a long time since I blogged, mainly because I’ve been so busy building up another product for public release. Now that the product is launched I’m hoping to spend a little more time covering technology topics. I figured what better way to restart the blogging than by writing about our new product.

The Problem with Contact Lists

Today nearly everyone carries a smart phone. Whether it’s a company issued phone or (more likely) a personal device, it’s hard to go anywhere and not find them. When you need to get in touch with someone you work with, the best way is usually to call or text their mobile phone if they aren't nearby. Sometimes it's a quick email from your smartphone if the message isn't urgent.

This is where the problems begin. Getting the people you work with onto your phone is largely up to each individual. Unless the company they work for has an Active Directory or LDAP server humming away—and it's been updated with everyones person mobile device—each person has to maintain their own contact list. Most small non-technology businesses don't have the resources to run a server like that.

For companies that only have a handful of employees, this isn’t really an issue. However once you hit the 15-20 people mark it becomes very difficult to balance. The most common way of dealing with this is by putting together a spreadsheet and either emailing it around or putting it on a shared drive (Dropbox, Google Drive, etc) that people can access. Pulling up this information is not generally very easy when you are sitting on a mobile device with a poor internet connection.

Enter EasyGrouper

Seeing this as an opportunity we developed EasyGrouper. What we wanted to build was a really simple way to put a list of just the people you work with—and their primary work contact information—on the web and on everyone's phones. It needed to be spreadsheet simple because that’s generally what we were replacing.

Main List

The idea was to make it completely separate from your regular contact list. The contact list model in most phones feels completely broken; every app wants to access my contact list; online apps like Facebook and Twitter want to mine my contacts to see who I communicate with; some desktop apps and online services want to synchronize my contact list and I get multiple copies of the same contact in my address book. For a common user—especially of the non-technical type—it’s a giant mess that people have just accepted as ugly.

Rather than exacerbate the problem, we decided to take a really simple approach. Put the list of people on our secure web site. Focus on the most basic contact information that people really need: name, title, email, phone numbers. Provide a web page that is responsive and looks good on mobile devices as well as custom apps that provide beautiful and responsive user experiences for the end user. Keep the data cached on people’s phones so the response time is extremely fast, and only perform updates to the data in the background.

Locations and Groups

Information Page

We went a little further than just putting together an employee list that would allow simple calling, texting and emailing. When people are out in the field or away and traveling they often find themselves needing to look up important information that is specific to the company or a project some people may be working on. Our Information pages are basically a complete web page that gets pushed down to each person’s phone automatically. Links in those pages are automatically linked into the features of your mobile device. Tapping a phone number will dial it, a link to a web site will load your web browser, an email address will load your mail client, etc.

This means that if a company wants to put up an information page for their office they can list out who the emergency contacts are, who to call for catering, directions to a client location, health insurance information, etc. Pretty much any information a person may need when they are out of the office.

Statuses

The one last feature we added was the ability to set a status. Very often when you are trying to find someone you work with and they aren’t where you expect them, you ask around: "Have you seen…?". The status feature lets you put a quick line next to your name in the list for a set time frame. It’s a great way to let people you work with know that you’re on vacation, at the dentist or traveling.

As we built and prototyped EasyGrouper we talked to a lot of small businesses about it. The reception we received was overwhelmingly positive and after having used it for a few months several of our early adopters have found EasyGrouper to be an indispensable tool.

We priced EasyGrouper to be very affordable. The client apps that we built for iPhone and Android are free and anyone can set up a 45 day trial account for up to 200 people. For US $25 per month a company with up to 50 employees can roll this out to everyone.

You can check out the video for EasyGrouper below or get more information by visiting our web site at www.easygrouper.com.

Warning - iOS5 Update Restore Error - Wait to upgrade

I patiently waited for the actual release of iOS5 - since I can't get myself an iPhone 4S for a little while without paying a premium. When I saw that iOS5 was available, I decided to update my 32GB iPhone 4 to the latest and greatest.

First off, Apple wanted me to upgrade OS X to 10.7.2 and iTunes to 10.5. That took the better part of an hour in itself, between downloading a pretty hefty collection of bits, installing them and then running a subsequent back up, I was a full hour in before I could even begin to upgrade my iPhone.

Once that was out of the way I tried updating the iPhone 4. Here's where the fun began.

First, an error

After sitting there for a very long time—I didn't pay close attention—iTunes reported this little error. Not good. When I clicked OK (it's not really OK but that's the one to push), I pulled my iPhone off the USB connector and the phone itself seemed fine. Still running iOS 4.3, and all of my content was sitting there fine.

I Googled up the issue and there was a huge thread on the Apple discussion forums on this. Since several people reported that they were able to upgrade by just continuing to try, I decided to give that a go. Three attempts in, I got something different. It looked like my iPhone was being updated!

Then suddenly I got the error again. Oh well. The problem was my iPhone was now in a restore state and not being recognized by iTunes - it had the little "plug me in to iTunes" graphic on the screen of the iPhone.

This was not good. I tried to do a restore and it would spin and spin, however I now got a different error message:

Looks like the Apple servers were completely overwhelmed. This was not good. Especially now that my iPhone was essentially in brick mode. I kept retrying and the restore would get to varying degrees of success, then eventually fail out. Sometimes with the above error, other times with one of these:

After this error, I would then get the old "Hey, this iPhone is in recovery mode..." error:

I tried everything. Rebooted my Mac several times, tried unplugging all the USB devices, etc. and still no luck. I was panicking - while I was sure I could get my apps back somehow, I had a lot of video and photos on my iPhone that hadn't been pulled off. I thought about the videos I had taken of my mom over the last year, right before she passed away. If those were gone, I was not going to be happy. I knew iTunes was making backups for me, and that my Mac was in turn backed up, but the way this was going, my confidence was not very high.

Eventually the iPhone appeared to restore and started up again. Then it flipped into "restoring" mode for my apps and then music and videos:

This process took the better part of an hour. All in, the upgrade took me about 3 hours from start to finish, and I still have an iPad 1 to update.

I think I'm going to wait a week or so to upgrade the iPad. If you are considering upgrading your iPhone now to iOS5, be very patient; you may be without your phone for a number of hours and the way the Apple servers are performing right now under load, perhaps a very long time.

UPDATE: My iPhone was successfully updated to iOS5 - total upgrade time was a little over 3 hours and took about a dozen retries at various stages. All of my information (contacts, calendar, photos, etc.) were intact.

I decided late this evening (Oct 12, 2011) to try updating my iPad, just to see if it would actually go through. This time the upgrade went much more smoothly - not a single error. It did require about 1.5 hours to complete however.

Remembering Steve Jobs

We mourn the loss of famous people—our entertainers, our leaders, our athletes—even though we never got a chance to meet them in person. They entered our lives through popular media and became part of it. We would watch them act or sing and it would engage us. They would speak about the important challenges we face and we would be inspired to address them. We would watch in awe as they performed incredible feats of skill, wondering how anyone could pull that off.

When one of those famous people die young, it's notable. That small window in our lives that they occupied goes dark and we miss them for a time.

Steve Jobs was different.

He wasn't an entertainer, yet he could capture the attention of the public and engage us. He didn't give many speeches, but the few he did were inspirational. Though he was not the classic business CEO he created multiple businesses that enjoyed extraordinary financial success. While these are all great accomplishments, it's not the main reason many people will remember Steve Jobs.

Steve had an uncanny ability to create things that became part of our lives. He didn't just build tools that made it easier to do something, he built tools that added joy to the task. He and his team put such a high value on the quality of a product that simply holding it in your hands left you with an appreciation for the attention to detail poured into it. There was a depth to the products Steve created; while something could be very simple on the surface, a little digging would lead to some cool discovery of a feature or capability that would make you smile. Like many of his product presentations, there always seemed to be "one more thing" subtly hidden on the device, waiting for you to discover it.

Personally I'm going to miss Steve's influence on the future of technology.

My thoughts are with Laurene, Steve's children and family, and the folks he worked with every day to make magic happen.