If you haven't already heard, a hiccup at Google recently left about 150,000 Gmail inboxes empty. As Ben Treynor, VP of Engineering and "Site Reliability Czar (24x7)" posted on the Gmail Blog, "I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centers?" Turns out it was a software upgrade glitch that affected all of those multiple copies, so 0.02% of Gmail users opened up their mail to find nothing. And 0.02% of all Gmail users is a lot of people.
3-2-1 backup rule, but giant corporations have to deal with backups on levels we can't even imagine. So how did Google recover the lost mail? Tape backups.
Tapes aren't a great first line of defense because they are sequential access and have long seek times, so it's not surprising that it took Google over a day to restore the lost messages. But it just goes to show that despite the many advances we've seen in storage technology sometimes that which might seem archaic is still the most reliable.