Archive for July 2007
Phanfare now backing up photos and videos to Amazon S3
I am happy to announce that we have moved our backups to Amazon’s Simple Storage Service, known as S3. All current backups go to S3 and we are copying over historical data. We currently have about 20 terabytes at Amazon and will have about 40 terabytes when all the data is moved over.
We also maintain a copy of customer photos and videos on our RAID servers in our NJ datacenter. Amazon promises multi-data center redundancy for S3 data, so Phanfare customers now have the peace of mind of knowing that their data is in at least three datacenters, on opposite coasts of the US (NJ and WA).
The natural question is, why did we do it? We did it because we wanted to provide the assurance of off-site backup and because the engineering costs (time and money) in building out something similar to S3 exceed any cost savings we might have realized by managing the storage ourselves in the medium term.
We actually get more redundancy than we had before. Before we backed up data on a second set of RAID servers in our NJ datacenter. Those servers were cheaper to operate than Amazon S3 assuming 2 year amortization, but they did not provide the same level of geographic or physical redundancy. So for us, using Amazon was not cheaper, but it was better. Including the opportunity cost of working on Phanfare’s core products versus working on offsite backup, using Amazon is a definite strategic win for us.
To make Amazon actually lower our overall long term costs, we would need to stop storing the data ourselves, instead just caching hot data. We have competitors that do that and it would be cheaper, but we are not positive it would be better. After all, right now, Amazon does not provide a Service Level Agreement (SLA) or even a phone number to call if you are unhappy with the Amazon web service. I don’t expect that Amazon will ever lose our data of course, but we would like an SLA before we bet our customers’ data on that.
Amazon’s web services are game-changing, especially to smaller companies. They allow small companies to have a cost position that rivals some of the biggest online competitors. Amazon’s web services also lower the cost of entry for new startups and hence increase competition and foster innovation. Both these things are good for consumers and we applaud Amazon for embarking on their ambitious plan of providing storage and compute in the cloud for other companies. I know they are also trying to amortize their own costs of development, but for us it is wonderful. With proper SLAs, we would consider using Amazon’s Elastic Compute Cloud too (EC2).
EC2 enjoys local area network (LAN) latency and bandwidth to S3 storage and that would make S3 that much more attractive as primary storage for Phanfare. One of the first rules of building a high performance system is to keep compute close to the data it operates on, and hence without using EC2, we would always need to cache data on our side for performance. The latency between NJ and Seattle is too long otherwise.
If you think about it, Phanfare does for consumers what Amazon does for us. Just as it would be difficult and expensive for a consumer to build a system to store his photos and videos into the cloud, accessible from anywhere and backed up in geographically distributed locations, it would be difficult and expensive for Phanfare to replicate Amazon’s level of web infrastructure.
An Engineer 