[...] Pie in the sky guesses as to why Amazon went down - but since Om is into Infrastructure now - he runs these sort of speculative posts.  Its actually written by someone named Allistair Croll. [...]

[...] Amazon went down today . . . a rare occurrence. Gigaom explains why it may have happened and what it means to you . . . read it here. [...]

[...] (Alistair Croll/GigaOM) Category: Techmeme | source article link Alistair Croll / GigaOM: Why Amazon Went Down, and Why It Matters  —  Amazon.com’s U.S. retail site became unavailable around 10:25 AM PST, and [...]

[...] have what??? Posted by: The ADC in Events, Opinion Alistair Croll over at Gigaom had an interesting dissection of Amazon’s recent outage and an interesting deduction based on the facts regarding what went [...]

[...] Friday, Amazon’s U.S. site went off the air, and later some of its other properties were unavailable. Lots of folks who wouldn’t let me quote [...]

[...] downtime incidents crossed our paths recently that we thought deserved comment. You probably have read about the first (Amazon), but maybe missed the second (Southern Company Nuclear Power [...]

[...] light of Amazon’s latest downtime issues, Gigaom explains why Amazon went down and why it matters. In a thorough explanation, Gigaom bets the problem to be with the CDN or AFE. The moral of their [...]

[...] isn’t the first time load balancers have been implicated in an outage at Amazon. At O’Reilly’s Velocity conference, conference co-chair Jesse Robbins talked about a [...]

Excellent and timely piece Om.

Not relevant to your blog, but wanted to put. I hate your search. It is really bad. When I search for “GigaOM Show”, it does not even come in the first 10 links.

One more which I would like to point out - can not go back using the back button (I use Opera) and I do not think you are using AJAX.

Please do something on it, at least on search.

Satya

Sorry. It is written by Alistair and thanks Alistair for the timely piece.

Others comments - I reall want the search to be better.

Satya

I humbly suggest the more accurate title “I Have No Idea Why Amazon Went Down, Either.”

Oh, my, I hit my limit. Translation, please?

Frankly, I wish I’d thought to grab the HTTP headers at the time, which would have told me a lot more.
As a friend of mine points out, the thing that’s answering this “speaks” HTTP. Which means it’s likely a proxy of some sort — it’s conversant, but the thing behind it isn’t. And there were multiple DNS entries in the DNS response, so whatever happened blanketed several sites
This could be the caching layer, the security layer, or anything else designed to sit between the Internet and the application servers themselves.

More than likely a human error, internal misconfig… as you can balance traffic prior to it hitting the network…

(link)

Oh, and @Liz: Some kind of proxy service died that likely lives in the data center, in front of the app server. That could be an HTTP-aware firewall or something else, likely a custom Amazon thing.

The bigger point here is that there’s always a point of failure, and web systems are a complex mixture of technologies.

Thanks, Alistair. I got lost somewhere there in the middle of Om’s analysis.

Time for all webservices to be 100% transparent on uptime issues. Trust.salesforce.com led the way, and interesting the new Acrobat.com also has a health.acrobat.com service …

Thanks for this uninformed piece. I forwarded it to my friends for a laugh.

To anyone the least bit knowledgeable, it’s pretty obvious that whatever happened was a huge deal. It wasn’t as simple as “some front ends went down”. It was truly an epic fail.

I do not agree on highly complex thing that Amazon is claiming. Replication or load balancing is not rocket science or something which deserves a Nobel prize. Tomcat is giving it for free, though volume of transaction will be low wihout code modification.

Security layer is not the case, you would have got HTTPS headers. Neither is caching layer.

What is most suprising is that apology page did not work, too! It seems not to be a software failure, rather a complete blackout of hardware, which is stunning.

I think people expect too much from web services. It makes sense to report it as news, but Google News has some 200+ redundant stories about the site going down and still know one has an answer. I’m not going to worry too much about someone not being able to buy a Kindle for a whopping 3 hours!

This speculation is ridiculous; admitting Amazon’s system is complex and then continuing to guess at causes of the outage is a fool’s game. Without information, which this ‘article’ has none of, you have no idea what did or did not happen.

it went down because of this:
(link)

Wow. Are you sure you want to stick to being a journalist? You seem to have a calling (and perhaps aspire?) to be an operations guys. It would’ve been much more interesting if you had taken the business impact angle.

By the way, have you ever run an operation of such size and complexity or do you just enjoy being a pundit? A very shallow and disappointing piece, I must say.

Yea, dude, Amazon is a mess. There is not a simple reason, it is literally 5-8 year old custom written perl and some other programming language I forgot.

Every developer who works there has to carry a pager around during a rotation…what does that tell you?

It is actually more surprising that Amazon only goes down as infrequently as it does.

Ask anyone who has worked there, under the hood, it is hardly a pillar of robust well architected software. They got it to work and they pretty much just tack shit on at this point.

Shoot! I meant Alistair’s analysis, not Om’s. I must remember to check the byline.

It’s not overly interesting that Amazon’s store was down from an online retailer perspective. It *is* interesting that the people selling EC3, S3, etc, can’t keep their core retail site up. More and more online startups are just thin wrappers around Amazon services, if they can’t keep one of their own sites up why should I have confidence they can keep mine up? Yeah, they’re probably different internal groups, shoemaker’s children, blah blah — that’s their problem, not mine.

word on the street: DOS Attack

Thanks Om. Appreciate your response.

You know what - for one content search wrt to GigaOM Show, I had to google and point to your website.

Missed an important one.

I am with Alistair’s article and initial analysis. Your website informed me first on this. And it will help me to influence my circle on decisions regarding the importance of robust software infrastructure. Thanks again.

@Liz, none of my comments were aimed at you or anyone in person.

It is this kind of news, comments and analysis, which makes software so lovable and exciting. And may be that is the reason your website is loved so much also.

Satya

@ryan: Sorry you felt that way. Glad you got a laugh, then.

@mel: I doubt it’s an outage due to a single product, given that they handle peak loads at Christmas and so on, although it’s possible. Remember Amazon got into the AWS business to use up excess capacity it had from such peaks.
Further investigation into the custom headers on the site suggest there’s a proxy layer that wasn’t working; what’s strange is that the layer went down across all IPs (which, admittedly, seem to go to the same router.)

@peterb: From a business standpoint it looks like lots of people have already estimated the cost of downtime based on annual revenues. This analysis ignores two things: The peaks and valleys of purchasing patterns, and the fact that many of those buyers will simply return later. And yeah, I’ve helped run some big sites, FWIW, but most were standard three-tiered environments.

@Geo: Websense and Narus are saying it’s probably not DDOS. The fact that traffic was working (albeit with the wrong HTTP contents) supports this to a degree.

No load balancer that I know of has a configurable “Apology page” when all real servers in the pool are unreachable. To the load balancer, the apology page would be seen as another active server and added to the rotation along with the normal httpd servers. Most of the time maintenance pages are manually brought up and down by the administrators.

You’ve also left out an important, but interesting failure mode in your analysis. There’s a good chance that users were directed (through GSLB and/or standard load balancing) to a set of nodes which had functioning HTTP but a non-functional back end. This is a very difficult situation for the load balancers to detect, as a server is answering the phone but then doing nothing once the “call” has been completed.

Too bad for Amazon, really. People need to take Internet operations far more seriously and understand that Amazon probably shares many of the same problems that Twitter is having right now: Rapid growth from increased demand, scaling, and redundancy.

Wasn’t amazon hit by the massive global botnet attack yesterday? My boss was talking about it. Youtube went down yesterday as well. Ditto imdb.

@john: You’re spot on about the detection, though: Administrators who do a simple HTTP up/down check on their load balancers, rather than looking for known strings in the page, wind up having “valid” but broken pages served to the outside world.
BTW I know lots of sites that have a policy-based apology rule (using something like F5’s iControl) but as you point out, that only works when the load balancer knows something’s awry.

we use (link) they provide us with both global dns and dns load balancing, as well as content cache and site redirection. there are other companies like internap and even ultra dns that offer similar products… strange amazon should invest

@john adams - check out haproxy, keepalived, ldirector, mod_throttle (?)
they all have automated sorry server capabilities. (link) is very sweet in my opinion, lots of features and flexibility and speed.

The web servers or some other monitoring system (monit maybe) should have been able to detect the dead backend servers and remove those from the mix.

I am off to get more salty snacks for the boss…

Alastair and Geo,

It most certainly was a DOS attack, I assure you.

And Seattle-ite is right about everything he says in his post, and that was in part why the DOS was possible.

Seems like, The site is down again….

I was thinking of Amazon as Google in Shopping domain (item search, reviews etc..).
Time to re-tink???

@John Adams

I think you haven’t been looking around very much. Modern load balancers (application delivery controllers) a la F5 BIG-IP, have configurable “apology” pages when all nodes are down. This technology has been around for quite a while, it’s not something new or unknown in the industry at all.

Lori

Check out David Scott’s interview at the Business Forum: (link) . It seems Amazon (and a lot of other organizations suffering data thefts, outages, bad projects, etc.) needs his book!

This may be a dumb question, but did the Amazon cloud go down too?

@john adams: Netscaler load balancers (as Amazon is rumored to use) can re-direct to a ‘Sorry Page’ when all backend services are down. It’s a simple config, and we require it on all our load balanced services.

That, and the fact that the HTTP error page that was presented looks just like the one generated by Netscalers when operating in proxy mode, indicates to me that the load balancer layer was up & functional, but there was nothing behind it to which to send traffic, and that there was no redirect enabled.

Having said that, re-directing a high volume site to a sorry page is a challenge itself. We maintain a load balanced pool of servers on a separate pair of load balancers, just to handle the sorry page from the load balanced applications.

–Mike

@Mel: I think that rumor is hilarious too. It wouldn’t surprise me that a bunch of gamers writing scripts to auto-buy available items could crash Amazon. That would be awesome.

I still have the headers in my scroll back buffer:

$ wget -S (link) -O /dev/null
–14:22:58– (link)
=> `/dev/null’
Resolving (link) . 72.21.210.11
Connecting to (link) |72.21.210.11|:80… connected.
HTTP request sent, awaiting response…
HTTP/1.1 503 Service Unavailable
Server: NS_6.1
Content-Length:62
Connection: close
14:22:58 ERROR 503: Service Unavailable.

It appears that they are indeed running Citrix Netscalers (Server: NS_6.1) which is what returned the 503 error you see above.

“Sorry” pages only work if configured; they are not a default. Maybe Amazon hasn’t gotten around to that. ;)

So how do you manage and diagnose such complex systems? You are probably talking about 1,000s of devices that all could be the root cause of the issue. How do you isolate the root cause? Classically, you’d have some type of monitor with rules to detect when certain issues occurred. This MIGHT point you to the right location. However, with IT being the main differentiator to the end customer, new changes are constantly being rolled out. The system is always growing, changing and the customer usage patterns are always altering over time. So the rules originally written tend to get lightened up so that you don’t have alert storms. Now with the rules loosened up, you might not detect the failure and, even if you do, the events will not be as helpful in detecting the root cause. You could throw more and more smart people at the problem and constantly update and maintain your set of rules. However, as the system gets more and more complex this human cost will grow at an enormous rate. You need something different, a tool that automatically detects issues and adjusts to the changing system and usage patterns. You need a tool that uses statistical analytics to weed through the noise of the system and determines the relationship between the business information and the IT information in order to allow you to quickly get to the root cause of issues like this. If you got a single alert that told you that the load balances were getting a higher than average reconnect rate, your number of sales was dropping below normal, your average load on all web servers was way below normal, the number of search transactions was way below normal, the normal of connected users was way below normal etc., you would be able to have a quick idea on where to start. No need to experience all your line of defenses being down again.

It’s too bad about Amazon going down for a spell but maybe it inspired someone or several someones to check out their local library. I hope so.