In early October, a major cloud provider lost 3 sites, one after another, within 4 days: fires caused by external factors shut down 2 data centers entirely and partially damaged the third. After the first night, the provider itself barely felt the loss — its services moved to neighboring sites. But dozens of its customers — banks, telecom operators, retail, government services — were down for most of the day. By day 4, the outages had reached the provider’s own services as well, and not only in its home country.
One cloud, one night, two outcomes, and 4 days later, a third. The difference wasn’t the provider but the decisions each of them made long before those days. And it’s the same picture we’ve been seeing in conversations about hosting in recent years: cloud, colocation, and rental are considered reliable by default, because “it’s Tier III and an SLA, after all.” Meanwhile, risk registers describe floods and meteorites in detail, while the loss of one specific building isn’t described at all.
Availability zones
One night, two outcomes
The cloud provides zones. Spreading a system across them is the customer’s job, not the provider’s.
The site is shut down entirely; the owner gives no recovery timeline and doesn’t confirm that it can be brought back into service at all. The provider’s own core services kept running. Customers who lived on that site had outages that dragged on for hours.
Why? Every public cloud divides resources into two types. Zonal ones — a virtual machine and its disk — physically sit in a specific building. Global ones — the cloud network, object storage buckets — aren’t tied to one zone, but they live inside a region. The provider says it plainly in its documentation: fault tolerance appears when an application is deployed across several zones at once.
In other words, the cloud provides zones, and spreading a system across them is the customer’s job. The provider built its services expecting to lose a site and lost one of several. A customer with everything in one zone lost its only one. At the moment of the incident, both had the same cloud. What differed were the decisions made a year or three earlier.
But 4 days later, the picture changed. When a third site went down, outages hit both those who were spread across zones and the provider itself. Multiple zones protect against independent failures: a fire in one building, an accident at one substation. They don’t protect against a threat that affects all buildings in a region at once — for that threat, the failure domain is no longer the building but the region and the provider itself. These are exactly the upper levels of the pyramid that people usually dismiss as “the end of the world”: in reality, they arrived within 4 days.
Someone else’s building
What you’re really renting
There’s no building in the word “cloud.” But there is a building — with an address, power feeds, cable routes, neighbors, and an owner.
“Cloud” is a convenient word: there’s no building in it. But there is a building. It has an address, a roof, power feeds, cable routes, neighbors, and an owner.
As long as everything works, you can’t see the difference between “my” server and “someone else’s.” It shows up the moment someone asks, “When will we be back up?” If the server is yours, your team answers that question. If it’s in someone else’s building, the partner’s press office does. That night, one of the affected services wrote exactly that to its users: “an incident at an infrastructure partner’s facility.” It had nothing more to say — it knew exactly as much about what was happening as its own customers did.
If the server is yours, your team answers the question “When will we be back up?” If it’s in someone else’s building, the partner’s press office does.
In someone else’s building, you don’t know what’s installed next to you, what the site’s real power feeds are and how many hours of fuel it has, in what order customers will be brought back after an incident, and where you are in that queue. And you can’t go there.
This isn’t a reproach to providers: a good provider has thought all of this through. But it has thought it through for its own business, not yours, and on the night of an incident, that’s the business it saves first. No contract clause will speed up the answer to your question.
Risk register
An upside-down pyramid
Floods and meteorites are described in detail. The loss of one provider building isn’t described at all.
Risk registers often carefully describe a flood — in a city about 500 meters above sea level — a meteorite strike, and region-scale disasters. If water has reached a machine room at that altitude, customers have bigger things to worry about than payments, and the question of service availability loses its meaning along with the business itself.
Meanwhile, the same registers treat colocation and cloud as ultra-reliable by default. The scenario “the provider’s building has ceased to exist” isn’t considered: the risk has been transferred to the provider, the provider has an SLA, the SLA has many nines. And yet nobody makes a backup to another region of the same cloud, and DR never even comes up.
But however many pages the SLA runs to and however many nines it has, on its own it solves nothing. If the data center has burned down, it’s your business that’s standing idle and losing customers — right now. By then, the operator’s business has already been hit, and it has other things to deal with besides your claims. Whether you get anything under the SLA, and then a penalty through the courts, is a matter of months, if not years. And even then, compensation is calculated as a percentage of the monthly service fee and has nothing to do with your revenue, regulatory fines, or reputation. So the risk hasn’t been transferred: what’s been transferred is the right to someday receive a small fraction of its cost.
And one more scenario that doesn’t appear in registers at all: the data center operator could have made a mistake itself. A mistake in its diesel generator calculations — capacity, startup, operation under real load. A mistake in calculating survivability without external power — fuel reserves, fuel deliveries, cooling while running autonomously. A mistake in calculating the site’s overall resilience to catastrophic events — fire, structural damage, separation of power feeds and routes. These calculations are taken on faith based on a Tier certificate, even though the certificate confirms the design and the built facility, not how the site behaves in an incident that nobody considered during design.
The lower levels of the pyramid are covered by equipment, the upper ones are described in documents. What falls out is the middle: an error in the operator’s calculations, the loss of one specific building, and the loss of access to it. And that is exactly the level where an organization still has a business, customers, and obligations, and where architecture decides whether downtime lasts minutes or weeks.
Cloud, colocation, rental, your own data center
A sensible pyramid: four hosting models
The hosting model changes who does the repairs. It doesn’t change who’s responsible for the second building.
If you lay out the risks by frequency and look at who covers each level, the picture for the four hosting models looks like figure 3.
Public cloud. The building, hardware, hypervisor, and network inside the site are on the provider. On you: distribution across zones, copies, keys, addresses, and the exit plan. The paradox is that in the cloud, the second building is closer than anywhere else — in the same console — and it’s in the cloud that it gets used least of all.
Colocation. The building, power, cooling, and security are on the operator. Hardware, spare parts, software, and links are yours. The equipment is yours and can be taken out, but access to the site is on the operator’s terms.
Bare metal rental. Someone else’s hardware in someone else’s building. Faulty components are replaced under the landlord’s SLA, and the generation is whatever it has in stock. Licenses, data, and architecture are yours. In terms of risk, it’s colocation with nothing to take out.
Your own data center. Everything is yours, including the engineering systems and the people who maintain them. But a single site of your own is the same single failure domain as a single zone in the cloud.
Three levels stay with you under any model. The first is a mistake, a failed change, or ransomware: copies and procedures protect against these, not the building. The second and third are the loss of a building and the loss of a city: only another building protects against these, and the only question is how far away it is. The hosting model decides who will fix the disks and air conditioners. Who will bring the system up somewhere else is your decision.
The hosting model decides who will fix the disks and air conditioners. Who will bring the system up somewhere else is your decision.
A sensible pyramid for a specific organization is built on five rules:
- the levels go from bottom to top by frequency, not by how spectacular they are: a disk failure in a large fleet is routine, a mistake or ransomware is a matter of a few years, and losing a building is a rare event, but over a ten-year horizon no longer a theoretical one;
- for each level, it’s written down who covers it — under the contract, not by gut feeling;
- the operator’s calculations — diesels, autonomy, site resilience — are verified, not taken on faith based on a certificate;
- the levels where you are the one covering them have an owner, a budget, and testing;
- levels above the loss of a city aren’t planned for but recorded in a single line as an accepted risk.
If your register marks the loss of someone else’s building as “transferred to the provider,” it’s worth opening the contract. As a rule, what you’ll find there is compensation as a percentage of the monthly bill — months later, not on the night your business is down.
Service inventory
The map comes first
Services, dependencies, failure domains, internal SLA, and customer impact — for each service separately.
Everything discussed above comes down to something that usually gets skipped: inventory. Not a list of servers from a procurement spreadsheet, but a map of what actually runs and what it depends on. You can’t build a risk pyramid for a fleet that nobody can list.
The minimum contents of such a map:
- hardware and resources — where it’s located, whose it is, which building and zone it’s in, and until what date it’s under support;
- services — not systems, but what a customer or employee actually uses: payments, app login, loan issuance, reporting;
- dependencies — which databases, queues, external APIs, DNS, certificates, and other parties’ clouds each service needs in order to work;
- failure domains — for each service separately: which building, which zone, which cluster, and which link takes it down;
- internal SLA — how long a service can be down and how much data can be lost, with RTO and RPO agreed with the business rather than taken by default;
- customer impact — how hard downtime of this particular service hits NPS and churn.
The last column is usually empty, even though it’s the one that sets priorities. An hour of mobile app downtime and an hour of internal portal downtime cost incomparably different amounts, yet in a risk register they’re often recorded as a single line: “IT infrastructure.” If you know each service’s impact on customers, you can see where a second zone pays for itself in a single incident and where a copy and a day to recover are enough.
The picture we come across most often is this: the internal SLA exists at the level of “everything at 99.9%,” nobody has drawn the dependencies, and a service’s failure domain coincides with the failure domain of its weakest dependency. So a service in three zones that goes to one zone for authorization lives in one zone.
The map goes stale with every release and every new contractor, so it’s worth reconciling it with every architecture change and before every failover drill.
Site loss scenarios
What can happen to a building
You can lose a site even if it’s physically intact.
A textbook example from telecom. A mobile operator with billing built properly. Subscribers don’t sit in one database — they’re spread across 7 first-tier databases, for online charging, and 5 second-tier databases, which handle service activations and deactivations and recurring charges. Rating of voice and internet traffic is moved into separate layers, each with 2 dozen VMs on 2 dozen virtualization hosts. Each physical database has its own storage system.
In other words, the failure domains are sliced so that any hardware problem affects only some of the subscribers. And so it was for years: something wasn’t working for someone, but you couldn’t say the service had “gone down” for a significant share of subscribers.
Until one day trouble hit the power supply of the entire building. Power dropped, dropped again a minute later — and that was it. The lighting stayed on, but there was no power for the servers.
A distributed system with all its parts in one building comes back up not as a distributed system but as one big one. A cold start of dozens of databases, arrays, and hundreds of VMs means a startup order, integrity checks, and reconciliation between tiers, and no amount of distribution speeds that up. Bringing things back in stages to about 70% took more than 12 hours, and the remaining 30% took another 2 days. Plus rating the traffic retroactively.
There’s no design error here in the usual sense. Every level of the pyramid below the building was covered impeccably. What was missing was the level above it — a second building. All the resilience turned out to be inside one failure domain, and that day it was the same one for everything.
A complete power loss and fire are the most obvious scenarios, but not the only ones. You can lose a site even if it’s physically intact.
- Fire and fire suppression. Even a local fire leads to the hall being de-energized. Gas fire suppression and the ventilation that follows take hours; water from neighboring rooms is even worse.
- Power to the entire facility. Tier describes the engineering systems inside the site. It says nothing about what happens if the site loses all its power feeds for a day, and the fuel, delivery contracts, and diesel capacity turn out to be less than the operator calculated.
- Routes. The site may be working but cut off: several fiber lines to the capital often run along a single corridor.
- Seizure and the owner. The hall is shut down over a case against a neighbor; the site is sold, the operator leaves the market — the building is standing, the lights are on, but you have to move out within 90 days.
Tier III, which commercial proposals often cite, is a good level, but it’s a property of a single building. Neither Tier III nor Tier IV promises that a building will survive the loss of itself: resilience at the building level comes only from a second building.
Backups and DR
A copy in the same zone
Even those who carefully back up every night don’t make a copy to another region.
After an incident, the question we hear most often is: “So where are the backups?” The answer is often an unpleasant one: in the same zone.
The protection ladder in the cloud is the same as on your own sites (for details, see issue 13, “Autosave mode=ON”), only each step is enabled and paid for separately, so it’s easy to skip. What each step survives is shown in figure 5.
From what we see, most stop at the second step. Even those who carefully back up every night don’t make a copy to another region.
And a copy isn’t DR yet. To come back up somewhere else, besides the data, you need:
- compute resources and quota for them — which, on the day of a mass outage, every customer of the downed zone will request at the same time;
- network, addresses, and routes;
- secrets and encryption keys that aren’t stored in the same zone;
- a startup order and a person who knows it;
- a test, run at least once, showing that all of this works.
A copy in another region without everything else means recovery in weeks, not hours, and it’s not what people have in mind when they write “risk covered by backup” in the register.
External components
Dependencies that aren’t on the diagram
A page can hang on someone else’s script even if your backend is working.
That night, even companies that didn’t have a single machine of their own in the downed zone went down.
A modern web page or app is assembled from dozens of external pieces: analytics scripts, sign-in through an external provider, payment widgets, maps, CAPTCHA, libraries from external CDNs, DNS at an external operator. If such a piece loads synchronously and without a timeout, the page waits for it forever.
These dependencies usually aren’t drawn on diagrams, but for the user it makes no difference: if the home page doesn’t open, the whole service isn’t working. This also includes partners who themselves live in the cloud: the payment gateway, the mailing service, external credit scoring. You may have spread your systems across three sites, but your partner hasn’t.
A separate lesson from those days is the line between data localization and service localization. In a neighboring country, the provider’s services ran with errors, and users were told that personal data is stored in the country, but the distributed infrastructure includes components in different countries, and a failure of one of them affects availability in several at once. So the requirement to store data locally is met, yet the failure domain still sits in someone else’s building abroad. If localization matters to your regulator, it’s worth asking about this as well: where not only the data lives, but everything the service can’t work without.
What to do about it: inventory the external calls on the critical path, put a timeout and graceful degradation on each one (“the map didn’t load” instead of “the page didn’t open”), and ask key partners where and how they’re hosted while the contract is still being signed.
Renting versus buying
Turnkey rental: what stays with you
People usually ask not “Is it reliable?” but “Why is it more expensive than it seemed?”
A separate branch of the same question is renting servers in a commercial data center instead of buying them.
One example from Central Asia. The customer has two sites of its own, with its main capacity on a hyperconverged platform. In a discussion with management, a proposal came up: instead of procuring a new generation, rent servers in a commercial data center.
At first glance, the landlord’s offer looked neat: a server with a large amount of memory, a Tier III site, 24/7 support. But on closer analysis, a noticeable share of the costs wasn’t included in the rental price.
- Licenses stay with you. BYOL model: the customer brings the OS, the virtualization platform, and everything else. The rented server was a previous generation with fewer cores, so each of the customer’s own servers required from 1.3 to 2 rented ones. Licenses counted per physical core grow along with that number (for details, see issue 08, “Lines and cores”).
- Storage is a separate line. The specification included only boot disks, not a single one for data.
- The link is separate. The service included external access with limited bandwidth. Replication between your own sites and the rented one requires a dedicated link, and the offer doesn’t include it.
- Compatibility is your problem. The customer had to check the server against the platform’s compatibility list on its own.
- Flexibility is relative. Minimum term — 12 months, prepayment for 6, deployment — from 5 business days if the equipment is in stock to 35 if it isn’t. In essence, this is leasing someone else’s equipment, not a pay-as-you-go cloud.
- Exit costs money. Moving data back, duplicated work during the transition period, reconfiguring the network.
And most importantly: renting in a single commercial data center changes nothing in the pyramid. It’s still one building, just someone else’s.
Renting isn’t always worse than buying: issue 07, “Four circuits,” covers the cases where it’s justified — temporary workloads, pilots, a second site for cold standby. But you need to compare the total cost over a 3-year horizon, not the monthly price of a server.
Addresses, route, people
Who runs the move
Hardware isn’t what determines how fast you move.
If a site is lost, everything comes down to how quickly you end up somewhere else.
Tellingly, as early as the second day, the provider reached agreements with several other cloud operators on an accelerated move for its affected customers. That’s the right step and the best answer to the question of whether you need a copy at another provider. Only everyone will be moving at once, and the first to go will be whoever already has a copy there.
- Addresses. If your services’ public addresses belong to the provider, moving means changing addresses, DNS, certificates, and settings at every external partner. Your own PI addressing and BGP let you redirect traffic to another site without changing addresses.
- Route. The link to the backup site must exist before the incident, not be ordered after it. On the day of a mass outage, carriers reroute backbone traffic, and the remaining nodes become overloaded for everyone at the same time.
- Capacity. In that same hour, every customer of the downed zone will want to come up on demand in another zone.
- People and procedure. Who makes the failover decision, in what order systems come up, who checks the result. If this isn’t on paper, it’ll be decided at night in a messenger chat.
Checklist
What to check in your own environment
Questions that are better answered before an incident, not after.
Bottom line
Instead of a conclusion
None of the customers who were down in those days chose “let everything go down.” They chose one zone because it was cheaper, copies nearby because it was faster, the provider’s addresses because that’s how it came out of the box. Together, that’s the debt from issue 12, “Debt and Default,” and in those days it was called in from dozens of organizations at once.
This isn’t an argument against the cloud or for your own data center. The question isn’t whether it’s yours or someone else’s, or how detailed the SLA is, but how many buildings you have and how many hours the move will take: on the night of an incident, it’s your business that’s down, and compensation comes later — if it comes at all.
Your server is truly yours if you can answer the checklist questions without calling a partner. If not, let’s lay out your pyramid together.