---
title: "Bulkheads at Entity Granularity"
subtitle: "Analyzing the virtual-actor model as a fault-isolation primitive, with Cloudflare Durable Objects and Facets as a case study"
discipline: "Computer Science"
articleType: "Methods"
authors: ["Dima Doronin"]
keywords: ["Distributed Systems", "Edge Computing", "Fault Isolation", "Blast Radius", "Durable Objects", "Multi-Tenancy", "Cell-Based Architecture"]
date: "2026-07-18T01:56:45.800Z"
---

# Bulkheads at Entity Granularity

*Analyzing the virtual-actor model as a fault-isolation primitive, with Cloudflare Durable Objects and Facets as a case study*

## Abstract

Large correlated outages increasingly arise not from capacity limits or bug density but from coupling: nominally partitioned workloads share a single critical dependency on their request path, which silently re-couples the fate of everything downstream. Secondary reporting on a recent AWS us-east-1 control-plane disruption is used only as a motivating datapoint, not as a verified root-cause record. We give a formal model of blast radius and show that fine-grained partitioning decays the independent-fault term as 1/C while remaining separate from a shared-dependency term max_d |S_d|/N. Against this model we analyze the virtual-actor (per-entity) execution model — a single-threaded, individually addressable actor with co-located durable state, realized by Microsoft Orleans, Akka Cluster Sharding, Azure Service Fabric Reliable Actors, and Cloudflare Durable Objects — as a bulkhead primitive at entity granularity. We map the four requirements of a bulkhead onto concrete actor mechanisms, then account for the model's limits: a shared routing control plane, per-entity single points of failure, explicit cross-entity protocols, and correlated storage. We also analyze intra-object capability isolation through Durable Object Facets as a nested bulkhead for untrusted sub-state. The contribution is a compact mapping between the classical fault-isolation literature and the virtual-actor model, with explicit accounting of the shared fate that entity-granularity isolation relocates rather than removes.

# Introduction

Large correlated outages are often a problem of *coupling* rather than raw capacity: a workload split into nominally independent regions, accounts, or tenants can still share one critical dependency on its request path, re-coupling the fate of everything downstream. Secondary reporting on a recent `us-east-1` disruption emphasized that resources deployed elsewhere can still be affected when global services route control-plane operations through a common endpoint [10, 11]. A separately reported trading-platform outage associated with a localized facility failure [12] has the same architectural shape, and a recent industry analysis attributes a growing share of outages to interactions between systems rather than to single failed components [13]. These incidents are illustrative only; the analytical claims below do not depend on any unverified root-cause detail.

The classical antidote is *bulkheading* — partition into independent replicas ("cells") so a failure in one cannot escape into another [8] — realized at scale as cell-based architecture and shuffle sharding [6, 9]. The obstacle has historically been cost: each cell carries routing, headroom, and operational surface. This article analyzes the **virtual-actor** model — a single-threaded, individually addressable actor with co-located durable state, activated on demand and reclaimed when idle [1] — as a bulkhead primitive at entity granularity. The model appears in Orleans grains [1], Akka Cluster Sharding [2], Service Fabric Reliable Actors [3], and Cloudflare Durable Objects [16].

This manuscript extends two prior articles by the same author on per-entity coordination boundaries [17, 18]. The new material here is fourfold: a compact blast-radius model that separates independent-fault decay from shared-fate floors (§2), a bulkhead-oriented mapping from actor mechanisms to isolation requirements (§3), a failure-mode catalogue of what the model does *not* solve (§4), and a nested-isolation discussion for untrusted sub-state (§5). Conclusions are claims about the model, not about any single vendor.

# 2. A compact model of blast radius

Let $\mathcal{T}=\{t_1,\dots,t_N\}$ be a set of $N$ entities and let $\mathcal{C}=\{c_1,\dots,c_C\}$ be a set of $C$ *cells*, each treated here as an independent failure unit. For the static-sharding case, an assignment $\sigma: \mathcal{T} \to \mathcal{C}$ maps each entity to exactly one serving cell. If cell $c$ fails, the availability blast radius is

$$
B(c)=\frac{|\{t\in\mathcal{T}: \sigma(t)=c\}|}{N}.
$$

Under uniform singleton assignment, the expected blast radius of a failed cell is $\mathbb{E}[B]=1/C$. A monolith has $C=1$ and therefore $B=1$; static sharding over $C$ cells yields $1/C$; and at one cell per entity ($C=N$) the independent-fault term becomes $1/N$. The value of the virtual-actor model is that it makes this fine-grained partitioning economically plausible by letting the runtime create cells on demand rather than by pre-provisioning them.

Independent-fault decay is only half the story. Let $\mathcal{D}$ be the set of critical shared dependencies on the request path, and let $S_d \subseteq \mathcal{T}$ be the set of entities affected when dependency $d \in \mathcal{D}$ fails. Define the correlated-fault blast radius for that dependency as

$$
B_{\mathrm{corr}}(d)=\frac{|S_d|}{N}.
$$

Then the worst correlated exposure in the design is

$$
\max_{d \in \mathcal{D}} B_{\mathrm{corr}}(d)=\max_{d \in \mathcal{D}} \frac{|S_d|}{N}.
$$

This term is separate from, and not bounded by, $1/C$. A system may drive the independent-fault term to $1/N$ while still retaining a near-global shared dependency. That is the sense in which entity-granularity isolation relocates rather than removes systemic risk.

Shuffle sharding is adjacent but not identical to the singleton model. If each entity is assigned a random $k$-subset of $M$ workers, then two entities overlap in expectation by a $k/M$ fraction of their shards [6, 9]. That statistic is an overlap measure rather than a direct availability bound, but it captures how much execution surface two tenants are likely to share. I use it here only to contrast one-cell-per-entity isolation with a design that intentionally permits partial co-residency.

# 3. Why virtual actors act like bulkheads

A bulkhead at entity granularity needs four properties.

1. **Independent addressing.** A request for one entity must resolve to that entity's coordination boundary and not to a shared tenant pool. Name-derived actor identity provides this property.
2. **Independent execution.** One poisoned request must stall at most one entity. Single-threaded actor execution provides that containment.
3. **Independent state.** Data corruption or schema damage must remain local to one entity. Per-actor durable state provides that locality.
4. **Independent lifecycle.** Fine-grained isolation must be cheap enough to keep around. Lazy activation and idle reclamation make one-cell-per-entity economically feasible.

These properties explain why a virtual actor is a useful *bulkhead primitive*. One authoritative instance per identity also makes per-entity consistency cheap: there is one serial decision point for writes to that entity, so many local invariants can be enforced without distributed locking or cross-cell consensus [4].

# 4. What the model does not solve

Entity-granularity isolation improves the independent-fault term, but it does not eliminate shared fate.

- **Shared routing or placement control planes.** Resolving a name to a live actor still depends on runtime infrastructure shared by many entities [10, 11].
- **Per-entity single points of failure.** Aggregate blast radius can drop to $1/N$ while any one entity still depends on one authoritative instance and its backing store [5].
- **Cross-entity invariants.** Consistency inside one actor does not solve transfers, quotas, or invariants that span multiple entities; those need explicit multi-actor protocols [4].
- **Correlated storage.** Private logical state can still rest on a shared physical substrate.

The right reading of the model is therefore conditional: it is excellent for isolating faults *within* one entity boundary, and incomplete for faults that are global to routing, storage, or coordination across many entities.

# 5. Nesting the bulkhead for untrusted sub-state

Some workloads need mutually distrusting sub-states inside one entity. Durable Object Facets provide an example [14]: a supervising object can host named child objects, each with its own private database, while the supervisor retains policy and control state. Untrusted code running in a child can therefore be confined to the child's local store rather than the supervisor's tables.

This is best understood as a **nested trust bulkhead**, not as a separate availability domain. A child bug can be prevented from mutating the supervisor's control state, which is a real containment benefit, but the supervisor and children still share the surrounding runtime and lifecycle. The mechanism sharpens *who may touch which state* more than it changes the outer failure domain.

# 6. Discussion

The reflexive response to regional incidents is often "go multi-region," but the model above shows why that is not enough. Adding regions raises $C$ only for the independent-fault term. It does not help if the request path still traverses a common control plane, storage substrate, or cross-entity coordinator. The useful design question is therefore not just "how many cells exist?" but also "which dependencies remain shared across those cells?"

Within that framing, the virtual-actor model is attractive because it makes per-entity isolation operationally affordable. Its limits are just as important as its strengths: it converts aggregate exposure into per-entity exposure, leaves shared control planes in place unless the application explicitly mitigates them, and pushes cross-entity correctness into protocol design. Those tradeoffs are acceptable only when the protected invariants are primarily local to one named entity.

# Conclusion

The virtual-actor model is a strong bulkhead primitive when the system's natural consistency boundary is one named entity with mostly local state. In that setting it drives the independent-fault blast radius toward $1/N$ while keeping the programming model simple. It is not a universal answer: shared control planes, correlated storage, and cross-entity invariants remain outside the protection envelope. The practical contribution is therefore not a claim that actor runtimes remove systemic risk, but a sharper accounting of which risks they contain cheaply and which ones they merely move.

# References

[1] Bernstein, P., Bykov, S., Geller, A., Kliot, G., and Thelin, J. *Orleans: Distributed Virtual Actors for Programmability and Scalability.* Microsoft Research Technical Report MSR-TR-2014-41, 2014. https://www.microsoft.com/en-us/research/publication/orleans-distributed-virtual-actors-for-programmability-and-scalability/
[2] Lightbend. *Akka Cluster Sharding.* Akka Documentation. https://doc.akka.io/libraries/akka-core/current/typed/cluster-sharding.html
[3] Microsoft. *Introduction to Reliable Actors in Service Fabric.* Azure Documentation. https://learn.microsoft.com/azure/service-fabric/service-fabric-reliable-actors-introduction
[4] Helland, P. *Life Beyond Distributed Transactions: An Apostate's Opinion.* CIDR, 2007. https://www.cidrdb.org/cidr2007/papers/cidr07p15.pdf
[5] Gilbert, S., and Lynch, N. *Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services.* ACM SIGACT News 33(2), 2002. https://doi.org/10.1145/564585.564601
[6] Brooker, M. *Workload Isolation Using Shuffle-Sharding.* Amazon Builders' Library. https://aws.amazon.com/builders-library/workload-isolation-using-shuffle-sharding/
[7] Miller, M. S. *Robust Composition: Towards a Unified Approach to Access Control and Concurrency Control.* PhD thesis, Johns Hopkins University, 2006. https://erights.org/talks/thesis/
[8] Amazon Web Services. *REL10-BP03: Use bulkhead architectures to limit scope of impact.* AWS Well-Architected Framework, Reliability Pillar. https://docs.aws.amazon.com/wellarchitected/latest/framework/rel_planning_for_recovery_failure_isolation_bulkheads.html
[9] Amazon Web Services. *Guidance for Cell-Based Architecture on AWS.* AWS Solutions Library. https://aws.amazon.com/solutions/guidance/cell-based-architecture-on-aws/
[10] Network World. *AWS hit by US-East-1 outage after data center thermal event.* https://www.networkworld.com/article/4168878/aws-hit-by-us-east-1-outage-after-data-center-thermal-event.html
[11] technical.ly. *Why AWS US-East-1 outages take down the internet.* https://technical.ly/entrepreneurship/aws-us-east-1-outage-explainer/
[12] InfoQ. *Coinbase postmortem: a localized AWS failure triggered a multi-hour trading outage.* https://www.infoq.com/news/2026/06/coinbase-aws-failure-postmortem/
[13] Uptime Institute. *Annual Outage Analysis 2026.* https://uptimeinstitute.com/resources/research-and-reports/annual-outages-analysis-2026
[14] Cloudflare. *Durable Objects in Dynamic Workers: give each app its own database.* Cloudflare Blog. https://blog.cloudflare.com/durable-object-facets-dynamic-workers/
[15] Cloudflare. *Outbound connections keep Durable Objects alive.* Cloudflare Changelog. https://developers.cloudflare.com/changelog/2025-03-17-durable-objects-outbound-connections/
[16] Cloudflare. *Durable Objects.* Cloudflare Developers Documentation. https://developers.cloudflare.com/durable-objects/
[17] Doronin, D. *Per-Entity Coordination at the Edge.* openscience.tech, published 2026-07-17.
[18] Doronin, D. *Coordination at the Edge.* openscience.tech, published 2026-07-12.