---
title: "The Leader You Never Elected"
subtitle: "Singleton coordination at the edge without a coordination service, and how Durable Objects break the coordinator-as-bootstrap-dependency cycle"
discipline: "Computer Science"
articleType: "Research Article"
authors: ["Dima Doronin"]
keywords: ["Distributed Systems", "leader election", "distributed locking", "fencing tokens", "Cloudflare Durable Objects", "edge computing", "coordination service", "single-writer", "control plane", "consensus"]
date: "2026-07-23T12:30:20.154Z"
---

# The Leader You Never Elected

*Singleton coordination at the edge without a coordination service, and how Durable Objects break the coordinator-as-bootstrap-dependency cycle*

## Abstract

Two production incidents from July 2026 — Coinbase's fifty-minute outage on July 14, in which a Kubernetes resource-name collision corrupted an Istio ingress gateway and a circular dependency in the deployment tooling blocked automated rollback, and the July 16 AWS CloudFront control-plane failure that took Canvas, Blackboard, and Hugging Face offline — share a structural fault: the machinery that changes and recovers a system is entangled with the system it manages, by shared fate and by a bootstrap cycle. Neither was a lock-service bug; the transfer we draw is structural. We examine two common reasons engineers deploy an application-owned coordination service — leader election and distributed locking — and argue that Cloudflare Durable Objects (DOs) dissolve much of that need by turning "who is the single writer for key k" into a naming operation rather than a quorum protocol, retiring one class of application-owned coordinator (the platform's own control plane remains, and we say so). We are rigorous about the boundary: DOs provide single-writer serialization and, via the output gate, durable, monotonic fencing-token issuance by construction — the property Redlock lacks — but the fencing gap identified by Kleppmann persists at the interface to any external resource, and coordination availability is deliberately sacrificed under partition (DOs are CP). We give formal safety/liveness statements, a proof from explicit platform assumptions that DO-issued tokens are strictly increasing across restart and migration, corrected resource-side enforcement using a strict guard, three concrete TypeScript Workers/DO patterns, and a map of where a purpose-built consensus system remains the right tool.

## 1. Two outages, one shape

On **14 July 2026**, Coinbase suffered a ~50-minute degradation of transfers, card debit authorization, settlement, and onchain services. The postmortem describes a small trigger — a *Kubernetes resource-name collision* made a routine deployment modify the wrong component (part of the Istio ingress gateway). The damage was contained; the *recovery* was not: the standard deployment/rollback tooling itself depended on the broken gateway, so operators fell back to a manual console rollback under emergency access [1, 2]. Two days later an AWS CloudFront incident traced to one Frankfurt availability-zone capacity limit escalated into a *global* VPC Origins control-plane failure, taking Canvas, Blackboard, and Hugging Face offline for hours [3].

**Neither was a leader-election or lock-service bug.** What they share is structural: the machinery that *changes and recovers* a system is entangled with the system it manages — by **shared fate** (Coinbase's rollback path shared fate with the gateway it was meant to fix) and by a **bootstrap cycle** (recovery needs the manager, the manager needs the system; the dependency graph has a cycle cut by hand under pressure). This paper studies one corner of that pattern: the coordination services deployed to obtain *a single authoritative writer* — for **leader election** and **distributed locking** — and argues that Cloudflare Durable Objects (DOs) are a clean foundational answer, not by a better consensus algorithm but by changing the question. Two bounds are stated up front: (i) DOs do not abolish coordination dependency, they *relocate* it from an application-owned service to the platform's placement/routing control plane, which remains a dependency; (ii) the model is **CP**, trading coordination availability under partition for consistency.

> A coordination service you must deploy is a coordination service you must bootstrap — and the bootstrap is where the outages live.

## 2. Definitions

Let $\Pi$ contend for a key $k$. An idealized global-time index $t$ is used only to *state* safety from an external observer's frame; no process reads $t$ — §3 treats local clocks as adversarial [4].

- **Mutual exclusion (locking):** $\forall t:\ |\{p:\mathrm{holds}(p,k,t)\}|\le 1$ (safety). Its liveness counterpart — some contender eventually acquires — is separate, and is where timeouts enter.
- **Leader election:** one member per monotone term $e$: $\forall e:\ |\{p:\mathrm{leader}(p,e)\}|\le 1$; liveness needs eventual synchrony. FLP forbids asynchronous consensus with one crash, so systems buy liveness with partial synchrony or failure detectors while keeping safety unconditional [5, 6, 7].
- **Fencing token:** $\tau\in\mathbb N$ per grant, strictly increasing over grant order $\prec$: $g_1\prec g_2\Rightarrow\tau(g_1)<\tau(g_2)$. It closes the gap between *holding* a lease and *acting* on it.

## 3. Why the coordinator becomes the problem

The textbook single-writer is a consensus service — ZooKeeper, etcd, Raft — or Kubernetes `Lease` objects in etcd [8, 5]. Often right (§8), but it must itself be deployed, upgraded, and recovered, and the tooling that does so often routes through the infrastructure it arbitrates: you cannot elect a recovery leader if electing one requires the thing you are recovering [1].

Even a perfectly available coordinator does not make a lock safe. Kleppmann's Redlock analysis: a process acquires a lease, pauses (a 30 s GC stall, an NTP jump, a 90 s network delay) past expiry, then *resumes* and writes — two writers, one instant. The fix is a fencing token — strictly monotone numbers, every write tagged, the *protected resource* rejecting any token not strictly above the highest accepted [9]. Redlock's flaw is not the missing lock but that it "does not generate a monotonically increasing number every time a client acquires a lock." Two lessons: **(L1)** clocks are for liveness, tokens for safety; **(L2)** the token generator must be a genuine single writer with durable state — a multi-master quorum cannot guarantee monotonicity; a single-writer store with durable increments can.

## 4. Durable Objects change the question

A Durable Object is a named, single-instance, stateful actor: **for a given ID, at most one instance runs at any time, anywhere, and every request for that ID routes to it** [10]. The ID is derived deterministically from a name, `idFromName(k)`, so the map from key to authoritative actor is a pure function each caller computes independently — no lookup, no election round. Two mechanisms make it a correct serialization point [10]: the **input gate** pauses new events while the object awaits I/O (run-to-completion), and the **output gate** holds outgoing messages until prior writes are durable. State lives in `ctx.storage`; SQLite-backed objects use `ctx.storage.sql`, with a per-object alarm via `setAlarm()`/`alarm()` [11].

You do not *elect* a leader for $k$; you *route* $k$'s work to its unique object. Against overclaiming: this eliminates the *application-owned* coordinator, not coordination itself — the platform's placement layer still coordinates, but it is not a service the application must deploy and recover, so one application-level bootstrap cycle (§3) is gone.

:::figure{caption="From a coordination service to a naming function. Left: contenders acquire from an external quorum service that is itself a deployed, application-owned dependency (the Coinbase cycle). Right: each key names its own single-instance object; authority resolves by idFromName with no election round and no application-owned service to recover."}
```
  CLASSIC (elect via coordinator)     DURABLE OBJECTS (name the writer)
  p1,p2,p3 ─► [ZK/etcd/K8s Lease]     req(k=A) ─► idFromName("A") ─► [DO A] sole writer
              │ quorum                req(k=A) ─► idFromName("A") ─► [DO A] same instance
     deploy ▲ │ ▼ recover ◄ bootstrap req(k=B) ─► idFromName("B") ─► [DO B]
     (application-owned cycle)        pure function; callers agree with no round
```
:::

## 5. What DOs guarantee — and what they do not

Platform guarantees, as explicit assumptions: **(A1)** at most one instance handles an ID at a time; **(A2)** on migration/restart a successor starts only after the predecessor stops; **(A3)** the output gate releases a response only after the writes it depends on are durable, and a durable write is visible to any later handler [10].

**Proposition (durable monotonic issuance).** *A DO that returns token $\tau$ only after the increment producing it is durable issues, over its grant order, a strictly increasing token sequence that survives restart and migration.*

*Proof.* By A1 and the input gate, `acquire` handlers do not interleave, so grants form a total order $g_1\prec g_2\prec\cdots$. Handler $g_i$ reads stored counter $c$, writes $c{+}1$, returns it; since handlers are sequential, consecutive grants read consecutive durable values, so $\tau(g_{i+1})=\tau(g_i)+1>\tau(g_i)$. Across a crash: by A3 the returned $\tau$ was durable before release, so it is not lost; by A2 the successor starts afterward and by A3's visibility clause reads that durable value, continuing the increment. Hence strict monotonicity across every boundary. $\blacksquare$

The Proposition says nothing about a **paused holder** — a client holding $\tau$ can stall, then write with a stale $\tau$; only the downstream resource can reject it. And nothing about **partition**: if the object's placement is isolated from some clients they cannot reach the sole writer, and no second instance is spun up (that would break A1). This makes DOs **CP** in practice — an engineering interpretation of [12], not a theorem from A1–A3: under partition they keep single-writer consistency over availability, the right trade for a lock but a conscious one.

**Resource-side enforcement (strict guard).** Let resource $R$ store the highest admitted token $H$ (initially 0) and accept a write tagged $\tau$ **iff $\tau>H$**, then set $H\leftarrow\tau$. The inequality must be *strict*: with $\tau\ge H$ a paused holder whose token equals $H$ could write again. It is also live, because every grant — including renewal — strictly increases the token (§6B), so each authorized write exceeds all previously admitted while any stale holder's token is $<H$. Proposition + strict enforcement gives **mutual exclusion of *effects* on $R$ even under arbitrary client pauses and clock error.** Exclusion of lease *holding* is unachievable against unbounded pauses; exclusion of *effects* is, and effects are what matter.

## 6. Three patterns (TypeScript, `cloudflare:workers`)

**A. Singleton by routing — no election.** When the "leader" was only ever a means to a single writer, route the key's work to its object.

```ts
import { DurableObject } from "cloudflare:workers";
export class ShardCoordinator extends DurableObject<Env> {
  async apply(op: Op): Promise<Result> {          // serialized by the input gate;
    const sql = this.ctx.storage.sql;             // this *is* the single writer
    const { seq } = sql.exec<{ seq: number }>("SELECT seq FROM meta WHERE id=0").one();
    sql.exec("UPDATE meta SET seq=? WHERE id=0", seq + 1);  // atomic: one handler at a time
    return this.applyToState(op, seq + 1);        // no mutex: callers queue, not race
  }
}
export default {
  async fetch(req: Request, env: Env): Promise<Response> {
    const key = new URL(req.url).searchParams.get("shard")!;
    const stub = env.SHARD.get(env.SHARD.idFromName(key));   // deterministic, global
    return Response.json(await stub.apply(await req.json()));
  },
} satisfies ExportedHandler<Env>;
```

Uniqueness is a property of the *name*: the coordination round is replaced by a hash.

**B. A lease authority that fences external actors.** To elect among *external* processes (a replica set's primary, queue workers, a one-runner migration), the DO grants leadership and issues tokens. Every grant, including renewal, strictly increases the token.

```ts
export class LeaseAuthority extends DurableObject<Env> {
  constructor(ctx: DurableObjectState, env: Env) {
    super(ctx, env);
    ctx.storage.sql.exec(`CREATE TABLE IF NOT EXISTS lease (id INTEGER PRIMARY KEY CHECK(id=0),
      holder TEXT, token INTEGER NOT NULL DEFAULT 0, expiresAt INTEGER NOT NULL DEFAULT 0);
      INSERT OR IGNORE INTO lease VALUES (0, NULL, 0, 0);`);
  }
  async acquire(candidate: string, ttlMs: number): Promise<Grant> {
    const now = Date.now();                         // server clock: TTL only, not safety
    const r = this.ctx.storage.sql.exec<Row>("SELECT holder,token,expiresAt FROM lease WHERE id=0").one();
    if (r.holder !== null && r.expiresAt > now && r.holder !== candidate)
      return { granted: false, holder: r.holder, expiresAt: r.expiresAt };
    const token = r.token + 1, expiresAt = now + ttlMs;   // strictly increasing
    this.ctx.storage.sql.exec("UPDATE lease SET holder=?,token=?,expiresAt=? WHERE id=0",
      candidate, token, expiresAt);
    await this.ctx.storage.setAlarm(expiresAt);     // deterministic reclamation
    return { granted: true, token, expiresAt };     // output gate: durable before returned (§5)
  }
  async alarm(): Promise<void> {                     // clear expired holder; token untouched → monotone
    this.ctx.storage.sql.exec("UPDATE lease SET holder=NULL WHERE id=0 AND expiresAt<=?", Date.now());
  }
}
```

The external worker renews before each externally-visible write (a fresh, strictly greater token) and tags the write with it.

**C. Strict enforcement at the resource.** If the protected resource is itself a DO, enforcement is a compare-and-set under the same single-writer guarantee:

```ts
async write(key: string, token: number, payload: Uint8Array): Promise<void> {
  const H = this.ctx.storage.sql.exec<{ token: number }>(
    "SELECT token FROM fence WHERE key=?", key).toArray()[0]?.token ?? 0;
  if (token <= H) throw new StaleFenceError(`token ${token} <= high-water ${H}`);  // STRICT (§5)
  this.ctx.storage.sql.exec(
    "INSERT INTO fence(key,token) VALUES(?,?) ON CONFLICT(key) DO UPDATE SET token=excluded.token",
    key, token);
  await this.persist(key, payload);                 // now known to be the current holder
}
```

For an *external* resource (R2, Postgres, a third-party API) the same strict check must live there — `WHERE token > :H`, a conditional put, an `If-Match`. That residual work the DO cannot do; what it buys is the hard half — a monotonic, durable, single-writer *source* of tokens no quorum had to provide.

## 7. Failure modes and trade-offs

- **Throughput ceiling.** One object is one serialized loop; a global singleton every request touches recreates the bottleneck. Mitigate with *fine keys* (per shard/account/job) — one hot object means the model is wrong.
- **Locality/latency.** One placement; distant callers pay round-trips. Fine off the hot path, worse for a chatty leader.
- **Migration.** By A2 the old instance stops before the new, so A1 holds; in-flight work is retried — the token makes retries safe.
- **Partition (CP).** An unreachable object is unavailable *by design*; paths that must write under partition need CRDT merge or a degraded mode.
- **Still a dependency.** Coordination moves to the platform control plane, which has its own failures. The gain is specific — no *application-owned* coordinator — not zero shared fate.

## 8. Where a real consensus system is still the answer

Reach for ZooKeeper/etcd/Raft or a purpose-built database when **(a)** coordination state needs its own replicated durability/availability SLO with tunable quorum and multi-region writes [8, 5]; **(b)** you need agreement over a large dynamic membership with cross-node invariants, not per-key authority; **(c)** cross-key transactions must be atomic over many keys — one object serializes one key, spanning many returns you to distributed-transaction problems; **(d)** you must promote a primary among nodes the platform cannot host — a DO can be arbiter and token source but cannot make an external replica accept writes. DOs are the right default for *keyed, single-writer* coordination and an excellent fencing-token authority; they are not a consensus engine.

## 9. Conclusion

Leader election and locking are usually solved by deploying a coordination service that becomes a shared-fate, circular-bootstrap dependency of exactly the kind that turned a small misconfiguration into a fifty-minute outage in July 2026. Durable Objects offer a different foundation: single-writer authority resolved by *naming*, monotone durable fencing tokens for free from the input and output gates, and a cellular blast radius from per-key placement — enforced safely with a *strict* high-water guard. Within its CP envelope and with fine keys it retires a class of application-owned coordinator and its bootstrap cycle; outside it — partition-tolerant writes, cross-key transactions, fleet-wide agreement — it is the wrong tool, and saying so is part of using it well.

## References

[1] Cryptotimes, "Coinbase Outage: Kubernetes Config Conflict Cause 50-Min Disruption," 22 Jul 2026. https://www.cryptotimes.io/2026/07/22/coinbase-outage-kubernetes-config-conflict-cause-50-min-disruption/
[2] Crypto Economy, "Coinbase Details July 14 Disruption," Jul 2026. https://crypto-economy.com/coinbase-details-july-14-disruption-that-halted-transfers-and-onchain-services-for-nearly-an-hour/
[3] TechTimes, "AWS CloudFront Took Down Canvas, Blackboard, Hugging Face in Control-Plane Failure," 19 Jul 2026. https://www.techtimes.com/articles/320971/20260719/aws-cloudfront-took-down-canvas-blackboard-hugging-face-control-plane-failure.htm
[4] L. Lamport, "A New Solution of Dijkstra's Concurrent Programming Problem," *Comm. ACM* 17(8):453–455, 1974.
[5] D. Ongaro, J. Ousterhout, "In Search of an Understandable Consensus Algorithm (Raft)," *USENIX ATC*, 2014.
[6] M. Fischer, N. Lynch, M. Paterson, "Impossibility of Distributed Consensus with One Faulty Process," *JACM* 32(2):374–382, 1985.
[7] T. Chandra, S. Toueg, "Unreliable Failure Detectors for Reliable Distributed Systems," *JACM* 43(2):225–267, 1996.
[8] M. Burrows, "The Chubby Lock Service," *OSDI*, 2006; P. Hunt et al., "ZooKeeper: Wait-free Coordination for Internet-scale Systems," *USENIX ATC*, 2010.
[9] M. Kleppmann, "How to do distributed locking," 2016. https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html
[10] Cloudflare, "Durable Objects" docs (single-instance guarantee; input/output gates; `idFromName`). https://developers.cloudflare.com/durable-objects/
[11] Cloudflare, "SQLite-backed Durable Object Storage." https://developers.cloudflare.com/durable-objects/api/sqlite-storage-api/
[12] S. Gilbert, N. Lynch, "Brewer's Conjecture and the Feasibility of Consistent, Available, Partition-Tolerant Web Services," *ACM SIGACT News* 33(2):51–59, 2002.