# Distributed Async Await — specification, server implementers' guide, and SDK authors' handbook — Full Content --- url: / title: Distributed Async Await --- # Distributed Async Await **Distributed Async Await** is a protocol for durable execution: a set of rules that lets a function survive process crashes, resume where it left off, and appear to a developer as if it ran uninterrupted. Any conformant server and any conformant SDK interoperate. [`resonatehq/resonate-specification`](https://github.com/resonatehq/resonate-specification) is the executable abstract machine in Lean 4 — the normative, machine-checkable definition of the protocol's core handlers and state transitions. These pages are the prose specification: the human-readable companion that explains the same protocol. Where prose and Lean disagree, the Lean model wins. ## Three tracks ## Also [**Resonate docs**](https://docs.resonatehq.io) — how to *use* Resonate, the reference implementation: SDK guides, tutorials, and deployment. --- --- url: /sdk/coroutines title: Coroutines in your language --- # Coroutines in your language Everything up to here has treated "the durable function" as a thing the worker drives one step at a time — run it until it blocks, suspend, resume, replay. This chapter is about the *thing being driven*: how a developer's function is shaped so that it can be paused at a durable step and resumed later, and how your SDK steps it forward. This is where the host language stops being a backdrop and starts dictating the design, because the mechanism that lets a function pause mid-flight and hand control back to your runtime is the single most language-specific decision in the whole SDK. [Earlier chapters](/sdk/replay-and-determinism) deferred the deep treatment to here, and named the split they were deferring: **TypeScript and Python express a durable function as a sync generator; Rust expresses it as an async future.** Neither is more correct. Each is simply how durable suspension expresses itself naturally in that language's runtime, and your first real design decision is which shape fits the grain of yours. ## The same step, two shapes A durable step is a point where the function stops, hands a description of what it wants to your SDK, and waits to be handed a result back. The whole question is what language construct *is* that stop-and-hand-back. In a generator language it is a `yield`. The developer writes a generator function and yields a description of each durable call; the SDK receives the yielded value, does the durable work, and resumes the generator by feeding the result back in: ```ts // TypeScript — a durable step is a yield function* transfer(ctx: Context, args: Args): Generator { yield ctx.run(debit, args.from, args.amount); // durable step const ok = yield ctx.run(credit, args.to, args.amount); // durable step → value return { ok }; } ``` In an async language it is an `.await`. The developer writes an ordinary `async fn`, and each durable call is awaited like any other future: ```rust // Rust — a durable step is an .await #[resonate::function] async fn transfer(ctx: &Context, args: Args) -> Result { ctx.run(debit, DebitArgs { /* … */ }).await?; // durable step let ok = ctx.run(credit, CreditArgs { /* … */ }).await?; // durable step → value Ok(Outcome { ok }) } ``` The two read almost the same on the page. The difference is *who holds the pause*. A `yield` hands control out of the function to whatever is iterating it — your SDK. An `.await` hands control to the async runtime, and your SDK has to arrange to be the thing the runtime hands it to. That difference is the rest of the chapter. It is `function*`, not `async function*`. The generator yields synchronously between steps; it never sees a JavaScript `Promise`. All the asynchronous work — talking to the server, awaiting local children — happens in the SDK's driver loop *around* the generator, not inside it. This is deliberate: keeping the developer's function synchronous-between-yields is what lets the SDK take complete control of when it advances, which is what replay needs. ## Driving a generator (TypeScript and Python) In a generator model your SDK owns a loop that steps the generator, inspects what came out, acts on it, and steps again. The generator is inert between calls; nothing happens until you call `.next()` (TS) or `.send()` (Python). TypeScript's driver is `Coroutine.exec` (`resonate-sdk-ts/src/coroutine.ts`). It instantiates the generator once and runs a trampoline: each turn calls `.next(input)` on the generator, reads the yielded value, and — depending on whether it is a local invocation, a remote one, or an await on an existing handle — either creates the durable promise and feeds the result back as the next `input`, or, if the awaited promise is still pending, returns a `Suspended` result and lets go (the [suspend/resume](/sdk/suspend-resume-settlement) machinery takes it from there). A yielded call is mediated by a `Decorator` (`resonate-sdk-ts/src/decorator.ts`) that translates each `Yieldable` — the union of `LFI`, `LFC`, `RFI`, `RFC`, and a future handle — into an internal instruction the loop acts on. Python's driver is the scheduler (`resonate-sdk-py/resonate/scheduler.py`), which advances each coroutine through `Coroutine.send` (`resonate-sdk-py/resonate/coroutine.py`). The shape of `send` is the whole model in miniature: it is called with `None` on the first step, with `Ok(value)` to feed back a resolved result, or with `Ko(error)` to throw an exception *into* the generator at the yield point — which is how a failed durable step surfaces as an ordinary `try`/`except` in the developer's code. What comes back out is the next thing the generator wants: a local invocation, a remote one, an await handle, or a terminal "I'm done." Both SDKs share a small, important trick. The developer writes `yield ctx.run(...)` and expects to get the step's *value* back — but "create the promise" and "await its result" are two different operations internally. The generator drivers fuse them: yielding a "call" (a collect-style invocation) auto-inserts the await, so the developer's single `yield` produces a value in one step. TypeScript tracks this with a `pendingCall` marker in the decorator; Python sets a `skip` flag in `Coroutine.send`. Same idea, both hide the two-step dance behind one `yield`. ## Driving a future (Rust) Rust has no generator to step. A durable function is a plain `async fn`, and the `#[resonate::function]` macro (`resonate-sdk-rs/resonate-macros/src/lib.rs`) turns it into an implementation of the `Durable` trait (`resonate-sdk-rs/resonate/src/durable.rs`), whose `async fn execute` *is* the developer's body. Your SDK drives it by `.await`-ing that future on the async runtime, not by stepping it. So how does an `.await` on an unsettled durable call pause the function without blocking a thread? The answer is that a not-ready durable call resolves its future *immediately* — with an error that means "suspend." A remote call whose promise the server reports as still `Pending` returns `Err(Error::Suspended)` from its future (`RpcTask::into_future` in `resonate-sdk-rs/resonate/src/context.rs`; `RemoteFuture` in `resonate-sdk-rs/resonate/src/futures.rs`). That error propagates up through the developer's `.await?` calls and out of `execute`, where the task driver in `resonate-sdk-rs/resonate/src/core.rs` catches it, collects the remote dependencies the function registered on the way out, and suspends the task. On resume, the whole `execute` future is run again from the top — and this time the previously-pending calls find their promises already settled (via preload), so their futures resolve with real values and the function walks forward to the next unsettled point. The generator SDKs suspend by *returning a value from the driver loop* (`Suspended`) — the SDK is the iterator, so it just stops iterating. Rust suspends by *propagating an error up the future chain* — the SDK is not the runtime, so it signals out-of-band and lets the runtime unwind. Both re-run the function from the top on resume and rely on replay to fast-path the completed steps. The host-language plumbing is entirely different; the durable-execution contract is identical. This is also why the [determinism contract](/sdk/replay-and-determinism) lands the same way in every language: re-running from the top only works if the function reaches the same durable steps in the same order, whether those steps are `yield`s the SDK counts or `.await`s the runtime re-drives. ## The invocation surface What a developer actually calls to make a durable step has two independent axes, and getting their names right is part of feeling idiomatic. - **Local vs. remote** — does the step run *in this process* (executed against your local function registry, tagged `resonate:scope: "local"`) or get dispatched as a promise *the server routes to any worker* in the target group (`resonate:scope: "global"`)? - **Invoke vs. collect** — does the call return a *handle* immediately so the function can start more work before awaiting any of it, or does it *auto-await* and hand back the result value directly? TypeScript and Python name the four combinations explicitly — `lfi`, `lfc`, `rfi`, `rfc` (local/remote × fire-invoke/fire-collect) — with friendly aliases for the common forms: `ctx.run` is the local collect, `ctx.rpc` the remote collect, and the invoke forms are `beginRun`/`beginRpc` in TypeScript (`resonate-sdk-ts/src/context.ts`) and the snake-case `begin_run`/`begin_rpc` in Python (`resonate-sdk-py/resonate/resonate.py`). The "collect" forms auto-await; the "invoke" forms return a future/promise handle you await later with a second `yield`. Rust uses a builder shape instead of four names: `ctx.run(f, args)` and `ctx.rpc("name", args)` each return a task object that is *both* awaitable and spawnable (`resonate-sdk-rs/resonate/src/context.rs`). Awaiting it directly (`.await`) is the collect form; calling `.spawn().await?` first gives you a handle (the invoke form) you can await separately. The mapping is exact even though the vocabulary differs: | Concept | TypeScript / Python | Rust | |---|---|---| | Local, get a value | `ctx.run` (`lfc`) | `ctx.run(f, a).await` | | Local, get a handle | `beginRun` / `begin_run` (`lfi`) | `ctx.run(f, a).spawn().await?` | | Remote, get a value | `ctx.rpc` (`rfc`) | `ctx.rpc("f", a).await` | | Remote, get a handle | `beginRpc` / `begin_rpc` (`rfi`) | `ctx.rpc("f", a).spawn().await?` | | Fire-and-forget | `ctx.detached(...)` | `ctx.detached("f", a).spawn().await?` | The "invoke" / handle forms are what make concurrency expressible. ## Composing concurrent durable calls Sequential durable steps are the easy case — `yield` one, get its value, `yield` the next. Concurrency is the reason the invoke forms exist: start several durable calls *before* awaiting any of them, then collect. In the generator SDKs, fan-out is a sequence of invoke-style yields that each return a handle, followed by a sequence of awaits on those handles: ```ts // TypeScript — fan-out then fan-in const a = yield ctx.beginRpc("fib", n - 1); // handle, not value const b = yield ctx.beginRpc("fib", n - 2); // handle, not value return (yield a) + (yield b); // await both ``` Python expresses the same with `ctx.rfi(...)` handles awaited later. Each invoke creates the promise (and, for remote calls, dispatches the work) immediately; the awaits then block on results that are already in flight. Because all the calls are created before any await, the work overlaps. In Rust the same intent rides the async runtime's own combinators. `.spawn()` hands back a `DurableFuture` backed by a channel, so two spawned calls run concurrently and you await both; `tokio::join!` over two `ctx.run(...)` tasks composes them cooperatively. The structured-concurrency story is the language's, not the SDK's — your job is only to make the durable task type compose with it. `detached` is the deliberate exception to all of this: a fire-and-forget call whose result the parent never awaits, used when you want to *start* durable work whose lifetime is independent of the caller. It is not part of structured fan-out/fan-in; it leaves a promise behind and returns. ## Choosing the surface your community expects The mechanism is dictated by the language; the *surface* is your call, and it is worth making deliberately. A few things the reference SDKs got right and that a new SDK should weigh: - **Match the language's blocking idiom exactly.** If durability does not ride the construct developers already use to wait for things — `await`, `yield`, the futures combinators — every durable call will read as foreign, and the "feels like a normal function" promise breaks. - **Make the common case one token.** Auto-awaiting "collect" calls so the default is `value = yield ctx.run(...)` (not a manual two-step invoke-then-await) is what keeps simple workflows simple. Reserve the handle forms for when the developer actually wants concurrency. - **Don't invent four names if the language gives you composition for free.** Rust folds invoke/collect into `.await` vs `.spawn()` because the async ecosystem already has a handle type; forcing `rfi`/`rfc` names onto it would fight the grain. In a generator language with no such handle convention, the explicit names earn their keep. Get this layer right and the whole engine underneath — promises, tasks, replay, suspend/resume — stays invisible to the developer, which was the point from the start. Next: [time, retries, and policies](/sdk/time-retries-policies) — durable sleep, scheduling without a live process, and how (and where) a retry policy rides along with a call. --- --- url: /sdk/encoding-and-codecs title: Encoding, codecs, encryption --- # Encoding, codecs, encryption A promise stores a value, but the server has no idea what your developer's value *is* — it never deserializes it, never inspects it, never runs their code. From the server's side a promise's payload is an opaque string it stores and hands back byte-for-byte. That opacity is deliberate, and it pushes a job entirely onto your SDK: turn a native value — a function's arguments, its return, a rejection — into a string on the way out, and back into a native value on the way in. The thing that does that is the **codec**, and this chapter is about building one, plus an honest accounting of where encryption actually stands. ## `data` is a string; `headers` give it meaning Recall the two-field shape a promise's `param` and `value` each carry: `data` and `headers`. The contract is small and worth stating exactly: - **`data`** is an *encoded string* (or absent). The Rust type says it plainly in its own doc comment: on the wire, `data` is a base64-encoded JSON string; only after the codec decodes it does `data` hold a real value (`Value` in `resonate-sdk-rs/resonate/src/types.rs`). TypeScript's `Value` (`resonate-sdk-ts/src/network/types.ts`) and Python's `DurablePromiseValue` (`resonate-sdk-py/resonate/models/durable_promise.py`, where `data` is typed `str | None`) carry the same idea. - **`headers`** is a string→string map that travels *alongside* `data` and tells the codec how to read it — content type, encoding, format markers. This is the forward reference the [promise lifecycle](/sdk/promise-lifecycle) chapter left open: headers exist so that decoding doesn't have to guess. The codec is the only component that understands the relationship between the two. It encodes a native value into `data` (and, optionally, writes `headers`), and on the way back it reads `headers` to decide how to turn `data` back into a value. ## The default pipeline All three SDKs build `data` from the same two transforms — **JSON serialization** (portable) and **base64** (safe to carry as a string) — with an optional **encryption** step layered in. The exact layering differs, and it's worth being precise: Rust encrypts the JSON *before* base64 (`Rust value → JSON → encrypt → base64`), while TypeScript base64-encodes first and encrypts the result (`value → JSON → base64 → encrypt`). With the default no-op encryptor the distinction is invisible; it only matters once a real cipher is in play, and it's the kind of thing to pin down rather than assume uniform. Decoding runs each SDK's pipeline in reverse. TypeScript wraps this in a single `Codec` class whose internal `JsonEncoder` does the JSON-and-base64 work and is careful about JavaScript's rough edges — it serializes `Infinity`/`-Infinity` and `Error`/`AggregateError` instances through sentinels so they survive a round-trip (`resonate-sdk-ts/src/codec.ts`). Its `encode` chains the JSON encoder into the encryptor; `decode` reverses it; and `decodePromise` decodes both `param` and `value` while preserving the original `headers`. Rust's `Codec` (`resonate-sdk-rs/resonate/src/codec.rs`) does the same in trait form — its doc comment spells the order out: "Rust value → JSON → encrypt → base64 → `Value`." Python takes a more compositional route, which is instructive precisely because it's built from small pieces. Its `Encoder` is a structural protocol — `encode`/`decode`, generic over input and output types (`resonate-sdk-py/resonate/models/encoder.py`) — and the SDK ships a handful that *compose*: `JsonEncoder`, `Base64Encoder`, `JsonPickleEncoder`, `HeaderEncoder` (writes a value into a `headers` key), `NoopEncoder`, plus two combinators — `PairEncoder` (run two encoders, produce a `(headers, data)` pair) and `CombinedEncoder` (pipe one encoder's output into the next). The default application encoder is assembled from these (`Options.get_encoder` in `resonate-sdk-py/resonate/options.py`): ```text PairEncoder( HeaderEncoder("resonate:format-py", JsonPickleEncoder()), # → headers JsonEncoder(), # → data ) ``` So a default Python value is written *twice*: a portable JSON form into `data`, and a Python-specific jsonpickle form into a `resonate:format-py` header. That second write is the seed of the cross-language caveat below. A clean division to carry into your own SDK: the *application* codec turns the developer's native values into `(headers, data)`, and a lower *transport* encoder handles the base64 wrapping at the network boundary (Python uses `Base64Encoder` here in its stores; TS and Rust fold base64 into the one codec). Keeping the application transform separate from the transport wrapping is what lets a developer swap in a custom value codec without touching how bytes ride the wire. ## A pluggable codec The reason the codec is a named, swappable component and not a hardcoded `JSON.stringify` is that JSON is not always enough — a value might need a domain-specific format, a more compact binary encoding, or compression. So the codec is injected at construction (`new Resonate({ encryptor })` in TS, `ResonateConfig` in Rust, a custom `Encoder` in Python's `Options`), and your SDK should expose the same seam. Two design notes the reference SDKs make concrete: - **Encode both directions of every field.** `param` (arguments) and `value` (result, including a rejection) all flow through the codec. A custom codec that handles arguments but forgets rejections will round-trip success and corrupt failure. - **Carry `headers` so decode never guesses.** Python's `HeaderEncoder` exists for exactly this — when a value is encoded in a non-default way, a header records which way, and decode dispatches on it. A codec that writes a custom `data` format without a header marking it has built a thing only it can read, and only as long as nobody changes it. ## Encryption: the honest state of it It is tempting to present encryption as a solved, uniform feature. It is not, and the handbook's habit of [surfacing the hard parts honestly](/sdk/introduction) applies — so here is exactly what ships. **TypeScript and Rust define an encryption seam and ship no cipher.** Each has an `Encryptor` abstraction — a TS `Encryptor` interface (`resonate-sdk-ts/src/encryptor.ts`) and a Rust `Encryptor` trait (`resonate-sdk-rs/resonate/src/codec.rs`) — wired into the codec as an injectable dependency. But the only implementation either ships is a `NoopEncryptor` passthrough. There is no AES, no ChaCha, no key management. Encryption is *possible* in TS and Rust by supplying your own `Encryptor`; it is not *provided*. **Python ships no encryption seam at all.** There is no `Encryptor` protocol, no encrypt/decrypt step in the pipeline. A developer who wants encryption in Python would have to smuggle it inside a custom `Encoder`, which is architecturally awkward because the encoder interface isn't scoped to it. And the two seams that *do* exist are not even the same shape: | SDK | Encryption seam | Cipher shipped | Operates on | |---|---|---|---| | TypeScript | `Encryptor` interface | none (`NoopEncryptor`) | the whole `Value` — can rewrite `headers` too | | Rust | `Encryptor` trait | none (`NoopEncryptor`) | raw bytes only — cannot touch `headers` | | Python | none | none | — | That `Value`-vs-bytes difference matters for anyone who needs to annotate encrypted data (a key id for rotation, an algorithm marker): the TS seam can write that into `headers` in the same step; the Rust seam can't reach `headers` without changing the codec itself. **Do not claim parity here.** If your SDK offers encryption, say which of these shapes it takes and what it actually ships, rather than implying a uniform feature the reference SDKs don't have. ## Cross-language compatibility and versioning Because JSON-then-base64 is the shared default, a value written by one SDK is *generally* readable by another — that's the payoff of the common pipeline. Two seams in that compatibility are worth knowing: - **Language-specific encoders don't travel.** Python's default jsonpickle header (`resonate:format-py`) carries Python class information (`{"py/object": ...}` markers) that only Python can rehydrate. A TS or Rust reader ignores the header and reads the plain-JSON `data` fallback — which is lossy for rich Python types. jsonpickle is Python-to-Python only; lean on the JSON `data` for anything cross-language. - **Error shapes differ.** TS and Rust serialize errors through a `__type` discriminator; Python uses an `__error__` key. An error object encoded by one SDK won't necessarily rehydrate *as an error* in another. If cross-language rejection fidelity matters to you, pin a shared error encoding rather than assuming one. On **versioning encoded data over time**, the honest answer is that none of the SDKs solves it: there is no schema-version field in `headers`, no forward/backward-compatibility guard, no migration path for old payloads. This is a real gap, and a place your SDK can do better deliberately — reserving a `headers` key for a codec/schema version costs nothing now and is the only inexpensive moment to add it. If you skip it, a value encoded today is a value you can never safely change the shape of, and the first time you try, replay over old promises will hand your decoder bytes it no longer understands. Next: [local mode for development](/sdk/local-mode) — running the whole engine in-process so a developer can build against it with nothing else installed, and the one emulation gap you have to be honest about. --- --- url: /sdk/function-registry title: The function registry & invocation surface --- # The function registry & invocation surface The worker loop (chapter 5) knows how to run a function. This chapter is about the two ends of that: how a developer's function gets a *name* the server can dispatch to, and how an invocation of that name becomes a durable promise the server actually delivers. It is the developer-facing surface — `register` and `run`/`rpc` — and the wiring underneath it. The pivotal fact, the one everything else hangs on: **a fresh process must be able to find the code for a promise it did not create.** When a worker resumes an execution after the original process is gone, all it has is a promise carrying a function *name*. It has to map that name back to a live function pointer. That is what the registry is for, and it is why durable functions are registered by name rather than passed by reference. ## The registry: names to code The registry is a map from name to function. A developer registers each durable function once at startup, and the worker consults the map whenever an execute message names a function to run. The shape differs only in richness across the reference SDKs: - **TypeScript** keeps a two-way map (`resonate-sdk-ts/src/registry.ts`) — name → version → item, and a reverse map from the function pointer itself, so a developer can invoke by either the string name or the function value. - **Python** mirrors that with forward and reverse dicts (`resonate-sdk-py/resonate/registry.py`). - **Rust** keeps a single `HashMap` keyed by name (`resonate-sdk-rs/resonate/src/registry.rs`); the name is a compile-time constant attached to the function by the `#[resonate_sdk::function]` macro. On the worker side, the dispatch is the same everywhere: read the function name out of the incoming promise's param, look it up in the registry, run what you find. If the name isn't registered, the worker can't run it — which is the failure mode to design for explicitly (a worker that receives work for a function it doesn't know about should surface that loudly, not silently drop it). ## Versioning registered functions Code changes while executions are in flight. A function registered today may need to resume an execution that a *previous* deploy started, and the two versions might not be interchangeable. The registry is where that's managed. TypeScript and Python both version the registry — name → version → function — and select the latest when the caller doesn't pin one (`Registry.latest` in both). The function version travels in the invocation param (below), so a resume can request the version that started the execution. Rust does not yet version the registry; its entries are name-only, and the source notes function versioning as not-yet-supported. If your language's deploys can overlap — and in production they will — design the registry to hold multiple versions of a name from the start, even if you select latest by default. Retrofitting versioning onto a name-only map is painful once executions depend on it. ## The invocation surface Invocation comes in two axes, and the developer-facing API exposes the cross product: - **Local vs remote.** A *local* call (`run` / `ctx.run` / `lfc`) runs the function as a durable step in the current execution. A *remote* call (`rpc` / `ctx.rpc` / `rfc`) dispatches it to a worker — possibly a different process, possibly a different service — and awaits the result through a durable promise. - **Await vs fire-and-forget.** The blocking forms (`run`, `rpc`) await the result. The `begin*` / handle-returning forms (`beginRun`, `beginRpc`, `begin_run`, `begin_rpc`) return a handle immediately so the caller can await later or not at all. A `detached` call goes further — it deliberately breaks lineage, spawning independent durable work that outlives the caller. All three SDKs land on the same vocabulary, varying only in idiom: TypeScript and Python expose `run`/`beginRun`/`rpc`/`beginRpc` plus `ctx.lfi`/`lfc`/`rfi`/`rfc` inside an execution; Rust uses builder structs (`resonate.run(...)`, `resonate.rpc(...)`) that you `.await` or `.spawn()`, and `ctx.run`/`ctx.rpc`/`ctx.detached` inside one. The distinction that matters at the protocol level is **whether the invocation creates a task**. `run` creates a task directly (the work is claimed locally), so it goes out as a `task.create`. `rpc` creates only a promise with a target tag and lets the server dispatch a task to whichever worker is listening — so it goes out as a `promise.create`. That tag is the whole mechanism. ## How an invocation becomes a targeted promise A remote invocation is a `promise.create` with a `resonate:target` tag. That tag is the trigger from chapter 5: a promise carrying it causes the server to create a task and push an execute message to the target address. No target, no task, no dispatch. The target value is an address resolved from the group or service name the developer named — both envelope SDKs run the bare name through a resolver (`target_resolver` in Rust, the network's `anycast`/`match` in TypeScript) to produce a `poll://any@group` address. What rides in the promise's `param.data` is the instruction for the remote worker — the function to run and the arguments to run it with. The interop core is `{func, args}`, and the SDKs add to it: | Field | TypeScript | Python | Rust | |---|---|---|---| | `func` | ✓ (name) | ✓ (name) | ✓ (name) | | `args` | ✓ | ✓ | ✓ | | `kwargs` | — | ✓ | — | | `version` | ✓ | ✓ | — | | `retry` | ✓ (encoded policy) | — | — | The portable subset every implementation can rely on is `{func, args}`; the rest are SDK-local extensions whose normativity is, per the spec, still an open question. If you want your SDK's invocations to be claimable by a worker built in another language, keep `{func, args}` clean and treat the extras as additive — and don't assume another SDK will honor your `version` or `retry` field. ## The convention tags, and a real divergence The invocation tags do more than route. They record the execution tree: which promise is the root of this whole execution, which is the direct parent, which branch this is. The server and SDK use them to thread resumption back to the right place. The TypeScript and Rust SDKs agree on the vocabulary they set on an invocation: | Tag | Role | Source | |---|---|---| | `resonate:target` | Delivery address — the dispatch trigger. | spec-reserved | | `resonate:origin` | The root promise of the whole execution tree. | spec-reserved | | `resonate:parent` | The direct parent promise. | spec-reserved | | `resonate:branch` | This execution branch (drives preload — chapter 8). | spec-reserved | | `resonate:scope` | `global` for remote, `local` for in-process steps. | SDK convention | The first four are in [the spec's reserved-tags table](/spec/execution-model/message-passing#reserved-tags) — the authoritative list, which also reserves `resonate:timer` (the durable-sleep tag from chapter 4) and `resonate:delay` (reserved for future use). `resonate:scope` is *not* a spec-reserved tag; it's a convention the SDKs use within the `resonate:` namespace to distinguish a local step from a remote one. Build the spec-reserved tags to the spec; treat `resonate:scope` as the implementation detail it is, and don't assume another SDK reads it. The Python SDK uses a *different* routing vocabulary: `resonate:invoke` where TypeScript and Rust use `resonate:target`, and `resonate:root` where they use `resonate:origin` — and it does not set `resonate:branch` at all. These are different routing primitives, not just different names, so a Python worker and a TypeScript worker in the same process group cannot dispatch to each other today. Which vocabulary the spec blesses as canonical, and when Python migrates, is an open question on the protocol. **Build to the `resonate:target` / `resonate:origin` consensus** (it is what the spec's reserved-tags table documents), and know the Python divergence exists if you are reading it as a reference or planning cross-SDK interop. ## Resolving a function on a fresh process Close the loop with the scenario this chapter opened on. A worker boots. It has a registry, populated at startup by the developer registering the same functions the previous process registered. The server pushes it an execute message for a task whose promise names function `chargeCard`, version 2. The worker looks up `("chargeCard", 2)` in its registry, gets a live function pointer, acquires the task, and runs it — replaying any steps already recorded (chapter 7). The original process is gone; the work continues, because the name in the promise and the name in the registry agree. This is why two disciplines from earlier chapters are load-bearing here: register every durable function at startup *before* you start receiving work, and keep names stable across deploys (version them rather than rename them). A name that resolves on the process that created an execution but not on the process that resumes it is the one bug that turns durable execution back into ordinary, mortal execution. Next: [replay and deterministic execution](/sdk/replay-and-determinism) — what actually happens when that resumed worker re-runs a function whose steps are already half-recorded. --- --- url: /sdk title: How to build a Resonate SDK --- # How to build a Resonate SDK There is a formal protocol specification with two complementary surfaces: [**resonate-specification**](https://github.com/resonatehq/resonate-specification) — the Resonate protocol as an executable abstract machine in Lean 4, the machine-checkable ground truth for handler and state-transition semantics — and the [**Distributed Async Await prose spec**](/spec), which defines the promise lifecycle, task model, and invariants a correct implementation must hold. There are three reference SDKs — [TypeScript](https://github.com/resonatehq/resonate-sdk-ts), [Python](https://github.com/resonatehq/resonate-sdk-py), and [Rust](https://github.com/resonatehq/resonate-sdk-rs) — that implement it. The layer between them is what this handbook is for: how you turn the protocol into a library that feels native to your language and that someone actually wants to use. It is written for the person building the next SDK. The normative protocol definition is the [Lean 4 executable abstract machine](https://github.com/resonatehq/resonate-specification) — the machine-checkable ground truth for the protocol's core handlers and state transitions. The [Distributed Async Await prose spec](/spec) documents the same protocol in a human-readable form: the promise lifecycle, the task and message model, the invariants a correct implementation must hold. This handbook assumes both as ground truth and teaches the layer above them — the design decisions, the idioms, and the traps that turn a wire-correct client into an idiomatic SDK. ## How the handbook is organized Fourteen chapters, in the order you need them. The arc runs from *why an SDK exists at all* to *what it takes to run one in production*. ## Who this is for You are comfortable in at least one language and you want to bring durable execution to it — because your community lives there, or because the runtime model is a better fit than the reference SDKs. You do not need to have read the server source. You do need to be willing to think about determinism, idempotency, and what happens when a process dies mid-flight. If instead you want to *use* Resonate — build agents, workflows, and services on an existing SDK — you want the [Resonate docs](https://docs.resonatehq.io), not this handbook. ## Have a question while you read [Echo](https://echo.resonatehq.io) is Resonate's docs assistant, and it knows these pages and the spec. If something here is unclear, ask it — it answers with citations back into the source. --- --- url: /sdk/introduction title: Why an SDK exists --- # Why an SDK exists A program that runs to completion on one machine, in one process, without ever being interrupted, does not need much from a library. You call a function, it returns, you move on. The standard library is enough. Durable execution starts from the opposite assumption: the process *will* be interrupted. It will crash, get redeployed, run out of memory, or simply be told to scale to zero while it waits on something slow. The work it was doing — the half-finished payment, the agent three tool-calls deep, the workflow sleeping for a week — has to survive that. And when a new process picks the work back up, it must continue as if nothing happened, not start over and not double-act. That guarantee is the entire reason a Resonate SDK exists. Everything you will build in this handbook is in service of making that guarantee feel like ordinary code. ## The job, stated plainly An SDK's job is to let a developer write a function that looks almost exactly like a normal function in their language — and have it become *durable* without them thinking about how. "Almost exactly" is doing a lot of work in that sentence, and most of this handbook lives inside it. This is the surface you are building for them — the experience that has to hold. The developer writes: ```text result = await someStep(input) ``` and what they get is a step whose result is recorded the first time it runs, so that if the process dies and another process resumes the function, that step is not run again — its recorded result is handed back instead. The function moves forward. It never moves backward. The developer should not have to know that a server recorded the result, that a promise was created and resolved, that a task was claimed and a lease was held. They wrote a function. It survived a crash. That is the experience you are building. ## Where the SDK sits It helps to be precise about the boundary, because it is the single most important architectural fact in this handbook. - **The server** is the source of truth. It holds the durable promises — the records of "this work was asked for" and, eventually, "this is how it came out." It hands out tasks, tracks leases, and enforces the invariants that make recovery safe. It does not run your code. - **The SDK** runs your code. It connects to the server, creates and reads promises, claims tasks, executes the developer's functions, records each step's outcome, and — crucially — knows how to *replay* a function from its recorded steps when a fresh process resumes it. The spec describes the server's side of this contract precisely: the wire envelope, the promise state machine, the task lifecycle, the invariants. You can read it as the set of bytes the server will accept and the guarantees it makes in return. The SDK is the other half of that contract, written in your language, for your developers. The [specification](/spec) tells you *what bytes to produce and what guarantees hold*. It is language-agnostic by design. This handbook teaches you *how to turn those bytes into a library that feels native* — which is necessarily language-specific, and is the part the spec cannot write for you. ## What "idiomatic" actually demands Wire-correctness is the floor, not the finish line. A client that produces exactly the right bytes is still only a client; the work that turns it into a library people reach for is making the durable function read like a normal function in the host language. In a language with `async`/`await`, durability should ride on `await`. In a language built on generators or coroutines, it should ride on `yield`. In a language with futures, it should compose with futures. The reference SDKs already span two quite different shapes of this — TypeScript and Python use a sync, generator-based model; Rust uses an async, future-based one — and a new SDK's first real design decision is which shape fits the host language's grain. Idiomatic also means the parts that *cannot* be hidden are surfaced honestly. Durable execution imposes three constraints on the developer's code, and a good SDK makes them legible rather than burying them: - **Determinism** — replayed code must take the same path it took the first time, or the recorded steps stop lining up. - **Idempotency** — a step may be *attempted* more than once across crashes; its externally visible effect must happen once. - **Activation lifetime** — a function may outlive the process that started it, so it cannot lean on in-process state surviving. You will meet each of these again, in code, in later chapters. For now the point is only this: an SDK is the thing that makes the easy parts disappear and the genuinely hard parts visible. Getting that split right is the craft. ## What comes next The next chapter, [the protocol at a glance](/sdk/protocol-at-a-glance), zooms out to the moving parts you will be implementing — promises, tasks, messages, and the loop that ties them together — so that when we start writing client code in [Talking to the server](/sdk/talking-to-the-server), the pieces already have names. You do not need to memorize the protocol to start. You need to hold one idea: **the developer wrote a function, and your job is to make it survive.** Everything else is detail in service of that. --- --- url: /sdk/local-mode title: Local mode for development --- # Local mode for development A developer's first encounter with your SDK should not require them to stand up a server. The whole engine — promise store, task dispatch, leases, the settlement chain — is a state machine, and a state machine can run in the same process as the worker driving it. **Local mode** is that: an in-process implementation of the server, swapped in behind the same interface the real network uses, so a developer can write a durable function and watch it survive crashes with nothing installed. This chapter is how to build it, what it can and cannot stand in for, and the one place where standing in for the real server quietly changes behavior — which, keeping with [surfacing the hard parts honestly](/sdk/introduction), you should surface rather than bury. ## The whole server, in a Map Local mode works because the server's contract is small and its state is plain data. The reference SDKs implement it as an in-process object holding the same state the real server holds and answering the same wire operations. TypeScript's `LocalNetwork` (`resonate-sdk-ts/src/network/local.ts`) embeds a `Server` whose entire state is a few maps — promises, tasks, schedules — plus timeout queues and an outgoing-message buffer. It implements the `Network` interface the real transport implements: `send` synchronously runs the request through the server's state machine and resolves with the response; `recv` registers a subscriber; and an `init` timer drives the clock (more on that below). Rust mirrors this almost line for line — its `LocalNetwork` wraps a `ServerState` behind an `Arc>` and is explicitly a port of the TypeScript `Server` (`resonate-sdk-rs/resonate/src/network.rs`). Python splits the two roles the TS server fuses: a `LocalStore` is the state machine, running on its own thread, and a `LocalMessageSource` is the push channel a worker reads from (`resonate-sdk-py/resonate/stores/local.py`, `resonate-sdk-py/resonate/message_sources/local.py`). The key design move in all three: **the same `Network`/store interface the HTTP transport implements is what local mode implements.** Nothing above the transport — not the worker loop, not the coroutine driver, not replay — knows or cares which is underneath. That is what makes local mode a faithful rehearsal rather than a separate code path: the SDK runs identically; only the bytes' destination changes. A real server has a wall clock advancing on its own. An in-process one doesn't — nothing makes a lease expire or a timer settle unless you tell it time has passed. The reference SDKs solve this two ways at once: an eager check that settles any past-due promise inline whenever the state machine is touched (so logical reads are always correct), plus a periodic tick — `LocalNetwork` fires `debug.tick` on a one-second interval (`resonate-sdk-ts/src/network/local.ts`; Rust's `ServerState::tick`) — to drive time-based transitions that no request would otherwise trigger. Without the tick, a durable sleep in local mode would never wake. ## What it emulates, and what it cannot Local mode is faithful to the parts of the contract that are *logic*: the full promise lifecycle (create, settle, idempotent re-create, timeout), the task state machine (pending → acquired → suspended → fulfilled), lease tracking, the [settlement chain](/sdk/suspend-resume-settlement) that resumes a suspended task when its awaited promise settles, and the preload that makes resumption inexpensive. A developer can write a workflow, kill the process mid-flight, restart, and watch it replay — the whole point — entirely in local mode. What it *cannot* emulate are the parts that are *distribution*, and being clear-eyed about these keeps you from testing a fiction: - **Multiple processes.** One in-process server is one process. You cannot exercise a task being claimed by a worker in a *different* process, which is the scenario the [version-and-lease machinery](/sdk/tasks-and-the-worker-loop) exists for. Local mode tests your recovery logic against a simulated crash-and-restart, not against a true concurrent claim. - **Persistence.** All state is in memory. Process exit is a clean slate — there is no on-disk durability to recover from, only in-process replay within a single run. - **Routing and load-balancing.** The real server routes anycast messages to one of many listening workers; the TS local server broadcasts every outgoing message to all subscribers, and Python's local store accepts exactly one connection. Multi-worker routing is not under test. - **Search, auth, and tracing.** The `*.search` operations return "not implemented" locally; token verification is a no-op; transport-level tracing is absent. - **Schedules, in Rust.** Rust's local server stores schedules but its tick does not fire them (its schedule stub reports a zero next-run time), so cron-driven creation is a TS/Python-only behavior locally. ## The emulation gap you have to name There is one place where local mode does not merely *omit* a distributed behavior but quietly *diverges* on a durability-relevant one, and — keeping faith with [surfacing the hard parts honestly](/sdk/introduction) — an SDK documents it loudly. Call it the zero-dependency-dev asterisk. **Leases are not refreshed in TypeScript and Rust local mode.** Against the real server, a worker holds a task's lease alive with a periodic heartbeat (`AsyncHeartbeat`, [heartbeat at TTL/2](/sdk/tasks-and-the-worker-loop)). In local mode the SDKs install a no-op heartbeat instead — TypeScript and Rust both route `LocalNetwork` to a `NoopHeartbeat` that sends nothing (`resonate.ts`, `resonate.rs`), so the lease is never renewed. There's a sharper edge in TypeScript specifically: its `AsyncHeartbeat` sends `task.heartbeat` with an *empty* task list (`resonate-sdk-ts/src/heartbeat.ts`), because against the real server liveness is looked up by process id — but the in-process server refreshes leases only for the tasks named in the request, so even if that heartbeat were wired to `LocalNetwork` it would refresh nothing. (Rust's `AsyncHeartbeat` does track its acquired tasks and would send them, but local mode installs the no-op heartbeat regardless.) The outcome is the same: in TS/Rust local mode, leases don't get renewed. The consequence bites a specific shape of code. A task is acquired; its step runs a side effect that takes *longer than the lease TTL* and has not yet recorded a durable result. The one-second tick fires, sees the lease expired, and releases the task back to pending — bumping it out from under the worker. The task is re-dispatched, the worker (now holding a stale version) is told to start over, and the still-running step **re-executes from the top**, side effect and all. With the default minute-long TTL it repeats every minute until the step finally checkpoints a durable promise. Steps that *have* already recorded a durable result are safe — replay fast-paths them — so the failure mode is precisely the un-checkpointed, side-effect-bearing, longer-than-TTL step. Two things to carry forward. First, **Python's local mode does not have this gap** — its local store knows its own tasks directly (no wire round-trip) and its heartbeat refreshes leases by scanning the connected process's tasks (`LocalTaskStore.heartbeat`), so leases hold. The gap is specific to the TS/Rust in-process servers. Second, whether the in-process heartbeat *should* be brought to parity with the real server is a tracked open question — don't quietly "fix" it as if settled. The right move for your SDK is to document the asterisk where developers will see it: local mode is for building and crash-testing your workflow's logic, not for trusting the timing behavior of a long, un-checkpointed side effect. ## Local mode is also your test harness The same in-process server that lets a developer build with nothing installed is what lets *you* test the SDK with nothing installed. The reference SDKs lean on it as the default fixture: TypeScript's `Resonate` auto-selects `LocalNetwork` when no server URL is configured, and unit tests construct it directly with a no-op heartbeat; Python parameterizes its test suite on `LocalStore` whenever no server host is present; Rust exercises `LocalNetwork` through its inline module tests. Because it exposes `debug.tick` (and `debug.reset`/`debug.snap` in TS), a test can advance time deterministically and assert on exact state — which is exactly the lever the next chapter builds on. Next: [testing your SDK against the spec](/sdk/testing-against-the-spec) — the server invariants your implementation must not violate, and how to find out whether it does. --- --- url: /sdk/production-concerns title: Production concerns --- # Production concerns You have built an engine that makes a function survive a crash. The last thing it has to survive is contact with production — and production is where the difference between a working prototype and a library people trust gets settled. This chapter is about the SDK *you* shipped running real workloads: what an operator needs to see when something goes wrong, what happens to in-flight work when you deploy a new version of a function, the levers that decide whether your SDK is fast or merely correct, and the handful of failure modes that will land in someone's logs at 3 a.m. The altitude here is deliberate — this is about running the SDK you wrote, not operating a Resonate cluster. The server's operations are its own story; yours is the library in the worker process. ## Observability: you can see less than you think The honest starting point is that the reference SDKs give you a foundation for observability, not a finished feature — and knowing exactly where the line falls saves you from promising your users something that isn't there yet. The TypeScript SDK carries an internal trace model (`resonate-sdk-ts/src/trace.ts`): every execution emits a sequence of structured lifecycle events — `run`, `rpc`, `spawn`, `block`, `await`, `resume`, `suspend`, `return`, `dedup` — with a set of well-formedness predicates that assert the sequence is sane. This is a genuinely useful spine. But it is an internal event model used mostly to validate the engine in tests; it is **not** OpenTelemetry, and there is no OTLP exporter wired to it. The same file's neighbor (`resonate-sdk-ts/src/network/http.ts`) carries a standing note that Prometheus-style metrics — request counts, failure counts, latency histograms — are specified but not yet implemented. Rust takes a different but parallel stance: it instruments the lifecycle with the `tracing` crate (`task acquired`, `starting execution`, `task fulfilled`, `task suspended`, errors with detail), which means an operator who installs a `tracing` subscriber gets structured logs — but the SDK ships no subscriber and no OTel bridge. Python has structured logging at the store boundary and no tracing layer beyond it. For your own SDK, that points two ways. Build the internal lifecycle-event spine — it costs little, and it is what every higher-level integration hangs off. And treat the OpenTelemetry span export and the metrics histograms as the real, unbuilt work they are, rather than letting your documentation imply a turnkey observability story the code doesn't back. An operator's first question when a workflow stalls is "where is it, and what is it waiting on" — the trace events answer it; make sure they can actually reach those events. ## Versioning functions while work is in flight This is the production concern most likely to surprise you, because it only shows up the first time you deploy a *change* to a function that has executions already running against the old code. The mechanism: when an execution invokes a function, the resolved function version is recorded in the task's parameters. When a worker later picks that task up — possibly a fresh worker after a deploy — it looks the function up *by name and version* in its registry. If the worker doesn't have that version registered, the task can't run. TypeScript and Python both build versioned registries for exactly this: a registry keyed by name *and* version, where you can register several versions of the same logical function and resolve "latest" or a specific one (`resonate-sdk-ts/src/registry.ts`, `resonate-sdk-py/resonate/registry.py`). Rust does not — its registry is keyed by name alone and rejects a duplicate registration outright (`resonate-sdk-rs/resonate/src/registry.rs`); the `version` it carries in options rides along for routing but never participates in dispatch. That divergence is worth stating plainly: **a graceful multi-version migration window is a TS/Python capability today; the Rust SDK has no version dimension in its registry.** The operational consequence is the same lesson in both worlds. If you deploy new workers that have only the new version registered, every in-flight task that recorded the old version will land on a worker that can't run it — and be dropped or released unrun, its promise left unresolved until it times out. The safe migration is the obvious one once you see the mechanism: keep the old version registered alongside the new one (run both, or register both in one worker) until every in-flight execution at the old version has settled, and only then retire the old registration. If you build a versioned registry, give your users that story explicitly; if you build a name-only one, be honest that a function's shape is effectively frozen while work referencing it is in flight. Watch for a spike in function-not-registered errors after a deploy. That is precisely the symptom of in-flight tasks at a version no live worker can serve. It is not a code bug to chase in the function body — it's a deploy-sequencing problem, and the fix is draining or co-registering the old version, not editing the function. ## Performance: the levers you actually hold A durable SDK's performance is dominated by how it talks to the server, and there are three levers worth understanding before you reach for cleverness. **Batching — the lever none of the reference SDKs pull yet.** Every promise and task operation today is its own request: TypeScript, Python, and Rust each issue one HTTP call per operation. A workflow that creates many small durable steps makes that many round-trips. There is local-work flushing inside the coroutine drivers, but no coalescing of *remote* operations into a batched call. This is both a caution — a chatty workflow pays per-step latency — and an opportunity: request batching is a real, unclaimed performance win for a new SDK, provided you preserve the per-operation semantics the engine depends on. **Connection reuse — make sure you have it.** Rust holds a single pooled `reqwest` client and reuses connections across calls; TypeScript rides the platform's pooled `fetch` and holds a persistent server-sent-events stream for inbound messages with bounded reconnect backoff. Python opens a session per call, so it reuses a connection only within a single retry loop, not across operations. Per-call connection setup is pure overhead on a hot path; pool and keep-alive by default. **Lease cadence — the one knob with a real tradeoff.** Heartbeat at half the lease TTL is the convention across all three SDKs (`ttl / 2`), and the tradeoff is direct: a shorter TTL recovers a dead worker's tasks faster but heartbeats more often, putting more load on the server; a longer TTL is lighter on the server but leaves a crashed worker's work stranded for longer. Pick the TTL for your workload's longest atomic step (so a healthy step never outruns its lease), and let the heartbeat follow at half of it. ## The failure modes that reach an operator Four failures account for most of what shows up in production logs, and an SDK author's job is to make each one legible rather than cryptic. - **Function not registered.** A task references a function name/version no worker has. It is dropped (TypeScript surfaces it as a registry error marked "will drop") or released to be retried by some other worker (Rust's `FunctionNotFound`), and if no worker has it, the promise hangs until timeout. As above, the usual root cause is a deploy, not the function. - **Version mismatch (409).** A worker tries to act on a task whose version has moved on — it lost its lease, another worker took over. The [worker loop chapter](/sdk/tasks-and-the-worker-loop) covers this: a `409` means *stop driving this execution*, not *retry harder*. Surface it as the benign-but-meaningful event it is, so an operator doesn't read normal recovery as an error storm. - **Auth failures.** A missing or rejected token comes back as a server error. The trap worth flagging: in at least the TypeScript taxonomy these are marked retriable and fold into the generic server-error path (`resonate-sdk-ts/src/exceptions.ts`), so an SDK that retries server errors indefinitely will hammer the server forever on a bad credential. Distinguish "retry might help" from "this will never succeed until a human fixes the token," and don't retry the latter into the ground. - **Encoding failures.** A value that can't be serialized (or a result that can't be deserialized) fails the step and is dropped. This is a developer-code problem the SDK should report with the offending step's identity, not swallow. Rust's error enum (`resonate-sdk-rs/resonate/src/error.rs`) and Python's store-error types (`resonate-sdk-py/resonate/errors/`) draw the same lines with different names. Whatever your language, the principle holds: name these failures distinctly, attach the execution and step they belong to, and make the retriable/terminal distinction impossible to get wrong — because the operator reading the log did not write your SDK, and the error message is the only documentation they have at that moment. ## What you actually built Step back from the production checklist and look at what the whole handbook added up to. You started with a function and a promise that it would survive a crash. You built the client that speaks to the server, the worker loop that holds a lease and never double-drives an execution, the replay that walks a function forward over its own recorded history, the suspend-and-resume that lets a wait cost nothing, and the host-language shape — generator or future — that makes all of it read like ordinary code. The production concerns in this chapter are not a separate subject; they are the same engine, seen from the operator's chair instead of the implementer's. The thing worth holding onto is the one the [first chapter](/sdk/introduction) opened with. A developer wrote a function, and it survived. Everything you built exists to make that sentence true and to keep it feeling unremarkable — and an SDK that earns that reaction, where durability is simply how functions behave and nobody has to think about the machine underneath, is the one that did its job. That is the bar. Build to it. --- --- url: /sdk/promise-lifecycle title: Promise lifecycle in code --- # Promise lifecycle in code The durable promise is the unit of truth (chapter 2), and this chapter turns it into code: the types that model it, the operations that move it through its lifecycle, and the one property — idempotency — that makes a distributed client safe. The authoritative definition of each handler is in the [resonate-specification](https://github.com/resonatehq/resonate-specification) — the Resonate protocol as an executable abstract machine in Lean 4. The [Durable Promise Specification](/spec/programming-model/durable-promise-specification) documents the broader promise contract including extensions not yet in the Lean model; both together are what this chapter implements. A promise has exactly two phases. It is **pending** until someone settles it, and then it is settled — permanently. The whole API divides along that line: a downstream side that creates and reads, an upstream side that settles. ## The value schema Before the operations, the data. A promise carries two values: a `param` (what it was created with) and a `value` (what it settled to). Both have the same shape, and getting that shape right early saves you from reworking every operation later. A `Value` is **`headers` plus `data`**: ```ts type Value = { headers?: Record; data?: string; }; ``` All three reference SDKs model this faithfully — `Value` in `resonate-sdk-ts/src/network/types.ts`, `DurablePromiseValue` in `resonate-sdk-py/resonate/models/durable_promise.py`, `Value` in `resonate-sdk-rs/resonate/src/types.rs`. The `headers` field is easy to skip on a first pass because the simplest examples never use it, and then you discover it is load-bearing: it is where content-type and encoding metadata ride alongside the payload, which is what lets the [codec layer](/sdk/encoding-and-codecs) know how to decode `data` on the way back out. Model `headers` from the start; do not bolt it on. The `data` itself is a string — the already-encoded payload. The promise layer does not know or care what is inside it; serialization happens one layer up, before the value reaches the wire. Keep that boundary clean: promises move opaque strings, codecs give them meaning. ## The terminal states A promise has one non-terminal state and four terminal ones. Your SDK models all five, because each one means something different to code that awaits the promise: | State | What it means to an awaiter | |---|---| | `pending` | Not settled yet. Keep waiting. | | `resolved` | Success. Resume with `value`. | | `rejected` | Failure. Resume by raising `value` as an error. | | `rejected_canceled` | Settled by an explicit cancel. A deliberate failure. | | `rejected_timedout` | The promise's timeout elapsed before anyone settled it. | The reference SDKs all carry the full set — the `state` union in `resonate-sdk-ts/src/network/types.ts`, the `PromiseState` enum in `resonate-sdk-rs/resonate/src/types.rs`, the `Literal[...]` in Python's `durable_promise.py`. The one that catches implementers is `rejected_timedout`. A promise is not silently abandoned when its deadline passes; the server transitions it to `rejected_timedout`, a *real terminal state*, and everything awaiting it resumes with that failure. If you model timeout as "still pending, but stale," your replay logic (chapter 7) will hang waiting for a settlement that already, in effect, happened. The envelope SDKs send these states lowercase (`rejected_timedout`); Python's older REST protocol uses uppercase (`REJECTED_TIMEDOUT`). Pick one representation for your internal types and normalize at the wire boundary — don't let the server's string casing leak into the rest of your SDK. ## Creating a promise `promise.create` is the downstream operation: it brings a pending promise into existence. The request carries the id, an absolute timeout, the `param` value, and the tags: ```ts // resonate-sdk-ts/src/promises.ts — Promises.create { kind: "promise.create", head: { corrId, version }, data: { id, timeoutAt, param: { headers, data }, tags }, } ``` Rust's `Promises::create` (`resonate-sdk-rs/resonate/src/promises.rs`) sends the same envelope with a `PromiseCreateReq { id, timeout_at, param, tags }`. Two fields deserve attention: - **`timeoutAt` is absolute, not a duration.** It is a Unix-epoch timestamp in milliseconds — the wall-clock moment the promise expires — not "30 seconds from now." Your SDK computes it from the developer's requested duration and the parent's deadline (a child never outlives its parent), then sends the absolute value. All three SDKs do this conversion at the context layer and put an absolute millisecond timestamp on the wire. The reference default when the developer specifies nothing is generous — 24 hours in both TypeScript and Rust. - **`tags` is where routing lives.** A bare promise created with no `resonate:target` tag is just a value that someone will settle by other means. Add `resonate:target`, and the server creates a *task* and pushes it to a worker (chapters 5 and 6). The tag is the difference between "a promise" and "a promise someone is dispatched to fulfill." ## Settling a promise At the wire level there is **one** settle operation: `promise.settle`, parameterized by the target state. The Lean spec's `PromiseSettleReq` carries `id`, `state`, and `value` — no resolve/reject/cancel split. The three SDK verbs are thin wrappers that each hard-code one target state before delegating to the same wire call: - **`resolve`** → `promise.settle` with state `resolved`, signaling success. - **`reject`** → `promise.settle` with state `rejected`, signaling failure. - **`cancel`** → `promise.settle` with state `rejectedCanceled`, signaling deliberate abandonment. Both envelope SDKs collapse these into one internal call. TypeScript's `Promises.resolve/reject/cancel` all delegate to a private `settle()` that sends `promise.settle` with the target state (`resonate-sdk-ts/src/promises.ts`); Rust's `resolve/reject/cancel` all delegate to a private `settle()` taking a `SettleState` enum (`resonate-sdk-rs/resonate/src/promises.rs`). Mirror that: one settle path, three thin public verbs. It keeps the value-encoding and error-handling in one place instead of three. Note what is *not* on this list: there is no "un-settle." A terminal promise is terminal. An attempt to settle an already-settled promise does not error blindly — it runs into the idempotency rules below, which is the whole point. ## Idempotency: why every write is safe to retry Here is the problem that idempotency solves, and it is the central problem of any distributed client. You send `promise.create`. The network drops the *response*. Did the create happen? You cannot know. You have to retry — and the retry must not create a second promise or corrupt the first. The protocol gives you two independent layers of protection, and a complete SDK understands both even when it leans mostly on the first: **The promise id is the primary idempotency key.** Promise ids are caller-chosen and meaningful, not random. Creating a promise with an id that already exists is a safe no-op: the server returns the *existing* promise, unchanged. So the retry above is safe by construction — same id, same promise, the second create just reads back the first. Both envelope SDKs rely on this directly: in TypeScript's local server model, `promise.create` for an existing id returns the existing record with `200` (`resonate-sdk-ts/src/network/local.ts`). This is why deterministic id generation (chapter 7) is not a convenience — it is the foundation of idempotency. **Idempotency keys give finer control, per the legacy durable-promise contract.** The Lean `resonate-specification` models `PromiseCreateReq` with only `id`, `timeoutAt`, `param`, and `tags` — no idempotency-key fields appear in the formal abstract machine. The finer-grained key model — `ikc` (idempotency key for *create*) and `iku` (idempotency key for *complete*) plus a `strict` flag — is defined in the [legacy Durable Promise Specification](/spec/programming-model/durable-promise-specification#state-transitions) and its [324-row state table](/spec/programming-model/durable-promise-specification#state-transitions), and is implemented by the reference server beyond what the Lean model captures. The Python SDK threads these explicitly — `ikey` and `ikey_for_complete` carried as request headers, matched in the store's transition logic (`resonate-sdk-py/resonate/stores/local.py`). The envelope SDKs currently lean on the id-as-key model and do not expose `ikc`/`iku` on their public surface. Support the key fields as a server-side capability you pass through; treat the id-as-key guarantee as the portable baseline. The asymmetry to internalize: **idempotency keys protect against duplicate *state*.** A promise's final value is what matters, so a safely-retried settle that lands the same value is harmless. This is the opposite of how tasks behave — a task's *version* protects against duplicate *progress* (chapter 5) — and the two mechanisms are deliberately different because they guard different things. `strict` (defined in the legacy Durable Promise Specification's transition table, and exposed by the Python SDK) tightens settlement: with `strict=false`, settling a promise that already timed out quietly returns its `rejected_timedout` state; with `strict=true`, the same attempt errors, so a caller that *needs* to know its settlement actually took effect can find out. The `strict` field does not appear in the Lean `resonate-specification` — it is a server-layer behavior beyond the current formal model. Whether `strict` belongs on an end-user surface at all is one of the open questions — treat it as a protocol-level capability to support, not necessarily a knob to put in front of every developer. ## Timeout and the timer tag Timeout is the one settlement that no caller performs — the server does it when `timeoutAt` passes and the promise is still pending, transitioning it to `rejected_timedout`. There is one deliberate exception, and it is how durable sleep is built: a promise tagged `resonate:timer` resolves on timeout instead of rejecting. Create a promise with that tag and a `timeoutAt` five minutes out, await it, and you have a durable five-minute sleep — one that survives a crash, because the deadline lives on the server, not in a process's timer. Both envelope SDKs set `resonate:timer` on their sleep promises (`resonate-sdk-ts/src/context.ts`, `resonate-sdk-rs/resonate/src/context.rs`) and flip the timeout state accordingly in their local server model. TypeScript and Rust use `resonate:timer` for this; the Python SDK uses `resonate:timeout`. They are not interoperable — a worker built against one name will not recognize the other's sleep promises. The spec's reserved-tags table names `resonate:timer`; build to that, and know the divergence exists if you are reading Python as a reference. The convention-tag mismatch across SDKs is a tracked open question, not a settled choice. ## Where SDK state ends and server truth begins The discipline that ties this chapter together: the server is the source of truth, and your in-memory promise objects are a cache of it. When you create a promise, you hold a `PromiseRecord`, but its authoritative state can change underneath you — it can be resolved by another execution, or timed out by the server, while your copy still says `pending`. Never treat your local record as final. A settlement, a read, or a resumption (chapter 8) is what tells you the real state. Build your promise types as snapshots with a known-as-of quality, not as the truth itself — and let the next chapters, on tasks and the worker loop, show how the server keeps that truth consistent across crashing, restarting workers. Next: [tasks and the worker loop](/sdk/tasks-and-the-worker-loop) — the claim on a promise, and the engine that drives it. --- --- url: /sdk/protocol-at-a-glance title: The protocol at a glance --- # The protocol at a glance The previous chapter made one promise: the developer writes a function, and your SDK makes it survive a crash. This chapter names the machinery that delivers on that — the four moving parts you will be implementing and the loop that ties them together. Nothing here is the full story; every piece gets its own chapter later. The goal is only that when we start writing client code, the words already mean something. The executable definition of every handler this chapter summarizes lives in [github.com/resonatehq/resonate-specification](https://github.com/resonatehq/resonate-specification) — the Resonate protocol as an abstract machine in Lean 4. When you need to verify what the server does in an edge case, that is the ground truth. There are four parts: **promises**, **tasks**, **messages**, and the **worker loop**. Hold them loosely for now. ## The server is a message bus, not a database The first thing to get right is what the server *is*, because it is easy to picture it as a key-value store for promises and then write a client that polls it. That client will work and it will be slow and it will miss the point. The server is a **message bus** that happens to durably remember things. It holds the promises, yes — but its active job is to deliver work to your workers and to wake them back up when the thing they were waiting on is ready. When a promise is created with a delivery address, the server *sends your worker a message* telling it to start executing. When that work finishes and something else was awaiting it, the server *sends another message* telling the awaiting worker to resume. Your SDK spends most of its life receiving those messages and acting on them. So the mental model is not "client calls server." It is "two parties exchange messages, and one of them never forgets." Keep that framing; it explains nearly every design decision that follows. ## Promises: the unit of truth A [durable promise](/spec/programming-model/durable-promise-specification) is a record of a future value that lives in the server's storage rather than in a process's memory. It is `pending` until someone settles it, and then it is settled forever — `resolved` with a value, or `rejected` with an error. That permanence is the whole trick. Because the outcome of a piece of work is written down in a place that outlives any single process, a fresh process can ask "did this already happen?" and get a truthful answer. Every durable step the developer writes becomes a promise; replay (chapter 7) is nothing more than reading those promises back instead of re-running the work. A promise has a small set of terminal states, and your SDK has to handle all of them: | State | Meaning | |---|---| | `pending` | Not yet settled. The value isn't available. | | `resolved` | Completed successfully, carrying a value. | | `rejected` | Completed with a failure. | | `rejected_canceled` | Settled by an explicit cancel. | | `rejected_timedout` | The promise's timeout elapsed before it settled. | The last one catches people: a promise that times out is not left pending: it transitions to a terminal `rejected_timedout`, and anything awaiting it resumes with that failure. (A promise tagged `resonate:timer` is the deliberate exception — it resolves on timeout instead, which is how durable sleeps are built.) Promises in code, including the value schema and the idempotency keys that make creation safe to retry, are [chapter 4](/sdk/promise-lifecycle). ## Tasks: the claim on a promise A promise is a value waiting to exist. A [task](/spec/system-model/tasks) is the *responsibility for producing it*. The two share an identifier and are created together, but they are distinct objects with distinct jobs: the promise owns the value, the task owns the claim. Tasks exist only when work has to be delivered to a worker — concretely, when a promise carries a `resonate:target` tag naming a delivery address. That tag is the trigger: create a promise with it and the server creates a task and enqueues an execute message to the address. Create a promise without it and you get a bare promise that someone has to resolve by other means. A task carries two things your SDK must respect: - A **version** — an optimistic-concurrency token. Every operation that mutates a task must present the version it expects. If the task has moved on without you, the server answers `409` and you re-fetch and reconsider. This is what makes it safe for a crashed worker's task to be handed to a successor: the original worker, if it comes back, presents a stale version and loses. - A **lease** — a deadline by which the worker must prove it is still alive, via heartbeat. Miss the deadline and the server releases the task back to `pending` (bumping the version) so another worker can claim it. Heartbeat is per-process, not per-task: one signal refreshes every task a worker holds. Versions and leases together are the recovery mechanism — they are how the protocol lets a function outlive the process that started it without ever letting two processes drive it at once. Tasks, the worker loop, and lease management get built in [chapter 5](/sdk/tasks-and-the-worker-loop). ## Messages: the wire envelope Everything the SDK and server say to each other rides a single uniform [envelope](/spec/execution-model/message-passing), the same shape over whatever transport you choose: ```text { kind: "promise.create", // which operation head: { corrId, version }, // correlation id, protocol version data: { ... } // operation-specific payload } ``` The response echoes the `kind` and the `corrId` — so a client multiplexing many in-flight requests over one connection can pair each reply to its request — and carries a `status` in its head. The status codes are a small HTTP-shaped set; two of them are load-bearing in ways worth flagging now: - **`409` Conflict** — a version mismatch, an invalid state transition, or a failed fence. Not an error to retry blindly; a signal that your view of the world is stale. - **`300` Continue** — the fast-path answer to `task.suspend`: the promise you were about to wait on has *already* settled, so don't suspend, just continue. You will meet this in [chapter 8](/sdk/suspend-resume-settlement); it is the difference between a snappy SDK and one that takes a network round-trip to notice work is already done. The two canonical messages flowing *to* your worker are **Invoke** ("start executing this") and **Resume** ("the thing you awaited is ready"). Those are the spec's names for the interaction; on the envelope wire the SDKs carry both as a single message kind, `execute`, and tell a fresh start from a resume by what's *in* it rather than by the kind (chapter 5 builds this). Recognizing and dispatching those messages is the heart of the worker loop. ## The worker loop, end to end Here is the whole protocol in one breath, with everything above in its place: 1. A worker connects to the server and waits for messages on a delivery address. 2. The server delivers an **Invoke** for a pending task. The worker claims the task with `task.acquire`, presenting the version. 3. The worker runs the developer's function. Each durable step creates a promise; the worker waits for the server's reply before treating the step as done. 4. When the function awaits something not yet ready, the worker tells the server `task.suspend` and stops holding the task active — it does not block a thread waiting. The server parks the task and remembers to wake it. 5. When the awaited promise settles, the server sends a **Resume**. A worker — maybe the same one, maybe not — picks the task back up and continues the function from where it left off, replaying the already-recorded steps rather than re-running them. 6. When the function returns, the worker settles its own promise with `task.fulfill`, and anything awaiting *it* gets its own Resume. That loop, made durable and idiomatic, is the entire job. Chapters 3 through 6 build it piece by piece; chapters 7 through 9 handle the genuinely hard parts hiding inside step 5. Every envelope's `head.version` carries the protocol version the request is built against, as a date-stamped string (the current reference SDKs target `2026-04-01`). The server may reject a version it does not support with `400`. The version changes only when the wire contract changes in a breaking way — the promise-API verbs, the envelope shape, and the status taxonomy in this chapter are stable across SDK releases. Pin the version your SDK targets in one constant and send it on every request; don't scatter the literal through your code. Treat the value as something you negotiate against your target server rather than a number you hard-code and forget. ## What you now have names for Five words that the rest of the handbook leans on: **promise** (the durable value), **task** (the claim on it, with its version and lease), **message** (the enveloped request/response), **Invoke / Resume** (the two messages that drive a worker, carried on the wire as `execute`), and **suspend / fulfill** (how a worker yields and completes). Next: [Talking to the server](/sdk/talking-to-the-server) — the connection, the transport, and the first real requests over the wire. --- --- url: /sdk/replay-and-determinism title: Replay and deterministic execution --- # Replay and deterministic execution This is the chapter the rest of the handbook has been building toward. Everything so far — promises, tasks, the worker loop, the registry — exists to make this one thing possible: a function that crashed halfway through can be re-run from the top by a fresh process and *not redo the work it already did*. That mechanism is **replay**, and it is the heart of durable execution. The idea is almost suspiciously simple. To resume a function, the SDK runs it again — from the beginning. As it runs, each durable step asks the server "has this step already happened?" For steps that ran before the crash, the answer is yes, and the recorded result is handed straight back without re-executing. For steps that hadn't run yet, the answer is no, and they execute for real. The function walks forward over its own history until it reaches the point it died, then keeps going into new territory. That simplicity rests on one demanding requirement, and most of this chapter is about earning it: the second run has to line up *exactly* with the first. ## Replay is re-execution, not a journal scan It's worth being precise about what replay is, because there's a tempting wrong model. The SDK does not load a journal of past steps and skip ahead. It *re-executes the function* — actually calls it again — and the re-execution naturally skips completed work because each completed step resolves instantly from its recorded promise. The "has this happened?" question is just `promise.create`. Recall from chapter 4 that creating a promise with an id that already exists returns the existing promise. So when the re-running function reaches its third step and tries to create the promise for it, one of two things happens: - **The promise exists and is settled** — this step ran before the crash. The create returns the recorded value, and the SDK feeds it straight back into the function as that step's result. No work is redone. - **The promise doesn't exist, or is still pending** — this step is new, or was in flight. The SDK runs it for real. All three reference SDKs implement exactly this. In TypeScript and Python, a `promise.create` that comes back already-settled feeds its stored value into the generator's next `.next()`/`.send()` call (`Coroutine.exec` in `resonate-sdk-ts/src/coroutine.ts`, `Computation` in `resonate-sdk-py/resonate/scheduler.py`). In Rust, the already-settled promise resolves the awaited future immediately from a preload cache (`Effects::create_promise` in `resonate-sdk-rs/resonate/src/effects.rs`). The shape differs; the mechanism is identical — recorded promises stand in for re-execution. A naive replay would round-trip to the server for every already-settled step — slow, for a function with a long history. The protocol avoids it with **preload**: when a worker acquires or resumes a task, the server hands back the already-settled promises in this execution's branch, and the SDK serves the replay from that cache instead of re-fetching. The envelope SDKs receive a `preload` array on acquire; you'll see it again in chapter 8, where it's what makes resumption fast. ## Deterministic ids: how a step finds its own past Here is the crux. Replay works by matching each step to the promise it created last time. The match is by **promise id**. So the second run has to generate the *same id* for the *same step* as the first run did — or step three of the replay will look up step three's id, find nothing, and re-execute work that already happened. The SDKs guarantee this by deriving child ids deterministically from the execution's structure, not from anything random or external. The scheme is a per-execution counter appended to the parent id: the durable steps come out as `parent.0`, `parent.1`, `parent.2`, … in the order the function reaches them (TypeScript and Rust count from 0, Python from 1 — the base doesn't matter, the determinism does). Same function, same path, same sequence of ids — every time. - TypeScript: an `InnerContext.seq` field, formatted as `` `${this.id}.${this.seq}` `` and incremented per step (`resonate-sdk-ts/src/context.ts`). - Python: a `_counter` producing `` f"{self._id}.{self._counter}" `` (`resonate-sdk-py/resonate/resonate.py`). - Rust: an `AtomicU32` seq formatted the same way (`resonate-sdk-rs/resonate/src/context.rs`). This is why deterministic id generation isn't a detail — it's the load-bearing wall. The counter resets to zero/one on each run, so the *nth* durable step always gets the *nth* id, and replay can find it. Which leads directly to the one rule your developers have to follow. ## The determinism contract The counter only lines up if the function takes **the same path** on replay — reaches the same durable steps, in the same order. That is the determinism contract, and it's the one constraint durable execution genuinely imposes on the code developers write. Concretely: the orchestrating code — the part *between* durable steps — must be a pure function of the recorded results. If a function branches on `Math.random()`, or the current time, or a value it read directly from a database, then on replay it might branch the other way, take a different step as its "second" step, generate a different id, and desynchronize from its own history. The recorded promises stop lining up, and durability silently breaks. The resolution is the rule that makes the whole model usable: **anything nondeterministic must itself be a durable step.** Don't call `random()` in the orchestration; call it *through* a durable step, so its result is recorded the first time and replayed every time after. The reference SDKs provide exactly these wrapped primitives: - **Time** — TypeScript's `ctx.date.now()` and Python's `ctx.time.time()` are durable local steps: the timestamp is recorded once and returned identically on replay (`resonate-sdk-ts/src/context.ts`, `resonate-sdk-py/resonate/resonate.py`). - **Randomness** — TypeScript's `ctx.math.random()` and Python's `ctx.random` are likewise durable steps. - **Sleep** — `ctx.sleep` is a durable timer promise (the `resonate:timer` tag from chapter 4): it suspends durably and, on replay, returns instantly because the timer promise is already settled. Rust takes a narrower stance — it does not ship `ctx.now()` or `ctx.random()` wrappers, leaving the developer to route nondeterminism through `ctx.run` explicitly, and uses `ctx.sleep` for durable delays. The principle is the same in every language: **nondeterminism enters only through a recorded step.** The general-purpose tool is the local durable call — `ctx.run` / `ctx.lfc` — which records *any* computation's result as a promise so replay can skip it. That's the escape hatch you give developers for reading a database, calling an API, or anything else the orchestration must not redo. ## Local steps are durable too It's worth making explicit, because it surprises people: a *local* step — a function that runs in-process, no remote dispatch — is still backed by a durable promise. `ctx.run` in all three SDKs creates a promise tagged `resonate:scope: "local"`, executes the function, and settles the promise with its result (`resonate-sdk-ts/src/context.ts`, `resonate-sdk-py/resonate/scheduler.py`, `resonate-sdk-rs/resonate/src/context.rs`). The locality is about *where it runs*, not *whether it's recorded*. This is what lets replay skip a local computation just as cleanly as a remote one — the recorded promise is authoritative either way. The flip side, which your SDK should make legible: a step that is *not* durable (an SDK may allow opting out for lightweight, side-effect-free work) is not replay-safe. It re-runs on every replay. That's a fine, deliberate optimization for a pure helper, and a quiet bug if a developer reaches for it around a side effect. Make the durable path the default and the non-durable path the conscious choice. ## A note on detection You may have noticed what's missing: none of the reference SDKs actively *detect* a determinism violation. There's no runtime guard that fires when the second run diverges from the first. The protection is purely structural — get the ids to line up and replay is correct; break the contract and replay silently maps steps to the wrong promises. This is a deliberate simplicity, and it shifts weight onto two things your SDK *can* control: making the deterministic path the easy, default path (wrapped time/random/sleep, durable-by-default steps), and documenting the contract clearly. A developer who routes all nondeterminism through durable steps never has to think about replay at all — which is exactly the experience chapter 1 promised. Surfacing violations is a quality-of-life feature you can add later; getting the structure right is the requirement. Next: [suspend, resume, and the settlement chain](/sdk/suspend-resume-settlement) — what happens at the boundary where a function stops running and waits, and how the value it was waiting for finds its way back in. --- --- url: /sdk/suspend-resume-settlement title: Suspend, resume, and the settlement chain --- # Suspend, resume, and the settlement chain A durable function spends most of its life *not running*. It starts, does a little work, and then awaits something — a remote call, a timer, a sibling execution — that won't be ready for milliseconds or months. This chapter is about that boundary: how a function stops without blocking anything, and how the value it was waiting for finds its way back in and starts it running again. The thing to hold onto: when a durable function awaits an unsettled promise, the worker does **not** sit on a thread waiting. It tells the server "I'm waiting on this," lets go of the execution entirely, and goes and does other work. The server remembers, and pushes the function back to a worker when the awaited promise settles. The wait costs nothing while it lasts — no thread, no held connection, just state the server keeps — which is what makes a week-long sleep or a paused human-approval step no more expensive to run than a fast one. ## Suspending: handing the wait to the server When the worker drives a function to the point where it awaits an unsettled promise, it issues `task.suspend`. This does two things atomically: it registers a callback — "resume *this* execution when *that* promise settles" — and it parks the task in the `suspended` state. The worker is now free. A function often awaits more than one thing at once (a parallel fan-out, an "await all of these"). The clean way to handle that is to register *all* the awaited promises in one operation, and both envelope SDKs do exactly that: a single `task.suspend` carrying an `actions` array of callback registrations, one per awaited promise (`Core.suspendTask` in `resonate-sdk-ts/src/core.ts`, `Core::suspend_task` in `resonate-sdk-rs/resonate/src/core.rs`). One round-trip, N callbacks, no window where some are registered and some aren't. This is a place the SDKs genuinely differ. TypeScript and Rust suspend *atomically* — one `task.suspend` registers every awaited promise's callback together. The Python SDK *sequences* it — a separate callback registration per awaited promise, short-circuiting as soon as one comes back already-settled. Both reach a correct resting state, but the atomic form is the one to build: it's a single round-trip, and it has no intermediate state to reason about if the worker dies mid-suspend. Whether atomic multi-callback suspend is the canonical form is itself a tracked open question; the envelope SDKs treat it as the default. ## The already-settled fast path: status 300 There's a race built into suspending. Between the worker deciding to suspend and the server processing it, the awaited promise might *already have settled* — the remote work was fast, or it finished while the suspend request was in flight. If the worker blindly suspended, it would register a callback on an already-settled promise and then wait for a resume message that has to make a full network round-trip to arrive. Slow, for something that's already done. The protocol closes this with the **`300` Continue** status. When the worker tries to suspend on a promise the server finds already settled, the server doesn't park the task — it answers `300`, meaning "don't suspend, the thing you're waiting on is ready, keep going." The worker drops straight back into the execution loop and continues, replaying the now-settled value (chapter 7) instead of waiting for it. Both envelope SDKs implement this directly. TypeScript's `suspendTask` treats a `300` response as "continue" and immediately re-enters `executeUntilBlocked` with the preloaded promises (`resonate-sdk-ts/src/core.ts`); Rust's `suspend_task` returns a `Redirect` variant carrying the preload and loops again (`resonate-sdk-rs/resonate/src/core.rs`). Python reaches the same outcome a different way — its sequential callback registration returns a resume immediately when the promise is already settled, a client-side equivalent of the fast path. The result is identical: work that's already done never costs a suspend/resume round-trip. The fast path is easy to skip on a first implementation — suspend always, resume always, it's correct. But a workflow that awaits ten quick steps in a row would eat ten full resume round-trips it didn't need. Implementing `300` turns those into in-loop continuations. It's the difference between an SDK that's correct and one that's also fast on the common case. ## The settlement chain: settle → resume → execute Now the other side. A promise settles — some upstream execution called `resolve` (chapter 4). What turns that settlement into a suspended function waking up? The **settlement chain**, and it runs entirely through the server: 1. **Settle.** The server records the promise as `resolved` (or `rejected`). 2. **Fire callbacks.** The server looks at the callbacks registered on that promise — the ones put there by suspending workers — and for each, transitions the parked task back to `pending` and enqueues a message to the awaiter's address. 3. **Execute / resume.** A worker — maybe the original, maybe a fresh one — receives that message, re-acquires the task, and re-runs the function. The awaited promise is now settled, so replay delivers its value at the point the function was waiting, and execution continues forward. The value doesn't travel *in* the resume message as the thing the function receives directly. The message just says "this task is runnable again." The worker re-acquires, and the settled value reaches the function through the ordinary replay path — `promise.create` for that step returns the now-settled record. Resumption and replay are the same machinery; resumption is just replay triggered by a settlement. The envelope SDKs use one message kind, `execute`, for both the *first* dispatch of a task and its *resumption* after a settlement — the worker can't tell "start" from "resume" by the message kind, and doesn't need to, because replay handles both identically. A separate `unblock` message exists for external *listeners* — addresses registered to be notified when a promise settles, such as a caller holding a handle and awaiting a result from outside the execution. The Python SDK names these differently — `invoke`, `resume`, and `notify` as three distinct messages. Same chain, different vocabulary; if you're reading across SDKs, map `execute`↔`invoke`/`resume` and `unblock`↔`notify`. ## Exactly-once resumption The contract that makes the chain trustworthy is that each registered callback fires **exactly once** when its promise settles. Not zero times (the function would hang forever), not twice (it would resume two copies of the same execution). The server enforces it structurally. When a promise settles and its callbacks fire, the server **clears the callback set in the same atomic step** that enqueues the resume messages (`Server.triggerCallbacks` in the local server model, `resonate-sdk-ts/src/network/local.ts`; Python clears its callbacks dict on settlement in `resonate-sdk-py/resonate/stores/local.py`). A second settlement event — there shouldn't be one, promises are terminal — finds no callbacks to fire. And registering a callback on an *already-settled* promise is a no-op that returns the settled state immediately rather than storing a callback that could fire later. That no-op is the same condition that produces the `300` fast path above: the two are the same invariant seen from two directions. This is worth stating carefully, because the imprecise version — "at-most-once resumption" — undersells it and invites the wrong mental model. The guarantee is **exactly-once consumption of a registered callback**: once you've successfully suspended on a pending promise, you *will* be resumed when it settles, and you will be resumed *once*. Build your SDK to lean on that — don't add defensive de-duplication on top of resume messages as if they might double-fire; the callback-consumption contract already prevents it, and layering your own dedup on top usually just hides a bug in how you registered the callback in the first place. ## Preload: resuming without re-fetching One efficiency closes the loop with chapter 7. The `preload` field appears on four response types in the Lean spec — `TaskAcquireRes`, `TaskCreateRes`, `TaskFenceRes`, and `TaskSuspendRes` — declared as `preload : List PromiseRecord := []`. No handler in the formal spec ever populates it with a non-empty value; the spec is silent on how or when a server should fill preload. What the reference SDKs do is an implementation strategy the server may apply: when a worker re-acquires a task to resume it, the server *can* return the settled promises from this execution's branch alongside the task, and the SDK loads them into a replay cache (`buildEffects` in `resonate-sdk-ts/src/util.ts`, `Effects::new` in `resonate-sdk-rs/resonate/src/effects.rs`; Python passes the settled leaf promise directly into its resume command). The `300` fast path carries a preload field too, used the same way. This optimization — when the server chooses to populate it — keeps resumption from getting slower as an execution's history grows; but because the Lean spec leaves the population semantics unspecified, your SDK should function correctly whether preload is empty or not, fetching settled promises on demand as the fallback. ## The shape of it Step back and the whole chapter is one loop, drawn around the wait: - A function awaits an unsettled promise → the worker `task.suspend`s (atomically, over all awaited promises) and lets go. - Unless the promise was already settled → `300`, and the worker just continues. - The promise settles → the server fires the callback exactly once, parks-to-pending, and dispatches a resume. - A worker re-acquires, replays over the preloaded history (if the server populates it; otherwise fetches on demand), and the function continues from exactly where it waited. That loop, plus the determinism of chapter 7, is the complete durable-execution engine. A function can suspend a thousand times across a thousand process lifetimes, and each resumption lands it back exactly where it was, with exactly the values it had. Everything from here — retries, codecs, local mode, conformance testing, production concerns — is refinement on top of an engine that, at this point, already works. Next: [coroutines in your language](/sdk/coroutines) — the concrete mechanics of suspending and resuming a function in your host runtime, where the generator-versus-future split finally gets its own chapter. --- --- url: /sdk/talking-to-the-server title: Talking to the server --- # Talking to the server This is where the client starts. Before promises, before tasks, before any of the durable machinery, an SDK needs one thing: a way to say something to the server and a way to hear something back. This chapter builds that layer — the transport — and nothing above it. The transport has two jobs that are worth separating in your mind from the start, because they have opposite shapes: - **Send** — the SDK initiates. You build a request, hand it to the server, and wait for the response. Classic request/response. - **Receive** — the *server* initiates. It pushes you a message — "start this task," "resume this one" — at a time of its choosing, because the work that unblocks you finished on some other worker you've never heard of. A client that only knows how to send is a client that has to poll, and a client that polls is slow and noticed. Getting receive right — a long-lived channel the server can push down — is most of the work in this chapter. ## The transport abstraction Both envelope-protocol reference SDKs put a single interface at the boundary so the rest of the SDK never knows which transport is underneath. In TypeScript it is the `Network` interface (`resonate-sdk-ts/src/network/network.ts`): ```ts export interface Network { readonly unicast: string; readonly anycast: string; match(target: string): string; init(): Promise; stop(): Promise; send: Send; // build a request, get a typed response recv: Recv; // register a callback for pushed messages } ``` Rust draws the same line with the `Network` trait (`resonate-sdk-rs/resonate/src/network.rs`), with an `async fn send` and a `recv` that takes a callback. The two methods that matter are right there: `send` for the requests you originate, `recv` for the messages the server pushes. Everything else — promises, tasks, the worker loop — is written against this interface, so the same SDK runs against a real server, an in-memory [local server](/sdk/local-mode), or a future transport you haven't written yet. Put the transport behind an interface on day one, even if you only ever implement one. The [local-mode](/sdk/local-mode) story, your test suite, and any future transport all depend on the rest of the SDK never reaching past this seam to touch an HTTP client directly. Both reference SDKs that speak the envelope ship at least two implementations of it. ## The request envelope Every request the SDK sends is the uniform envelope from [the message-passing spec](/spec/execution-model/message-passing) — a `kind`, a `head`, and operation-specific `data`: ```ts { kind: "promise.create", head: { corrId, version }, data: { id, timeoutAt, param, tags }, } ``` Two fields in the head are load-bearing and easy to get wrong: - **`corrId`** — a correlation id, unique per request. The server echoes it back unchanged in the response. This is what lets you multiplex many in-flight requests over one connection and still pair each reply to the request that asked for it. TypeScript generates it with `randomUUID()`; Rust uses a timestamped string. The value doesn't matter; the uniqueness and the echo-check do. Both SDKs reject a response whose `corrId` doesn't match the request they're waiting on. - **`version`** — the protocol version, a date-stamped string. Both envelope SDKs pin it in one constant (`VERSION` in `resonate-sdk-ts/src/util.ts`, `PROTOCOL_VERSION` in `resonate-sdk-rs/resonate/src/lib.rs`), currently `2026-04-01`, and stamp it on every request. The server may answer `400` to a version it doesn't speak. The response comes back with the same `kind` and `corrId`, plus a `status` in its head. Build your send path so that a non-success status is a first-class outcome, not an exception you forgot to catch — `409` in particular (covered in chapters 4 and 5) is a routine, expected answer that your higher layers need to reason about, not a crash. ## The poll transport: receiving work over SSE HTTP send is the easy half — POST the envelope, await the response. The interesting half is receive, and the protocol's portable answer is the **poll transport**: the worker holds open a long-lived HTTP `GET` to the server, and the server streams messages down it as [Server-Sent Events](/spec/execution-model/message-passing#poll-address). The worker never exposes a port; it makes an outbound connection and listens. Each message arrives as an SSE frame — a line `data: {json}` terminated by a blank line. The receive loop's whole job is: hold the connection, read frames, parse each `data:` payload as a message, and hand it to the registered callback. In TypeScript, `PollMessageSource` (`resonate-sdk-ts/src/network/http.ts`) wraps the standard `EventSource` and fans each parsed frame out to the `recv` callbacks. In Rust, `HttpNetwork` (`resonate-sdk-rs/resonate/src/http_network.rs`) spawns a task that reads the response byte-stream and scans it for `data:`-prefixed lines, because there is no built-in `EventSource` to lean on. The pattern to copy, in any language: 1. Open a `GET` to the poll endpoint for your group and process id. 2. Read the stream line by line. On a `data:` line, parse the remainder as a message and dispatch it. 3. On disconnect, reconnect with backoff — the connection *will* drop, and a worker that doesn't reconnect silently stops receiving work. Reconnection is not optional polish. Both SDKs back off exponentially on a dropped connection — TypeScript caps around 30 seconds, Rust around 60 — and reset the backoff the moment a connection succeeds. The server holds your task's lease only as long as you heartbeat (chapter 5); a worker that drops its poll connection and never comes back has its in-flight tasks released to other workers, which is correct. A worker that *flaps* — reconnecting in a tight loop with no backoff — is hammering the server during exactly the moments it is least healthy, which is not. ## Addresses: where messages are delivered For the server to push you work, it has to know where "you" is. That is an **address**, and for the poll transport it takes the form: ```text poll://cast@group[/id] ``` The `cast` is the delivery mode — `uni` for unicast (this specific worker) or `any` for anycast (any worker in the group). The `group` is the logical pool workers register under; the optional `id` is a specific worker within it. Both SDKs derive two addresses for themselves on startup — a unicast `poll://uni@group/pid` and an anycast `poll://any@group/pid` — from the group and process id they were configured with (`resonate-sdk-ts/src/network/http.ts`, `resonate-sdk-rs/resonate/src/http_network.rs`). The two modes do different jobs, and your SDK surfaces both: - **Anycast** (`any@group`) is how work gets distributed. A promise targeting `poll://any@workers` is delivered to *one* worker in the `workers` group — the server picks. This is what makes a worker pool a pool: spin up ten, they share the load. - **Unicast** (`uni@group/worker-1`) pins delivery to one specific worker. This matters for resumption — when a suspended execution is resumed, the server can prefer the worker that has relevant state warm, falling back to anycast if that worker is gone. An address with both parts (`poll://any@my-service/abc123`) does exactly this: prefer `abc123`, accept anyone. When the developer invokes a function "on the `workers` group," your SDK turns that group name into a target address — both SDKs have a small resolver that wraps a bare target into `poll://any@target` (`target_resolver` in Rust, `match` in TypeScript). That target string is what rides on the promise's `resonate:target` tag, which — as the next chapters show — is the thing that makes the server create a task and push it to a worker at all. ## A note on Python, and on async Two of the reference SDKs — TypeScript and Rust — speak the `{kind, head, data}` envelope this chapter describes. The Python SDK currently speaks an older REST protocol: separate endpoints (`POST /promises`, `PATCH /promises/{id}`), idempotency and `strict` carried as HTTP headers, and a different message vocabulary on the poll stream. If you are reading Python source as a reference, read it for its [execution model](/sdk/coroutines), not its wire format — the envelope is the shape to build against, and whether Python converges on it is an open protocol question the spec itself flags. This is also the first place the host language's concurrency model asserts itself. Rust's transport is `async fn` end to end — sends are awaited futures, the receive loop is a spawned task on the async runtime. TypeScript awaits `fetch` for sends and takes `EventSource` callbacks for receives on the single-threaded event loop. Python blocks: a dedicated daemon thread holds the SSE `GET` and reads it line by line. None of these is more correct than the others — each is the idiomatic way to hold a long-lived connection in that language. Build the receive loop the way a networking library in your language would, not the way another SDK did it. With a transport that can send envelopes and receive pushed messages, you can make the first real requests. Next: [the promise lifecycle in code](/sdk/promise-lifecycle) — creating, reading, and settling durable promises, and the idempotency that makes every write safe to retry. --- --- url: /sdk/tasks-and-the-worker-loop title: Tasks and the worker loop --- # Tasks and the worker loop A promise is a value waiting to exist. A [task](/spec/system-model/tasks) is the responsibility for producing it — and this chapter builds the engine that takes on that responsibility: the worker loop. It is the most important loop in the SDK, because it is where durability stops being a data model and becomes a running system that survives crashes. The hard requirement the loop has to meet, stated once: a function can outlive the process that started it, and yet *no two processes may ever drive the same execution at the same time*. Those two facts are in tension — to be durable, work must be re-claimable by a new process; to be correct, it must never be doubly-claimed. The protocol resolves the tension with two mechanisms, and most of this chapter is learning to respect them: the **version** and the **lease**. ## What a task is, and when one exists A task and its promise share an id and are created together, but they are different objects with different jobs: the promise owns the value, the task owns the claim on producing it. Critically, **a task exists only when work must be delivered to a worker** — that is, when the promise carries a `resonate:target` tag. Create a promise without a target and there is no task; nobody is dispatched. Create one with a target and the server creates a task in `pending` and enqueues an execute message to the address. A task moves through a small lifecycle — `pending` (claimable), `acquired` (a worker is on it), `suspended` (the worker is waiting on other promises), and the terminal `fulfilled`. (There is also `halted`, an administrative state a task enters via `task.halt` and leaves via `task.continue`, which returns it to `pending`, claimable again — so it is *not* terminal, and not part of the normal loop.) Your loop drives the ordinary transitions. ## Claiming a task When an execute message arrives over the transport (chapter 3), the worker claims the task with `task.acquire`, presenting the task's **version**: ```ts // resonate-sdk-ts/src/core.ts — Core.onMessage { kind: "task.acquire", head: { corrId, version }, data: { id: task.id, version: task.version, pid, ttl }, } ``` Rust's `Core::on_message` (`resonate-sdk-rs/resonate/src/core.rs`) does the same: it receives `(task_id, version)` from the transport and calls `Sender::task_acquire` with the id, version, the worker's process id, and a TTL. A successful acquire transitions the task `pending → acquired` and hands back the root promise (and any preloaded promises — chapter 8). Now you hold the claim, and the clock on your lease starts. There is a second way a task is born already-claimed: when *this* worker is the one invoking, `task.create` creates the task directly in `acquired`, skipping the round-trip — you created the work, so you already hold it. The `resonate:target` tag rule is nuanced: a fresh `task.create` whose promise does not yet exist succeeds whether or not the tag is present — the server creates both the promise and the acquired task unconditionally (the `| none =>` branch in `spec/02-actions/T-02-task.create.lean`). The `resonate:target` check fires only when the promise *already exists*: if the existing promise lacks the tag, the server returns `422`. In practice your SDK should always include a target tag on dispatched work — but a missing tag is not a pre-flight rejection on a fresh create. Both paths converge on the same next step: run the function until it blocks or finishes. ## The version: optimistic concurrency The version is an optimistic-concurrency-control token, and it is the entire answer to "how do we let a crashed worker's task be re-claimed without two workers fighting over it." Every mutating task operation must present the version the worker believes is current. The rule the server enforces: - The version **advances on each fresh `task.acquire`** — a re-claim always hands back a higher version than the previous holder saw. It does *not* change while the task simply sits in `pending` waiting to be re-offered; the increment lands on the *next* acquire, not the moment the task returns to `pending`. (The spec diagram reads as though the bump happens the instant a lease lapses; the shipped server does it on re-acquire — see [testing against the spec](/sdk/testing-against-the-spec).) - A mutating operation with a **stale version gets `409` Conflict.** The task has moved on without you. Walk the crash through it. Worker A acquires a task at version 3 and starts working. A's process freezes (GC pause, network partition, it doesn't matter). A's lease expires; the server returns the task to `pending`, still version 3 and claimable. Worker B acquires it — and *that acquire* advances it to version 4. B runs. Now A unfreezes, finishes, and tries to fulfill at version 3 — and the server answers `409`, because the live version is 4. A's work is rejected. There was never a window where both A and B could commit; the version closed it. This is why a `409` is not an error to retry blindly — it is the system *correctly* telling you that you are no longer the one driving this execution. Handle it by stopping, not by retrying harder. A promise carries idempotency keys; a task carries a version. They are different on purpose. Idempotency keys make a repeated write *converge* — settle the same promise twice, fine, same value. A version makes a repeated claim *fail* — only one version is live at a time. Promises protect against duplicate **state**; tasks protect against duplicate **progress**. Retrying a task operation with the same version is not idempotent: it works once, then the next operation bumps the version and your old one is dead. ## The lease and the heartbeat The version protects correctness; the **lease** is what makes recovery *happen*. An acquired task carries a lease — a deadline by which the worker must prove it is still alive. Miss the deadline and the server returns the task to `pending`, free for another worker to acquire — and that next acquire advances the version, leaving your stale claim behind. Prove liveness and you keep the claim. Liveness is proven by **heartbeat**. A worker holding forty tasks does not send forty separate heartbeat requests; it sends one `task.heartbeat` that lists every task it wants to keep alive, and the server refreshes the lease on each listed task. Two important clarifications from the Lean spec (`spec/02-actions/T-05-task.heartbeat.lean`): first, the server refreshes **only the tasks explicitly named in `req.tasks`** — it does not do a "find all tasks for this pid" lookup. Every task the worker wants to keep must be in the request list. Second, `pid` in the heartbeat request is used as a **per-task validation guard**, not a lookup key: for each listed task, the server checks `t.pid == some req.pid` and silently skips any task whose state, version, or pid does not match — returning `200` regardless. A task omitted from the list, or listed with the wrong version, will not have its lease refreshed and will expire normally. Both SDKs that manage leases this way run a single timer: TypeScript's `AsyncHeartbeat` (`resonate-sdk-ts/src/heartbeat.ts`) fires one `task.heartbeat` on an interval; Rust's `AsyncHeartbeat` (`resonate-sdk-rs/resonate/src/heartbeat.rs`) does the same, tracking the set of active `(id, version)` pairs and sending them together. Cadence is a tradeoff your SDK picks: heartbeat too rarely and a transient network blip costs you the lease; too often and you flood the server. The reference convention is to heartbeat at **half the lease interval** — enough slack to absorb one missed beat. The actual numbers diverge across the SDKs today, which is worth knowing: | SDK | Default lease (TTL) | Heartbeat cadence | |---|---|---| | TypeScript | 60 s | ~30 s (TTL/2) | | Rust | 60 s | ~30 s (TTL/2) | | Python | 10 s | ~5 s (TTL/2) | The lease is not a request timeout — it is "how long the server waits for a sign of life before assuming you died." Set it comfortably longer than the longest stretch of work your SDK does *between* heartbeats. The default TTL is itself an open spec question (the SDKs disagree, 60 s vs 10 s); whatever you choose, make sure a single durable step can't routinely outrun a heartbeat, or healthy work will get its lease pulled out from under it. ## Fencing a side effect The version makes *committing to the server* safe — a stale worker can't fulfill. But what about effects the server can't see? If your function charges a credit card and then crashes before recording it, a re-claiming worker will charge it again. The protocol's answer for "I am about to do something the server can't undo, and I need to be certain I'm the unique current claimant" is **`task.fence`**: a conditional operation that succeeds only if the task is still acquired at the presented version. Fence before you externalize an irreversible effect, and a worker that has silently lost its lease finds out *before* it acts, not after. It is the version check applied to the dangerous moment. (The idempotency that makes most steps safe to replay is chapter 7; fence is the tool for the steps that can't be made idempotent.) ## The loop, end to end Put it together and the worker loop is: 1. **Receive** an execute message (chapter 3). 2. **Acquire** the task with `task.acquire`, presenting the version; start the heartbeat if it isn't already running. 3. **Run** the developer's function. Each durable step creates a promise and waits for the server's reply. 4. **If the function blocks** on something not yet ready, tell the server `task.suspend` and stop driving it — don't hold a thread spinning. The server parks the task and will send a resume when the awaited promise settles (chapter 8). 5. **If the function returns,** settle its promise and complete the task in one atomic `task.fulfill`. Both envelope SDKs send a single `task.fulfill` carrying an embedded `promise.settle` action (`resonate-sdk-ts/src/core.ts`, `resonate-sdk-rs/resonate/src/core.rs`) — one operation, so there is no window where the promise is settled but the task isn't. 6. **If anything goes wrong** before completion, `task.release` (or just let the lease lapse) so another worker can take over. TypeScript releases explicitly on an execution error (`Core.releaseTask`); Rust does the same in its error path. Releasing with your version lets the server reject a release that's already stale. ## Structuring the loop for your runtime Step 4 is where your language's concurrency model decides the shape of everything. The loop must never block a thread waiting on a durable promise — those waits can be *days* long. So the loop's spine is: drive a function until it yields a dependency, hand that dependency to the server, and free the executor to do other work until a resume comes back. How you "drive until it yields" is the host language's call, and the reference SDKs split two ways here — a split you'll meet head-on in chapters 7 through 9: - **TypeScript and Python drive a generator.** The developer's function is a generator that `yield`s a description of each dependency; the SDK steps it with `.next()` / `.send()`, gets the next dependency, and parks. Re-entry resumes the same generator. - **Rust drives a future.** The developer's function is an `async fn`; the SDK `.await`s it, and a dependency that isn't ready surfaces as a signal the SDK collects, suspends on, and re-runs from on resume. Neither is more correct; each is how durable suspension naturally expresses itself in that language's runtime. What's invariant across both is the contract with the server — acquire with a version, heartbeat to hold the lease, suspend rather than block, fulfill atomically, release on failure. Build *that* faithfully, and the loop is durable no matter which runtime shape you chose. Next: [the function registry and invocation surface](/sdk/function-registry) — how the developer's functions get names the server can dispatch to, and how an invocation becomes a targeted promise. --- --- url: /sdk/testing-against-the-spec title: Testing your SDK against the spec --- # Testing your SDK against the spec A durable execution SDK has an unusually demanding correctness bar: a bug doesn't just produce a wrong answer, it can silently double-execute a side effect or lose work across a crash — failures that don't reproduce on a happy-path run and don't show up until production hits the exact interleaving that triggers them. Ordinary unit tests don't reach this. This chapter is about the testing that does: anchoring on the protocol's named invariants, replaying scenarios for conformance, injecting the faults the model is supposed to survive, and — the lesson that outranks all of them — checking your assumptions against the *shipped* server rather than the prose about it. ## The server invariants are your test oracle The protocol's [task model](/sdk/tasks-and-the-worker-loop) names a set of invariants the server maintains over every task at all times. They are the most useful thing to test against, because they are precise, they are total (they hold after *every* operation, not just at the end), and they translate directly into assertions. The [specification's task model](/spec/system-model/tasks) states seven: 1. **Every task has a corresponding promise** — no orphan tasks. 2. **Every pending task has a retry timeout** — so a task that's claimable but unclaimed will eventually be re-offered. 3. **Every acquired task has a lease** — so a worker that dies is detected. 4. **Every suspended task has at least one callback** registered on a promise it awaits — so it can be woken. 5. **No suspended task has an already-consumed callback** — a settled promise can't leave a task parked forever. 6. **No suspended task has a timeout** — suspension is open-ended; it waits on a settlement, not a clock. 7. **No fulfilled task has a timeout** — a terminal task holds nothing. The way to use these is to drive your SDK through a sequence of operations, snapshot the server state after *each* one, and assert all seven hold on every snapshot. A violation pins the bug to the operation that produced it. This is exactly how the reference conformance tooling works — it replays operation sequences and checks an invariant set after every step — and it's the shape your own conformance suite should take. Several of these invariants carry *formal identifier names in the spec source* that state the *opposite* of the rule — a `no-timeout`-style name guarding "must have a timeout," and vice versa. (The plain descriptions above are right-way-round; it's the spec's symbol names that invert.) If you write your assertions from the names alone you will get them inverted and your tests will pass on broken behavior. Assert the *described* condition. This is a small thing that has bitten reviewers, which is the whole reason it's worth a callout. ## Verify against the shipped server, not the prose Here is the lesson that matters most, learned the hard way: **the specification text and even a separate spec repository can drift from what the server actually does, and the shipped server is the tiebreaker.** When an invariant or wire detail is load-bearing for your test, confirm it against the server's source — the state machine in `resonate/src/oracle.rs` (the `op_task_*` and `op_promise_*` handlers) — not against a prose restatement that may lag the code. A concrete example sits inside invariant 3. The spec's diagram implies a task's version increments *at the moment its lease expires* and it returns to pending. The shipped oracle instead carries the task back to pending at its current version and increments on the *next acquire* (`op_task_acquire` in `resonate/src/oracle.rs`). The end guarantee — that no two workers ever hold the same live version, so a stale worker's write is rejected — holds either way; but a conformance test that asserts "version incremented the instant the lease lapsed" will fail against the real server even though the server is correct. The fix is not to loosen the test; it's to write the assertion against what the server actually guarantees (the version *will* have advanced before anyone re-acquires) and to treat a mismatch between prose and code as a documentation bug to flag, not a server bug to work around. A second example cuts the other way, and it's the trap in its purest form. The TypeScript in-process [local server](/sdk/local-mode) rejects a task-create whose promise lacks a `resonate:target` tag — but the shipped oracle does *not* (`op_task_create` in `resonate/src/oracle.rs` only validates the tag's address format *when the tag is present*; a tag-absent create passes). This behavior is now independently verifiable against the public Lean spec: `spec/02-actions/T-02-task.create.lean` in [`resonatehq/resonate-specification`](https://github.com/resonatehq/resonate-specification) confirms that a fresh `task.create` (promise does not yet exist, the `| none =>` branch) does not check for `resonate:target` at all. A conformance test written against the convenient local model would assert a rejection the real server and the Lean spec both never make, and fail an SDK that is actually correct. The model is not the oracle. Verify against the server — and when a detail is load-bearing, cross-check against the Lean spec as the formal ground truth. The strongest version of this discipline is *differential* testing: run the same operation sequence against your SDK-driven [local server model](/sdk/local-mode) and against a live shipped server, snapshot both, and diff. Any divergence is either a bug in your implementation or a drift in your model — both worth knowing. The reference tooling does exactly this, running an in-process model and a real server side by side under identical seeds. ## Replaying recorded scenarios Conformance, concretely, is a corpus of operation sequences — `promise.create`, `task.acquire`, `task.suspend`, settlements, timeouts — replayed against your implementation with the invariant check after each step. Two flavors are worth building: - **Hand-written transition scenarios** that target specific tricky paths: the [`300`-continue fast path](/sdk/suspend-resume-settlement), a lease expiring mid-step, an idempotent re-create, a suspend that races a settlement. Each is a small, named, deterministic sequence whose expected end state you assert exactly. - **Generated sequences** — a fuzzer that emits random-but-valid operation streams against a seed and checks the invariants on every snapshot. This is the closest thing to property-based testing the protocol invites: *for all valid operation sequences, the invariants hold.* It catches the interleavings you wouldn't think to write by hand. A reusable, cross-SDK conformance corpus — the same recorded scenarios every implementation runs — is the natural endpoint of this chapter. Treat the *idea* as the target to build toward; don't assume a public, drop-in suite you can point your CI at today. Until one is published, your conformance corpus is something you grow alongside your SDK, anchored on the invariants above and the shipped server as oracle. ## Injecting the faults the model is supposed to survive Durability is a claim about behavior *under failure*, so the tests that matter most cause failures on purpose. The reference SDKs take two complementary approaches, both worth borrowing. **Deterministic simulation.** Python's simulator (`resonate-sdk-py/resonate/simulator.py`) runs the whole system — a server and one or more workers — as a discrete-event loop over a seeded random source and a step-advanced clock. It drops a fraction of messages, delivers them out of order, shuffles each component's inbox, and occasionally removes a worker entirely to model a crash. Because everything derives from the seed, a failing run is perfectly reproducible: capture the seed, replay the exact interleaving, debug it. The assertion is invariance of the *outcome* — across all the chaos, a workflow's final result must be the one correct result. This is the highest-value test you can write for a durable SDK, because it exercises the recovery machinery against the adversary it was built for. **Targeted stubs.** Rust's test utilities (`resonate-sdk-rs/resonate/src/test_utils.rs`) take a finer-grained route: a `StubNetwork` that can be told to return a `300` redirect on suspend (exercising the already-settled fast path), to settle a promise out from under a running execution (modeling an external completion mid-step), and that records every request it received so a test can assert exact wire-format correctness. Where the simulator proves the system survives chaos, the stubs prove individual branches — the suspend-races-settlement path, the redirect path — are taken correctly. Neither reference SDK uses a formal property-testing framework (no QuickCheck/Hypothesis/proptest with shrinking); the seeded simulator and the fuzzer fill that role. If your language has good property-testing tooling, pointing it at determinism and idempotency — *re-running a step yields the same recorded result; replaying a completed prefix changes nothing* — is a worthwhile addition, not a replacement for simulation. ## Building confidence before you ship Stack these and you have a real correctness story: invariant assertions after every operation, hand-written scenarios for the tricky transitions, a seeded fuzzer for the ones you didn't think of, fault injection for the recovery paths, and a differential check against the shipped server to catch drift in your own model. The throughline is the one this chapter opened on — **trust the running server over any description of it.** An SDK that passes its own tests against a model that has drifted from the real server passes nothing that matters. Anchor on the invariants, verify against `oracle.rs` when a detail is load-bearing, and let the shipped server have the last word. Next: [production concerns](/sdk/production-concerns) — what running the SDK you built actually demands, from observability to versioning functions while work is in flight. --- --- url: /sdk/time-retries-policies title: Time, retries, and policies --- # Time, retries, and policies Three things a durable function needs that the core engine doesn't give it for free: the ability to *wait* for a stretch of wall-clock time and survive a crash during the wait, the ability to start work *on a schedule* with no process running at the moment it fires, and the ability to *retry* a step that failed without the developer hand-rolling a loop. None of these is a new primitive. Each is built from promises and tasks you already have — which is the point of this chapter. Once you see how, you also see where each one's behavior is settled and where it is still genuinely open across the reference SDKs. ## Durable sleep is a timer promise A durable sleep has to be more than `setTimeout`. If the process dies during a week-long wait, the wait must survive and resume; and on [replay](/sdk/replay-and-determinism) a completed sleep must return instantly rather than sleeping again. Both fall out of building sleep as a promise. `ctx.sleep` creates a promise tagged `resonate:timer`, with a `timeoutAt` set to the wake time and *no* `resonate:target` tag — so, per the [task model](/sdk/tasks-and-the-worker-loop), no task is dispatched. Nothing executes; the promise sits with a deadline. When that deadline passes, the server settles the promise on its own. The tag is what tells it how: ```rust // resonate-sdk-rs/resonate/src/context.rs — sleep_create_req tags.insert("resonate:timer".into(), "true".into()); // no resonate:target → no task, just a deadline ``` On the server side, an expiring promise is resolved or rejected based on that tag (`timeout_state` in `resonate/src/oracle.rs`): a `resonate:timer` promise settles as **resolved** when its deadline passes (the sleep elapsed, normally), whereas an ordinary promise that hits its timeout settles as **rejected/timed-out** (it ran out of time, an error). One field distinguishes "the wait finished" from "the work expired." TypeScript builds the identical request in `sleepCreateOpts` (`resonate-sdk-ts/src/context.ts`), and both cap the timer's `timeoutAt` at the parent execution's own timeout so a sleep can never outlive the function that started it. On replay the mechanics are the ones you already built: the timer promise is already settled, so re-creating it returns the settled record immediately and the function walks past the sleep without waiting. A week-long sleep costs nothing while it runs and nothing to replay. TypeScript and Rust tag the sleep promise `resonate:timer`, and the shipped server keys its resolve-vs-reject decision on exactly that tag. The Python SDK, which still speaks the [older wire protocol](/sdk/talking-to-the-server), tags its sleep `resonate:timeout` instead (`resonate-sdk-py/resonate/conventions/sleep.py`), matched by its local store. If you are reading across SDKs, don't take Python's tag as the wire contract — write your timer to the `resonate:timer` tag the current server understands, and treat the Python name as an artifact of the protocol it hasn't migrated off yet. ## Scheduling without a live process Sleep waits *inside* a running function. Scheduling is the other case: start a function on a recurring cadence when there may be no process awake at all when it fires. The answer is to make the *server* the thing that creates the work. `schedule.create` registers a cron expression plus a template for the promise to spawn each time it fires (`Schedules.create` in `resonate-sdk-ts/src/schedules.ts`): ```ts await resonate.schedules.create( scheduleId, "0 * * * *", // cron — hourly promiseIdTemplate, // server substitutes {{.id}} / {{.timestamp}} promiseTimeout, { promiseData, promiseTags }, // tags carry resonate:target to route a worker ); ``` The server stores the schedule with a computed next-run time. When the clock reaches it, the server materializes a promise from the template — and if the template's tags include a `resonate:target`, it creates a task and enqueues the execute message, exactly as if a worker had created the promise itself (`op_schedule_create` and the tick path in `resonate/src/oracle.rs`). Then it advances the next-run time. No process needs to be awake at the firing instant; a worker only needs to be listening to *pick up* the task once it exists. Scheduling, in other words, is promise-creation moved server-side and put on a timer. ## Retry policies, and where they run A durable step can fail. A retry policy decides whether and when to run it again — and unlike sleep and scheduling, this is an area where the reference SDKs genuinely diverge, so it's worth separating what's settled from what's open. **The policies themselves.** TypeScript and Python both ship four: `Constant`, `Linear`, `Exponential`, and `Never` (`resonate-sdk-ts/src/retries.ts`; `resonate-sdk-py/resonate/retry_policies/`). Each computes a delay from an attempt number, and `Never` declines to retry at all. `Exponential` is the substantive one — delay = `min(base * factor^attempt, maxDelay)` — and the two SDKs agree on its defaults: base 1s, factor 2, max delay 30s, attempts effectively unbounded. **Neither adds jitter** to the backoff in any policy; if you want jitter in your SDK, that is a design choice you are adding, not a convention you are matching. **The defaults.** Both TS and Python pick the policy *by the shape of the function*: a generator/coroutine function defaults to `Never`, and an ordinary function defaults to `Exponential` with the values above (`lfi`/`lfc` in `resonate-sdk-ts/src/context.ts`; `Options.retry_policy` in `resonate-sdk-py/resonate/options.py`). The reasoning is sound — a coroutine is itself made of durable steps that each retry, so retrying the whole orchestration would double up — and the two SDKs agree exactly. Two open edges here. First, **Rust has no application-level retry policy at all**: its `Options` struct (`resonate-sdk-rs/resonate/src/options.rs`) carries tags, target, timeout, and version — no retry field, no policy enum, no SDK-level retry loop. A step that fails in the Rust SDK is not retried by the SDK. Second, even the TS/Python agreement (`Never` for generators, `Exponential(1s, ×2, 30s)` otherwise) is a *convention the two implementations share*, not something the specification pins as normative. Document the consensus, note that Rust is an outlier by omission, and treat "what the default must be" as unresolved rather than inventing an answer. **Where the retry actually runs** is itself a three-way split worth knowing, because it changes the shape of the bytes on the wire: - **TypeScript** encodes the policy into the call's parameters: a remote invocation puts `retry: opts.retryPolicy?.encode()` inside `param.data` (`rfi`/`rfc` in `resonate-sdk-ts/src/context.ts`), so the policy travels with the task to whatever worker picks it up. - **Python** keeps the policy in the scheduler and applies it process-side: on a failed step the scheduler computes the next delay and re-enqueues the work as a `Delayed(Retry(...), delay)` (`resonate-sdk-py/resonate/scheduler.py`). The policy never leaves the process; the param carries no retry field. Python also guards the deadline — it won't schedule a retry whose delay would push the attempt past the promise's timeout. - **Rust** transmits no retry field at all, consistent with having no policy system. This is the [param-shape divergence](/sdk/function-registry) showing up again: the same logical concept ("how should this be retried") lives in the wire payload for one SDK, in process memory for another, and nowhere for the third. When you build yours, decide deliberately which, and make the choice legible. ## Virtualizing time for tests One last time-related capability, and it sits half in the protocol and half in test infrastructure. Testing a week-long sleep or a daily schedule can't mean waiting a week. The server therefore supports *advancing its notion of now* on demand. The TypeScript SDK exposes this as a `resonate:debug_time` request header and a `debug.tick` operation (`resonate-sdk-ts/src/network/types.ts`): a client can stamp a request with a synthetic "now," or tick the server's clock forward, and the server processes every promise timeout, lease expiry, retry, and schedule firing that falls at or before that time (`op_debug_tick` and `resolve_time` in `resonate/src/oracle.rs`). The in-process [local network](/sdk/local-mode) leans on exactly this — it drives `debug.tick` on an interval to make timeouts fire without a real wall clock. `debug.tick` and `resonate:debug_time` are real operations on the server's wire API, so they are protocol-normative in the sense that the server implements them. But only the TypeScript SDK exposes them in its client types. Python has a `StepClock` (`resonate-sdk-py/resonate/clocks/step.py`) used by its deterministic simulator — an in-process test clock, *not* a wire concept — and Rust surfaces neither. Frame time virtualization as testing infrastructure, and treat "must a conforming SDK expose `debug_time` injection?" as an open question rather than a settled requirement. Next: [encoding and codecs](/sdk/encoding-and-codecs) — how a step's arguments and results become the encoded strings a promise actually carries, and what the headers alongside them are for. --- --- url: /server/abstract-machine title: The abstract machine: ServerState, effects, and handlers --- # The abstract machine: ServerState, effects, and handlers The normative specification for a conformant DAA server is not a document — it is a running Lean 4 program: [`resonatehq/resonate-specification`](https://github.com/resonatehq/resonate-specification). Everything in this track is anchored on that code. This chapter explains the machine's shape so you can read the source directly and trust what it says. ## The state The machine holds one value: ```lean structure ServerState where config : ServerConfig promises : List PromiseObject tasks : List TaskObject schedules : List Schedule promiseTimeouts : List PromiseTimeout taskTimeouts : List TaskTimeout scheduleTimeouts : List ScheduleTimeout outbox : List OutboxEntry ``` Those eight fields are the entire server state — every object the server knows about lives in one of them. There is no connection pool, no clock, no filesystem reference. `ServerState` is a plain inductive value; two states are equal iff their fields are equal. This makes the spec executable as a pure function and mechanically comparable against any implementation you build. `ServerConfig` carries one field today: `retryTimeout : Nat := 5000` (milliseconds). Everything else — auth, transport, indexes — is outside the machine. ## The monad The machine's computation type is: ```lean abbrev M := StateM ServerState ``` `StateM ServerState` threads a `ServerState` through every operation without any IO. A handler runs; the state may change; the handler returns a result and the new state. Nothing else can happen inside `M`. This is the source of determinism. ## Effects: the only way to touch state Handlers do not reach into `ServerState` directly. They use a fixed set of named effects defined alongside the state in [`spec/01-objects/state.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/01-objects/state.lean): | Component | Effects | |---|---| | promises | `getPromise` / `setPromise` | | tasks | `getTask` / `setTask` | | schedules | `getSchedule` / `setSchedule` / `delSchedule` | | timeouts | `setPromiseTimeout` / `delPromiseTimeout` | | | `setTaskTimeout` / `delTaskTimeout` | | | `setScheduleTimeout` / `delScheduleTimeout` | | outbox | `setMessage` | Each effect is a small `M` action — a keyed lookup, upsert, or delete on one list. They compose. A handler is a sequence of effect calls. When you implement a storage backend, you are implementing these eight families. Get them right and every handler composes on top for free. One narrow exception: the settlement scrub — removing a newly-settled promise's id from every pending promise's callbacks list — is inlined at all three settlement sites rather than factored into a named effect. Watch for it; it is not a separate call. ## Handlers: pure functions from request to result Every protocol handler has the same signature: ```lean Req → (now : Nat) → M Res ``` `now` is the only way time enters the machine. No handler calls a clock. No handler reads an environment variable. Given the same `ServerState`, the same request, and the same `now`, a handler always produces the same `Res` and the same next `ServerState`. This is the totality property: every input produces exactly one output, no exceptions, no panics, no "it depends." The `taskAcquire` handler is a clean example. It checks state, checks version, transitions the task, arms a lease timeout, and returns: ```lean def taskAcquire (req : TaskAcquireReq) (now : Nat) : M TaskAcquireRes := do match ← getTask req.id with | none => return { status := 404 } | some t => ... let t := { t with state := .acquired, version := t.version + 1, ttl := some req.ttl, pid := some req.pid, resumes := [] } setTask t delTaskTimeout t.id setTaskTimeout t.id 1 (now + req.ttl) return { status := 200, task := some t.toRecord, ... } ``` No surprises. The version increment, the timeout arm, the status code — all visible in one screen of Lean. ## Reading the repo The spec has two directories under `spec/`: ```text spec/ 01-objects/ types.lean ← wire records, request/response types, enums state.lean ← ServerState, PromiseObject, TaskObject, effects, M 02-actions/ 00-resume.lean ← internal: enqueueResume (settlement cascade) 02-timeouts.lean ← internal: promise/task/schedule timeout transitions P-01-promise.get.lean ← protocol handlers: P- prefix = promise P-02-promise.create.lean ... T-01-task.get.lean ← T- prefix = task T-02-task.create.lean ... S-01-schedule.get.lean ← S- prefix = schedule ... ``` The naming convention is `{P|T|S}-{NN}-{handler.name}.lean`. When you need to verify a claim about `task.acquire`, open `T-03-task.acquire.lean`. When you need the wire shape of `TaskRecord`, open `types.lean`. The file names are the index; you do not need to grep. `00-resume.lean` and `02-timeouts.lean` are not protocol handlers — they are internal transitions the environment fires. The timeout file handles four transitions across the three timeout lists: promise expiry, task retry, task lease expiry, and schedule fire. Both import `state.lean` for effects and compose the same way protocol handlers do. Two functions in `state.lean` are declared `opaque`: ```lean opaque nextCron : (cron : String) → (after : Nat) → Nat opaque expand : (template id : String) → (timestamp : Nat) → String ``` `opaque` means the spec asserts these functions exist and satisfy their signatures, but does not define their implementation. `nextCron` (cron arithmetic) and `expand` (schedule promise-id templating) are left to the implementer. They are specified in behavior — the timeouts handler shows exactly how they are called — but not in algorithm. This is an intentional design boundary, not an oversight. ## The mental model for implementers Your server will be concurrent. It will have threads, connection pools, a transaction layer, and lock contention. None of that appears in the spec, and that is the point. The spec defines the **observable sequential behavior** your server must produce. For any sequence of requests your server processes, there must exist some sequential order in which the abstract machine produces the same sequence of responses and the same final state. Your server may be concurrent, but its observable behavior must be explainable by some sequential execution of the machine. This gives you a concrete testing strategy: replay a sequence of requests through the abstract machine, capture its state after each step, run the same sequence through your server, and diff. Any divergence is a bug in your server. Exactly one of the states is wrong. The spec says nothing about how you serialize concurrent requests, what isolation level your storage must provide, or how you handle races between a lease expiry and an in-flight heartbeat. These are real implementation problems — they just live outside the machine's boundary. See [Unspecified surface](/server/unspecified-surface) for a full accounting of what the spec deliberately leaves open. The spec's silence on concurrency is not a gap you need to fill before you can use it. You can run the machine on a single thread for testing, verify your handlers match it, and then add concurrency to your production server separately. The machine is the oracle for correctness; your concurrency strategy is the oracle for performance. The next chapter covers what the machine's state actually contains — the exact fields of every record, which are visible on the wire, and which are internal. See also [testing and conformance](/server/testing-and-conformance) for how to run the machine against your implementation in practice, and [state model](/server/state-model) for the full record definitions. --- *Verified against [`resonate-specification@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3).* --- --- url: /server/envelope-and-transport title: The request/response envelope and transport --- # The request/response envelope and transport Every operation your server handles — `promise.create`, `task.acquire`, `task.suspend`, all of them — arrives inside the same JSON envelope and leaves inside a matching response envelope. Get this layer right once and every handler composes on top of it cleanly. Get it wrong and the status codes become meaningless noise. ## What the Lean spec defines — and what it doesn't The [Lean specification](https://github.com/resonatehq/resonate-specification) defines the abstract state machine: per-handler `Req` and `Res` types in [`spec/01-objects/types.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/01-objects/types.lean), handler semantics in `spec/02-actions/`. It does **not** specify how those types are encoded on the wire — the concrete JSON field names, the header shape, or the protocol-version string. Those are defined by the reference server. The next two sections describe the reference server's choices and label them as such. If you deviate, your server is not wrong by the Lean model, but SDK clients that speak the reference protocol will fail against it. The Lean types are still your ground truth for handler semantics. Every `status` value in every `Res` type in `types.lean` — and every transition in the `02-actions/` files — is what your handlers must implement. The wire encoding just carries those semantics to the client. ## Envelope shape *The following field names and structure match the reference server's `RequestEnvelope` / `ResponseEnvelope` types in `src/types.rs`.* ### Request ```json { "kind": "promise.create", "head": { "corrId": "abc-123", "version": "2026-04-01", "auth": null }, "data": { ... } } ``` | Field | Type | Notes | |-------|------|-------| | `kind` | string | Operation name. Routes to the handler. Must be non-empty. | | `head.corrId` | string | Client-chosen correlation ID. Echoed back unchanged in the response. Use it to match async responses. | | `head.version` | string | Protocol version. The only accepted value today is `"2026-04-01"`. A mismatch produces `400`. | | `head.auth` | string \| null | Optional bearer token. Ignored when auth is not configured. | | `data` | object | Handler-specific payload. Must be a JSON object (not an array or scalar). | ### Response ```json { "kind": "promise.create", "head": { "corrId": "abc-123", "status": 200, "version": "2026-04-01" }, "data": { ... } } ``` | Field | Type | Notes | |-------|------|-------| | `kind` | string | Echo of `req.kind`. Always present, even on error. | | `head.corrId` | string | Echo of `req.head.corrId`. | | `head.status` | integer | Protocol status code (not HTTP status). See the table below. | | `head.version` | string | Always `"2026-04-01"` in the current reference server. | | `data` | any | On success: typed handler response object. On error: a string error message. | ## HTTP transport The Lean machine models no transport at all — but the prose specification does: [the message-passing protocol](/spec/execution-model/message-passing#address-schemes) names HTTP as the required transport, which every conformant implementation must support for both send and receive (other schemes are optional). The concrete binding below — routes, headers, JSON shape — is the reference server's. The reference server exposes a single route: ``` POST / Content-Type: application/json ``` All operations use this one endpoint. There are no per-resource URLs, no query parameters, no path segments that carry operation semantics. The `kind` field in the envelope is the entire routing key. The HTTP response status mirrors `head.status`. A `200 OK` from the HTTP layer means the envelope parsed and the handler ran; whether the operation itself succeeded is in `head.status`. (They will agree for success, and they agree for client errors — a `400` in the head is a `400 HTTP` — but the important thing is `head.status` is always authoritative.) Other transport bindings (WebSocket, message queue, in-process channel) are not prohibited, but none are specified. The reference server implements HTTP only. Earlier versions of the server exposed per-resource REST routes (`/promises`, `/tasks`, `/schedules`). The reference server returns `410 Gone` on those paths. Any client still using them needs updating before it will work against a conformant server. ## A concrete example: promise.create `promise.create` is the simplest handler to study because it always returns `200` — there is no 404 (idempotent: a re-create returns the existing promise), no 409, and no other success code. Every branch in [`spec/02-actions/P-02-promise.create.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-02-promise.create.lean) ends with `status := 200`. **Request:** ```json { "kind": "promise.create", "head": { "corrId": "c7f2a0b9", "version": "2026-04-01" }, "data": { "id": "payments.charge.txn-42", "timeoutAt": 1800000, "param": { "headers": { "content-type": "application/json" }, "data": "{\"amount\": 100}" }, "tags": { "resonate:target": "poll://any@workers" } } } ``` **Response:** ```json { "kind": "promise.create", "head": { "corrId": "c7f2a0b9", "status": 200, "version": "2026-04-01" }, "data": { "promise": { "id": "payments.charge.txn-42", "state": "pending", "param": { "headers": { "content-type": "application/json" }, "data": "{\"amount\": 100}" }, "value": {}, "tags": { "resonate:target": "poll://any@workers" }, "timeoutAt": 1800000, "createdAt": 1720300000000, "settledAt": null } } } ``` Field-by-field notes: - `data.id` — Promise identifier. The same string echoed back in `promise.id`. Dot-separated segments encode the call tree; the prefix is the origin. - `data.timeoutAt` — Epoch milliseconds. If this timestamp is already in the past at create time, the promise is born `rejected_timedout` (or `resolved`, for timer promises with `resonate:timer: "true"`). - `data.param` — Arbitrary input payload: optional `headers` map and optional `data` string. The server stores and returns it verbatim. - `data.tags` — Key/value pairs with semantic meaning. `resonate:target` is the delivery address for the execute message the server sends to a worker. Valid schemes: `http://`, `https://`, `poll://cast@group[/id]`, `gcps://project/topic`, `bash://`. A promise without `resonate:target` has no associated task and delivers no message. - `promise.createdAt` — Set by the server. Do not trust the client to supply it. - `promise.settledAt` — `null` until the promise reaches a terminal state; then epoch milliseconds of settlement. ## Status code vocabulary These codes appear in `head.status` on every response. They are protocol-level codes, not HTTP status codes, though the reference server maps them to matching HTTP statuses. | Code | Meaning | Which handlers | |------|---------|----------------| | `200` | Success. `data` is the typed response object. | All handlers on success. | | `300` | Immediate resume. At least one awaited promise is already settled or expired — nothing is registered, the task stays `acquired`, and the client must handle the awakenings in-band. | `task.suspend` only. | | `400` | Bad request. Envelope malformed, `kind` unknown or empty, `data` not an object, `version` not accepted, or field-level validation failure. | All handlers. | | `403` | Forbidden. Debug operations are disabled on this server. | `debug.*` handlers only. | | `404` | Not found. The addressed resource does not exist. | `promise.get`, `promise.settle`, `promise.register_callback`, `promise.register_listener`, `task.get`, `task.acquire`, `task.release`, `task.fulfill`, `task.suspend`, `task.fence`, `task.halt`, `task.continue`, `schedule.get`, `schedule.delete`. Not `task.create` — its existing-promise branch answers `409`/`422`, never `404`. | | `409` | Conflict. The operation is semantically valid but cannot proceed: wrong task state, version mismatch, or the resource already exists in an incompatible form. | `task.create`, `task.acquire`, `task.release`, `task.fulfill`, `task.suspend`, `task.fence`, `task.halt`, `task.continue`. | | `422` | Unprocessable. The request is structurally valid but violates a semantic constraint — for example, an awaiter promise that has no `resonate:target` tag, an awaited promise that does not exist during suspension, or a `task.create` where the promise exists but carries no target. | `promise.register_callback`, `task.create`, `task.suspend`. | | `500` | Internal error. Storage failure or unexpected condition. | All handlers (as a catch-all). | | `501` | Not implemented. | See below. | | `503` | Retriable storage contention (nothing committed). Documented reference-server design intent for serialization failures — not in the Lean model, and **not yet emitted**: the current reference server surfaces these as `500`. | None today. See [storage strategies](/server/storage-strategies). | ### Notes on the surprising codes **`300` is only ever from `task.suspend`.** No other handler returns it. It signals that at least one awaited promise is already settled or expired, so the server registered nothing and skipped parking the task — the awaited work is already done and the worker still holds the claim. The Lean spec defines this path explicitly: [`spec/02-actions/T-06-task.suspend.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-06-task.suspend.lean) line 27, when any awaited promise is `!= .pending` or past its `timeoutAt`. Do not confuse this with an error. The worker should re-run as if it received a resume message. See [state model](/server/state-model) for the full task lifecycle. **`promise.create` is 200-only.** Every branch in the Lean spec returns `status := 200`. A re-create of an existing promise returns the existing record — idempotent, still 200. There is no 409 or 404 from this handler. **`501` and the search handlers.** The Lean spec marks all three search operations as unimplemented: [`P-06-promise.search.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-06-promise.search.lean), [`T-11-task.search.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-11-task.search.lean), and [`S-04-schedule.search.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/S-04-schedule.search.lean) each contain only `return { status := 501 }`. The reference server implements all three and returns `200` with paginated results. This is a real divergence: the Lean model says search is not part of the spec; the reference server ships it. A minimally conformant server must handle the 501 cases (so clients don't crash on unknown status codes), but search implementation is not a spec requirement. If you implement it, match the reference server's pagination shape (`cursor`-based, max 1000 per page). **`409` vs `422` — which is which.** The split is not arbitrary: `409` means "your operation is valid but the current state won't allow it" (a worker presenting a stale version is the canonical case); `422` means "you referenced something in a way the server can't satisfy" (an awaited promise that doesn't exist, an awaiter that has no target). In practice: if you see a 409 from `task.acquire`, stop — you've been outrun. If you see a 422 from `task.suspend`, the promise ID you passed doesn't exist on this server. ## Cross-references The message-level delivery contract — execute and unblock message structure, address schemes, anycast vs unicast delivery semantics — is specified in [Message Passing Protocol](/spec/execution-model/message-passing). That page covers what the server sends *to* workers; this chapter covers what workers send *to* the server. For how status codes map to task and promise state transitions, see [State Model](/server/state-model). For the surface the spec deliberately leaves open (search semantics, auth, concurrent transport bindings), see [Unspecified Surface](/server/unspecified-surface). --- *Verified against [resonate-specification`@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3) and reference server [`c8d7c7b`](https://github.com/resonatehq/resonate/tree/c8d7c7b).* --- --- url: /server/implementing-promises title: Implementing the promise handlers --- # Implementing the promise handlers Five specified handlers plus one deliberate stub (`promise.search`) cover the entire promise surface. They share two mechanics you will apply constantly: **timeout projection** — when a pending promise's `timeoutAt ≤ now`, return a computed settled record (`resolved` for a timer promise, `rejectedTimedout` otherwise) *without writing it*; the `onPromiseTimeout` transition is what eventually writes the settlement — and **idempotency** (returning the current state instead of an error on a repeated operation). The full projection story is in [Timeouts and projection](/server/timeouts-and-projection). This chapter applies both handler by handler. All six are specified formally in [`resonatehq/resonate-specification`](https://github.com/resonatehq/resonate-specification/tree/83d64c3/spec/02-actions). --- ## promise.get — P-01 [`P-01-promise.get.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-01-promise.get.lean) Return the promise identified by `req.id`, applying timeout projection if it is pending and expired. ```text 404 promise does not exist 200 + promise promise exists (pending, projected, or already settled) ``` If the promise is pending and `timeoutAt ≤ now`, do **not** write anything — return a projected record with: - `state` = `resolved` for timer promises (`resonate:timer = "true"`), `rejectedTimedout` for everything else - `settledAt` = `timeoutAt` The actual settlement write happens in the timeout sweep, not here. Projection is a read-time computation; `promise.get` is never the trigger for a state change. --- ## promise.create — P-02 [`P-02-promise.create.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-02-promise.create.lean) This handler is the busiest in the server. It branches on whether the promise exists, whether it is already expired, and whether it carries scheduling tags. ### When the promise does not yet exist **Case A — `timeoutAt > now` (normal create):** 1. Write the promise in `pending` state. 2. If the promise is *external* — carries `resonate:target` or is a timer promise — register a promise timeout at `timeoutAt`. External promises must be swept; promises without a target or timer tag are settled by explicit protocol operations and need no sweep. 3. If `resonate:target` is set: create a task in `pending` at `version = 0`. - If `resonate:delay` is absent, or its parsed value is ≤ `now`: schedule a retry timeout at `now + retryTimeout` and enqueue an `execute` message to the target address. Work starts immediately. - If `resonate:delay` is set and its value is > `now`: schedule a retry timeout at `delay` only — **do not enqueue execute yet**. The execute fires when that timeout fires. See [Outbox and delivery](/server/outbox-and-delivery). 4. Return 200 with the new promise record. `retryTimeout` is a server configuration value; the default in the spec is 5000 ms (`ServerConfig.retryTimeout` in `state.lean`). **Case B — `timeoutAt ≤ now` (already-expired create):** The caller asked for a promise with a deadline in the past. Write it directly as settled: - `state` = `resolved` (timer) or `rejectedTimedout` (everything else) - `createdAt = settledAt = timeoutAt` If `resonate:target` is present, also create a task, but in `fulfilled` state — there is no work to dispatch on an already-expired promise. Return 200. ### When the promise already exists Return it as-is — or projected if it is pending and expired — and return **200**. `promise.create` never returns 409. A repeated call with the same `id` returns 200 with the existing promise, regardless of whether the new request's parameters match. This surprises engineers coming from create-APIs that use 409 to signal "already exists." Here, idempotency is the default: callers can safely retry a create after a network failure, and the server absorbs the duplicate. Do not add a 409 path — the spec has none. The [durable promise specification](/spec/programming-model/durable-promise-specification) documents an extended surface (`ikc`/`iku`/strict modes) that the reference server implements. The Lean model does not specify these modes; a conformant-to-Lean server need not implement them. --- ## promise.settle — P-03 [`P-03-promise.settle.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-03-promise.settle.lean) Settle is a **single parameterized verb**: `state` is a parameter to the request, not a separate operation per outcome. Resolve, reject, and cancel are argument values — there are no `promise.resolve` or `promise.reject` endpoints. ```text 404 promise does not exist 200 + promise promise settled (or already settled / projected) ``` ### The write path — pending and unexpired When the promise is pending and `timeoutAt > now`: 1. Snapshot the promise's current `listeners` and `callbacks` lists. 2. Write the promise: set `state = req.state`, `value = req.value`, `settledAt = now`, and clear `callbacks` and `listeners` to empty. 3. Remove the promise timeout (`delPromiseTimeout`). A settled promise has no deadline to sweep. 4. If a task exists for this promise id: transition it to `fulfilled`, clear `pid`, `ttl`, and `resumes`, and remove its timeout (`delTaskTimeout`). 5. Run the **settlement scrub** (see below). 6. Enqueue an `unblock` message to every address in the snapshotted listeners list. See [Outbox and delivery](/server/outbox-and-delivery). 7. Call `enqueueResume` for each id in the snapshotted callbacks list. ### The settlement scrub After a promise settles, it can never wake another awaiter — but other pending promises may still have its id in their callbacks lists. The scrub removes those stale registrations: ```lean -- From P-03-promise.settle.lean modify fun s => { s with promises := s.promises.map fun q => if q.state == .pending then { q with callbacks := q.callbacks.filter (· != p.id) } else q } ``` Walk every promise in the store. For each one that is still pending, filter the settled promise's id out of its callbacks list. Non-pending promises are left alone. **Why this matters for storage design.** The scrub is a cross-record write — one settlement event triggers a deletion across an unbounded number of rows in whatever table backs promise callbacks. A naive relational schema where callbacks are stored as a JSON array on the promise row makes this an `UPDATE ... SET callbacks = ...` for every pending promise. A separate `callbacks` join table reduces this to a single `DELETE FROM callbacks WHERE awaiter_id = :settled_id`, which scales. The choice of schema here has a measurable effect on settle latency at scale. See [Storage strategies](/server/storage-strategies) for the tradeoffs. ### enqueueResume `enqueueResume` (`00-resume.lean`) transitions the awaiter's task based on its current state: - **`suspended`**: move the task to `pending`, set `resumes = [awaitedId]`, schedule a retry timeout at `now + retryTimeout`, and enqueue an `execute` message to the awaiter's `resonate:target` — but only if the awaiter promise exists and its `resonate:target` tag is non-empty; an absent or empty target sends nothing (the state change still happens). The awaiter is now claimable again. - **`pending`, `acquired`, or `halted`**: the task is already active; append `awaitedId` to its `resumes` list (deduped). No message is sent — the task will read the updated resumes when it next interacts with the server. - **`fulfilled`**: no-op. The awaiter is done. Note that `enqueueResume` does not check the awaiter promise's deadline. Deadline enforcement on the awaiter happens through the task's own lifecycle — an expired awaiter's own settlement fulfills its task. ### Idempotency If the promise is already settled (any non-pending state), return it with 200. If it is pending but `timeoutAt ≤ now`, apply timeout projection and return 200. In both cases no write occurs and no messages are enqueued. The reference server (`oracle.rs`, `trigger_settlement`) implements the same three phases — fulfill the task, process callbacks, notify listeners — through `trigger_fulfilled`, `trigger_callbacks`, and `trigger_listeners` respectively. This is corroboration, not the definition. --- ## promise.register_callback — P-04 [`P-04-promise.register_callback.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-04-promise.register_callback.lean) A callback is an **awaiter promise id**. Registering one tells the server: when the awaited promise settles, run `enqueueResume` for the awaiter. ```text 404 awaited promise does not exist 422 awaiter promise does not exist, OR awaiter lacks resonate:target 200 + promise awaited promise (pending, projected, or settled) ``` Validate in this order: 1. Look up `req.awaited`. If absent, 404. 2. Look up `req.awaiter`. If absent, 422. 3. Check that the awaiter promise carries `resonate:target`. If not, 422. A callback without a target address is useless — `enqueueResume` has nowhere to send the execute message, so the server rejects the registration upfront rather than silently losing the wake-up. If the awaited promise is pending and `timeoutAt > now`, **and** the awaiter is also pending and `timeoutAt > now`: add `req.awaiter` to the awaited promise's callbacks list (deduped via `addCallback`). If registration is skipped, the response is still `200` with the awaited promise — but the two skip cases look different to the caller. If the awaited promise is already settled (or expired, and therefore projected settled), the caller sees the settled state and knows to resume immediately. But if the awaited is still pending and only the *awaiter's* condition failed (expired or non-pending), the caller receives a `200` carrying a **pending** awaited promise with no signal that the registration was dropped. An SDK must not treat a pending-state `200` from `promise.register_callback` as proof the callback landed. Both the awaited and the awaiter must be pending and unexpired for a registration to land. If the awaited is pending but the awaiter has already expired, the callback is silently skipped — a dead awaiter has no task to resume. This is intentional: the settlement scrub (P-03) handles the reverse direction (settling the awaited removes its id from other promises' lists), so missing this registration causes no permanent leak. --- ## promise.register_listener — P-05 [`P-05-promise.register_listener.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-05-promise.register_listener.lean) A listener is an **address** for `unblock` message delivery. Where a callback wakes a durable awaiter through the task system, a listener delivers a direct notification to any subscriber — an external process, a gateway, a polling client. ```text 404 awaited promise does not exist 200 + promise awaited promise (pending, projected, or settled) ``` If the awaited promise is pending and `timeoutAt > now`: add `req.address` to the promise's listeners list (deduped via `addListener`). The address will receive an `unblock` message when the promise settles, either via `promise.settle` (P-03) or the timeout sweep. If the promise is already settled or expired: return the current or projected state without registering. The caller can read the settlement value directly from the response — no notification will come. There is no 422 here. Any address string is accepted; the server does not validate that the address resolves to a reachable subscriber at registration time. See [Outbox and delivery](/server/outbox-and-delivery) for delivery semantics. --- ## promise.search — P-06 [`P-06-promise.search.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-06-promise.search.lean) ```text 501 not implemented — unspecified surface ``` The Lean spec returns 501 unconditionally. Search semantics — filter shape, pagination, index requirements, sort order — are unspecified. See [Unspecified surface](/server/unspecified-surface). The reference server implements a search endpoint with tag-based filtering and cursor pagination. That behavior is the reference server's design, not a spec requirement. If you implement search, use the reference server as a practical starting point, but do not treat its behavior as normative. --- *Verified against [resonate-specification`@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3).* --- --- url: /server/implementing-schedules title: Implementing schedules --- # Implementing schedules A schedule is a cron-driven promise factory. It does not execute work; it fires `promise.create` at each tick, and the promise machinery — tasks, routing, callbacks — handles everything downstream. The four CRUD handlers are straightforward; the semantics live entirely in the timeout firing side. ## The schedule object A schedule carries everything needed to describe one class of recurring work: | Field | Role | |---|---| | `id` | Schedule identifier | | `cron` | Cron expression — syntax is **implementation-defined** | | `promiseId` | Template string for the created promise's id — expanded per occurrence | | `promiseTimeout` | Duration (ms) added to `cronTime` to compute each promise's `timeoutAt` | | `promiseParam` | Value passed as `param` to each created promise | | `promiseTags` | Tags passed to each created promise | | `createdAt` | When the schedule was created | | `nextRunAt` | Next scheduled fire time; maintained by the server | | `lastRunAt` | Most recent fire time; `none` until the first tick fires | ## S-01 — schedule.get Lookup by id. Returns `200` with the schedule object, or `404` if none exists. No side effects. ([S-01-schedule.get.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/S-01-schedule.get.lean)) ## S-02 — schedule.create `schedule.create` is fully idempotent. The spec ([S-02-schedule.create.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/S-02-schedule.create.lean)) checks for an existing schedule first: ```lean match ← getSchedule req.id with | some s => return { status := 200, schedule := some s } | none => let s : Schedule := { ..., nextRunAt := nextCron req.cron now, lastRunAt := none } setSchedule s setScheduleTimeout s.id s.nextRunAt return { status := 200, schedule := some s } ``` If a schedule with the given id already exists, it is returned unchanged — no update, no merge. If it does not exist, the server constructs it with `nextRunAt := nextCron req.cron now` and immediately arms the schedule timeout. **The response is always `200`.** There is no `201 Created`. A caller that needs to distinguish a first create from a duplicate must compare the returned schedule's `createdAt` against its own record. **`nextCron` is declared `opaque` in the spec.** Its contract — "return the next fire time strictly after the given instant" — is specified; the cron format it interprets is not. Whether you support standard five-field POSIX cron, a six-field variant with seconds, or something else is your choice. Document it; clients depend on it. See [Unspecified surface](/server/unspecified-surface). ## S-03 — schedule.delete Lookup by id. If not found: return `404`. If found: call `delSchedule` and `delScheduleTimeout` atomically, then return `200`. ([S-03-schedule.delete.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/S-03-schedule.delete.lean)) The timeout disarm is required. Deleting the schedule record while leaving the timeout entry alive means the handler fires against a gone schedule — a no-op in the spec, but a wasted wakeup and a source of confusion in practice. The spec removes both or neither. ## S-04 — schedule.search The spec ([S-04-schedule.search.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/S-04-schedule.search.lean)) returns `501 Not Implemented` unconditionally. Implement it the same way. Schedule search behavior is unspecified. See [Unspecified surface](/server/unspecified-surface). ## The firing side: onScheduleTimeout and catchUp When a schedule's timeout fires, the handler does not fire once — it fires for every missed occurrence since the last run. The spec calls this the catch-up loop. ([02-timeouts.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/02-timeouts.lean), lines 77–97) ```lean partial def catchUp (now : Nat) (s : Schedule) : M Schedule := do if s.nextRunAt ≤ now then let cronTime := s.nextRunAt let promiseId := expand s.promiseId s.id cronTime let _ ← promiseCreate { id := promiseId, timeoutAt := cronTime + s.promiseTimeout, param := s.promiseParam, tags := s.promiseTags } cronTime catchUp now { s with lastRunAt := some cronTime, nextRunAt := nextCron s.cron cronTime } else return s def onScheduleTimeout (id : String) (now : Nat) : M Unit := do match ← getSchedule id with | none => pure () | some s => let s ← catchUp now s setSchedule s delScheduleTimeout s.id setScheduleTimeout s.id s.nextRunAt ``` Trace through one cycle: 1. `catchUp` checks `s.nextRunAt ≤ now`. If yes, this occurrence is due. 2. `cronTime := s.nextRunAt` — the **logical** time of this occurrence, not the wall clock `now`. 3. The promise id is computed with `expand s.promiseId s.id cronTime`. **`expand` is declared `opaque` in the spec.** The function signature is `(template id : String) → (timestamp : Nat) → String`; the template substitution syntax is not specified. See [Unspecified surface](/server/unspecified-surface). 4. `promiseCreate` is called with `cronTime` as the `now` argument. The promise's `timeoutAt` is `cronTime + s.promiseTimeout` — anchored to when the tick *should* have fired, not when the handler ran. 5. The function recurses with `nextRunAt := nextCron s.cron cronTime` — advancing from the just-fired tick, not from the current wall clock. Each recursive step covers exactly one occurrence. 6. Base case: `s.nextRunAt > now`. Return the updated schedule. After `catchUp`, `onScheduleTimeout` writes the updated schedule (with new `nextRunAt` and `lastRunAt`), disarms the current timeout entry, and arms a new one at `s.nextRunAt` — the next upcoming tick. ### Normative shapes **The catch-up shape is normative.** You must call `promiseCreate` once per missed occurrence, in chronological order, each using its logical `cronTime`. Skipping missed ticks, or firing only the most recent, violates the spec. **`promiseCreate` is idempotent.** If a promise with the expanded id already exists, the call returns the existing promise and does nothing. Repeated catch-up calls for an already-fired tick are safe. **Tick-anchored timeouts are a real edge case.** Each created promise gets `timeoutAt = cronTime + s.promiseTimeout`. If `promiseTimeout` is small and the server was down long enough, `cronTime + s.promiseTimeout` may already be in the past. The `promise.create` handler creates an already-expired promise in the settled state rather than pending. Test this path explicitly — an expired-at-creation promise does not dispatch a task. If a server restarts after a long outage, `catchUp` fires `promiseCreate` for every missed occurrence before returning control. A schedule that ticks every minute with a server down for an hour means sixty synchronous promise creates in one timeout handler invocation. Account for this in your storage layer and test it explicitly. ## Cross-references - [Timeouts and projection](/server/timeouts-and-projection) — the three timeout lists and when `onScheduleTimeout` is dispatched - [Implementing promises](/server/implementing-promises) — `promiseCreate` semantics that `catchUp` relies on, including the already-expired creation path - [Outbox and delivery](/server/outbox-and-delivery) — promises created by catch-up dispatch tasks and send execute messages if they carry a `resonate:target` tag - [State model](/server/state-model) — `Schedule` and `ScheduleTimeout` struct definitions - [Unspecified surface](/server/unspecified-surface) — `nextCron` (cron syntax) and `expand` (promise-id template syntax) are both implementation-defined --- *Verified against [resonate-specification`@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3).* --- --- url: /server/implementing-tasks title: Implementing tasks: the full handler reference --- # Implementing tasks: the full handler reference This chapter walks every task handler in the spec — what it validates, what state it writes, and where the sharp edges live. Read [the task model](/spec/system-model/tasks) for the invariants your state must satisfy at all times, and [implementing promises](/server/implementing-promises) before this chapter — `task.fence` delegates directly to those handlers. The mutating handlers share a common validation spine for acquired-task operations: task exists (404), promise exists (409), task state correct (409), promise pending and unexpired (409), version matches (409). Learn the pattern once and each handler is a short diff from it. --- ## `task.get` (T-01) [`T-01-task.get.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-01-task.get.lean) — a read with a projection rule. If the task is not found — or the task is found but its promise is not — return `404`. If both exist and the associated promise is `pending` with `timeoutAt > now`, return the stored task record as-is (`200`). Otherwise return a synthesized record with `state=fulfilled`, `pid=null`, `ttl=null`, `resumes=[]` — also `200`. ```text task or promise not found → 404 promise pending + unexpired → 200 (real record) promise settled or expired → 200 (projected fulfilled) ``` The projection is a read-time transformation, not a write. The stored task record is unchanged; you are presenting the logical state the client should see once a promise can no longer be fulfilled. Full projection semantics are described in [timeouts and projection](/server/timeouts-and-projection). --- ## `task.create` (T-02) [`T-02-task.create.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-02-task.create.lean) — atomic promise creation with immediate task acquisition. The path branches on whether the promise already exists. **Promise absent:** If `a.timeoutAt > now`: create the promise (`state=pending`) and the task (`state=acquired`, `version=1`, `pid` and `ttl` set from the request). Arm the promise timeout at `p.timeoutAt` (`setPromiseTimeout` — unconditional here, unlike `promise.create`, which arms it only for external promises) and the lease timeout. Return `200` with both records. If your implementation fails to arm the promise timeout, `onPromiseTimeout` never fires for this promise — listeners are never unblocked and suspended awaiters never wake. If `a.timeoutAt ≤ now`: the window has already closed. Create a pre-settled promise — `state=resolved` for a timer promise, `state=rejectedTimedout` otherwise — and a fulfilled task (`version=0`, no `pid`/`ttl`). Return `200`. The work was never live; treat this the same as finding an already-fulfilled task. **Promise exists:** First check: the existing promise must carry the `resonate:target` tag. If it does not, return `422`. Then look up the task: | Task state | Response | |---|---| | No task found | `409` | | `fulfilled` | `200`, returned as-is | | `pending` | re-acquire: `version++`, `state=acquired`, set `pid`/`ttl`, clear `resumes`, delete retry timeout, arm lease timeout; `200` | | `acquired`, `suspended`, or `halted` | `409` | A fresh `task.create` whose promise does not yet exist succeeds whether or not the request carries a `resonate:target` tag — the tag check does not appear in the absent-promise branch of the spec. The check fires only when the promise already exists. Do not pre-validate tag presence on a fresh create; you will reject requests the spec accepts. --- ## `task.acquire` (T-03) [`T-03-task.acquire.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-03-task.acquire.lean) — the standard entry point for a worker claiming a dispatched task. Validate in this order: 1. Task not found → `404`. 2. Promise not found → `409`. 3. `t.state != .pending` → `409`. 4. `p.state != .pending` or `p.timeoutAt ≤ now` → `409`. 5. `t.version != req.version` → `409`. On success: `version++`, `state=acquired`, set `pid`/`ttl`, clear `resumes`, delete the retry timeout, arm the lease timeout. Return `200` with both records. Clearing `resumes` matters: a task can accumulate buffered wakeups while `pending` (awaited promises settling before re-acquire), and the acquiring worker must not see a prior cycle's phantom resumes. **Version semantics — read this carefully.** Version increments on every `pending→acquired` transition. That transition happens in two places: `task.acquire` (T-03) and the re-acquire branch of `task.create` (T-02, when the promise exists and the task is `pending`). It does not change anywhere else. An earlier draft of the protocol prose described this inverted — as if version incremented at the moment a task returned to `pending`. The Lean spec is unambiguous: the `version + 1` expression appears only in the `pending→acquired` branches of T-02 and T-03. `task.release` (T-08), `task.continue` (T-10), lease-expiry recovery, and the resume path (`00-resume.lean`) all return a task to `pending` at its stored version. The *next* acquire bumps it. If your implementation increments version on release or lease expiry, the execute message that re-dispatches the task will carry the wrong (pre-bump) version, and a correct worker will fail its version check on acquire. --- ## `task.fence` (T-04) [`T-04-task.fence.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-04-task.fence.lean) — version-guarded delegation to promise handlers. Validate: task exists (404), promise exists (409), `t.state=acquired` (409), promise `pending+unexpired` (409), version match (409). On success, delegate to the **full** `promise.create` or `promise.settle` handler — the complete implementation including task dispatch and settlement chains, not a stripped copy. The outer response status is always `200` (the fence succeeded); the inner action result carries its own status, which the caller must inspect. This is how a worker makes writes safe against its own zombie predecessors. A process that lost its lease cannot pass the version check; it discovers it is dead before it acts, not after. See [implementing promises](/server/implementing-promises) for the handler semantics being delegated to. --- ## `task.heartbeat` (T-05) [`T-05-task.heartbeat.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-05-task.heartbeat.lean) — lease refresh for a batch of tasks. The server iterates `req.tasks`. For each entry, it checks: task exists, `state=acquired`, `version=ref.version`, `pid=req.pid`, and the associated promise is `pending+unexpired`. If all conditions hold, it deletes the existing timeout and arms a new one at `now + t.ttl`. Entries that fail any check are silently skipped. Response is always `200`. There is **no error signal for a lost task** in the heartbeat response. If another worker has re-acquired a listed task, the heartbeat simply doesn't refresh it. The original worker learns about the loss on its next mutation attempt, which returns `409`. Two corollaries for your implementation: - Do not infer from a `200` heartbeat that all listed tasks are still live. - The server does not look up tasks by `pid`; it only validates the tasks explicitly named in `req.tasks`. A task absent from the list — even one held by the requesting worker — will not have its lease refreshed. --- ## `task.suspend` (T-06) [`T-06-task.suspend.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-06-task.suspend.lean) — park on awaited promises. Standard validation: task (404), promise (409), `t.state=acquired` (409), promise `pending+unexpired` (409), version match (409). Each action in `req.actions` names an awaited promise; if any named promise does not exist, return `422`. **Before committing the suspension**, scan every awaited promise. If any is already settled or expired (`p.state != .pending` or `p.timeoutAt ≤ now`), do not suspend. Instead: clear the task's `resumes` field (reset any buffered wakeups), and return `300`. The task remains `acquired`. The worker must handle the awakenings in-band. If all awaited promises are genuinely pending, register the task's id as a callback on each, transition `state=suspended`, clear `pid`/`ttl`/`resumes`, delete the lease timeout. Return `200`. A `300` from `task.suspend` means at least one awaited promise is already settled or past its `timeoutAt` deadline (the spec condition is `pa.state != .pending ∨ pa.timeoutAt ≤ now` — an expired-but-stored-pending promise also triggers it). The task was never suspended; it remains `acquired`, and the worker still holds the claim and the version. The worker must check which awaited promises are done, collect their results, and continue driving the execution. Build your worker loop to handle `300` as the fast path for awakenings that race the suspension request. --- ## `task.fulfill` (T-07) [`T-07-task.fulfill.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-07-task.fulfill.lean) — atomic settlement of promise and task. Standard validation. On success: 1. Settle the promise: write `state`, `value`, `settledAt`; clear `callbacks` and `listeners`. 2. Delete the promise timeout. 3. Set task to `fulfilled`: clear `pid`, `ttl`, `resumes`. 4. Delete the task timeout. 5. **Settlement scrub:** scan all pending promises and remove the just-fulfilled promise's id from their callback lists. A callback pointing at a now-fulfilled promise is unreachable, and leaving stale registrations would violate the invariant that no suspended task holds a dead callback. 6. Emit an `unblock` message to every listener address. 7. Call `enqueueResume` for every awaiter callback — waking each suspended task that was waiting on this promise. Steps 6 and 7 are the settlement propagation chain. See [outbox and delivery](/server/outbox-and-delivery) for how outbox messages reach workers. --- ## `task.release` (T-08) [`T-08-task.release.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-08-task.release.lean) — return a task from `acquired` to `pending`. Standard validation. On success: `state=pending`, clear `pid`/`ttl`, delete the lease timeout, arm a retry timeout. Emit an execute message to `resonate:target` carrying the task's **current version** (not incremented). The task is now claimable again. The version in the execute message is the version the next worker will present to `task.acquire`, which will then bump it. Emitting the current version is correct; do not pre-increment it here. --- ## `task.halt` and `task.continue` (T-09, T-10) Administrative pause and resume — not part of the normal worker loop. **`task.halt` ([T-09](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-09-task.halt.lean)):** Look up the task (404 if absent). Then: - `state=fulfilled` → `409`. - `state=halted` → `200` (idempotent). - Any other state (`pending`, `acquired`, `suspended`) → `state=halted`, clear `pid`/`ttl`, delete task timeout → `200`. There is **no version check** and no promise lookup. `task.halt` is an administrative override; it does not require the caller to be the current claimant. **`task.continue` ([T-10](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-10-task.continue.lean)):** Look up the task (404 if absent). If `state != .halted` → `409`. Look up the promise (404 if absent). On success: `state=pending`, arm a retry timeout, emit an execute message with the current version. Return `200`. A halted task holds no lease and no retry slot but retains its version. The next `task.acquire` bumps the version as usual. --- ## The resume path `enqueueResume` is an internal operation called for each callback on a settling promise — by `task.fulfill` (T-07), by `promise.settle` (P-03), and by the `onPromiseTimeout` transition. It is not an HTTP handler; your server calls it directly during settlement. Source: [`00-resume.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/00-resume.lean). | Awaiting task state | Action | |---|---| | `suspended` | `state=pending`, `resumes=[awaitedId]`, arm retry timeout, emit execute with **current version** (only if the awaiter promise exists with a non-empty `resonate:target`) | | `pending`, `acquired`, or `halted` | Buffer `awaitedId` in `task.resumes` (deduplicated); no state change, no execute | | `fulfilled` | No-op | The execute message on resume carries the pre-acquire version. This is correct: the worker that receives the execute calls `task.acquire` with that version, and `task.acquire` is what bumps it. Bumping here would put the execute message out of sync with the stored version, breaking the version check on the next acquire. The buffering behavior for `pending`/`acquired`/`halted` tasks ensures that a wakeup arriving while a task is already running (or paused) is not lost. The worker reads the accumulated `resumes` field when it next inspects its task record. --- ## `task.search` (T-11) [`T-11-task.search.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-11-task.search.lean) returns `501 Not Implemented`. The filtering contract is unspecified. See [unspecified surface](/server/unspecified-surface). --- *Verified against [resonate-specification`@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3).* --- --- url: /server title: Build a server --- # Build a server The Distributed Async Await protocol is specified twice, on purpose. [`resonatehq/resonate-specification`](https://github.com/resonatehq/resonate-specification) is the executable abstract machine in Lean 4 — the normative, machine-checkable definition of the protocol's core handlers and state transitions. The [/spec pages](/spec) are the prose specification: the human-readable companion that explains the same protocol. Where prose and Lean disagree, the Lean model wins. This track is for engineers implementing a conformant server — on a new substrate, in a new language, or as an authoritative study of what conformance actually requires. ## Durable execution is a protocol, not a product Most workflow engines ship as a system you adopt: run their cluster, use their API, accept their operational model. The Distributed Async Await design inverts that. The protocol defines an abstract machine — a state, a set of atomic effects on that state, and a set of request handlers defined as pure functions over both. Any platform that can durably store that state and execute those effects atomically can become a durable-execution provider. This matters practically. Your organization may already operate a relational database, a wide-column store, or a message broker. If that substrate can provide durable storage and an atomic compare-and-swap-equivalent primitive, it can speak this protocol. You do not need to adopt new infrastructure — you need to implement the protocol spec against the infrastructure you already run. The honest mechanism is: **implement the protocol spec against your platform.** Not "drop in a storage adapter to an existing server." Three independent implementations prove the point across three radically different storage paradigms; none of them is a plugin to any other. ## What "conformant" means A conformant server implements the protocol handlers the Lean abstract machine defines and holds the invariants the machine specifies after every handler invocation. That is the entire contract. Concretely: - Every handler the spec defines — `promise.get`, `promise.create`, `promise.settle`, `promise.register_callback`, `promise.register_listener`, `task.get`, `task.create`, `task.acquire`, `task.fence`, `task.heartbeat`, `task.suspend`, `task.fulfill`, `task.release`, `task.halt`, `task.continue`, `schedule.get`, `schedule.create`, `schedule.delete`, plus the internal `resume` and `timeouts` transitions — must behave as the corresponding Lean function specifies. (The three `*.search` handlers also have Lean definitions: `501`, deliberately.) - State transitions must be atomic with respect to the spec's state model. A handler either completes all its effects or none. - The invariants the machine holds over promises, tasks, schedules, timeouts, and the outbox must hold after every transition. What is *not* specified is as important as what is. Transport encoding, authentication, search semantics (`promise.search`, `task.search`, `schedule.search` are `501` stubs in the spec), cron-expression evaluation, and concurrency limits are intentionally left to the implementer. The Lean machine does not model HTTP or JSON at all — though the [prose spec's message-passing protocol](/spec/execution-model/message-passing#address-schemes) does name HTTP as the required transport, with the envelope encoding defined by the reference server. No specific storage engine is required. Those decisions are yours. The handlers and their state semantics are not. The mechanism that verifies conformance — an oracle-diff harness that replays operation sequences and checks invariants after every step — exists and is used internally. It is not yet public. Treat it as an open question rather than a drop-in tool your CI can point at today. The [testing and conformance chapter](/server/testing-and-conformance) covers how to build toward that bar now. ## Three implementations as proof Three implementations exist today across three distinct storage paradigms. Together they demonstrate that substrate independence is a property the protocol already has, not an aspiration. ### resonatehq/resonate — relational SQL, the reference server [`resonatehq/resonate`](https://github.com/resonatehq/resonate) is the mature reference implementation. Written in Rust, licensed Apache-2.0. It ships with three built-in storage backends — SQLite, Postgres, and MySQL — selectable at startup. Its HTTP/JSON wire protocol defines the envelope all existing Resonate SDKs speak. This is the implementation to study first. It is the most complete, the most tested, and the one the SDKs were built against. When this track says "the reference server," it means this. ### resonatehq/resonate-on-scylladb — wide-column NoSQL [`resonatehq/resonate-on-scylladb`](https://github.com/resonatehq/resonate-on-scylladb) is a complete, independent reimplementation of the protocol in Go, backed by ScyllaDB (a Cassandra-compatible wide-column store). It is source-available under BUSL-1.1 (free for development and evaluation; commercial license for production; converts to Apache 2.0 on 2030-07-01). Its consistency mechanism is ScyllaDB Lightweight Transactions — Paxos-based compare-and-swap — which realizes the atomic effects the spec requires. It speaks **the same HTTP/JSON envelope as the reference server** (protocol version `2026-04-01`, `kind`-multiplexed POST). Any existing Resonate SDK points at it unchanged. It is a true drop-in backend swap. This is an official Resonate HQ project (not a community fork), actively developed. Its test rigor is exceptional: oracle-diff against the reference server, random-kill fault injection across 53 named invariants (at the time of writing), and Porcupine linearizability verification. It is not yet at 1.0 and has known open issues. Label it accordingly — most credible non-reference production story, not yet production-ready. ### resonatehq/resonate-on-nats — message broker [`resonatehq/resonate-on-nats`](https://github.com/resonatehq/resonate-on-nats) is a complete, independent reimplementation of the protocol in Go, backed by NATS JetStream (a message broker and key-value store). It is source-available under BUSL-1.1 (free for development and evaluation; commercial license for production; converts to Apache 2.0 on 2030-07-01). Each promise's state lives as a single JSON blob per origin in JetStream KV, written with optimistic CAS. Its transport is **NATS pub/sub — not HTTP**. Existing Resonate SDKs cannot talk to it without a NATS-aware client or bridge. This is not a drop-in backend swap; it is a NATS-native architecture, distinct from the HTTP envelope the other two servers share. This server is experimental — created June 2026, approximately three weeks of development, minimal test coverage. It is architecturally elegant and demonstrates the most surprising paradigm leap (a message broker as a durable-execution substrate), but it is not a production story today. ### What the spread proves Relational SQL, wide-column NoSQL, a message broker. Three storage paradigms, three different consistency mechanisms (MVCC + WAL, LWT/Paxos, JetStream KV CAS), and in the NATS case a different transport entirely. All three implement the same protocol and interoperate with the same SDK layer (with the noted NATS exception on transport). The protocol is substrate-independent. Neither Go server is a plugin to the Rust server. Each independently reimplements the protocol specification from scratch. That independence is the point: the spec is the shared contract, not the Rust implementation. ## Chapters in this track Read [/spec](/spec) first if you have not — the protocol semantics are defined there. The SDK track ([/sdk](/sdk)) documents the other side of the wire: how a client consumes the handlers you are implementing. | Chapter | What you get | |---|---| | [Abstract machine](/server/abstract-machine) | The Lean 4 model as an implementer's map — state components, effect types, handler shapes, and how to navigate the spec repo as your primary reference. | | [State model](/server/state-model) | The five state components (promises, tasks, schedules, timeouts, outbox) and the invariants your storage layer must maintain over them at all times. | | [Envelope and transport](/server/envelope-and-transport) | The HTTP/JSON wire format the reference server uses, what is specified versus what is convention, and how a non-HTTP transport relates. | | [Implementing promises](/server/implementing-promises) | All five specified promise handlers — `get`, `create`, `settle`, `register_callback`, `register_listener` — and the settlement scrub inlined at all three settlement sites. | | [Implementing tasks](/server/implementing-tasks) | All nine specified task handlers from `acquire` through `halt`/`continue`, the version fencing mechanism, and lease semantics. | | [Implementing schedules](/server/implementing-schedules) | The three specified schedule handlers, the `nextCron` question, and the catch-up transition for schedules that fire while the server is down. | | [Timeouts and projection](/server/timeouts-and-projection) | How the server fires timeout transitions, what projection means (observing settled state before the transition persists), and the implementation ordering that makes projection safe. | | [Outbox and delivery](/server/outbox-and-delivery) | The two outbox message types (`execute` and `unblock`), atomic enqueue with their triggering transitions, and delivery guarantees. | | [Storage strategies](/server/storage-strategies) | How the three existing servers map spec effects to relational SQL, wide-column CAS, and JetStream KV — and the patterns that generalize to other substrates. | | [Testing and conformance](/server/testing-and-conformance) | The invariants your implementation must hold, how to build an oracle-diff harness, fault injection, and the state of the public conformance question. | | [Unspecified surface](/server/unspecified-surface) | Everything the spec deliberately leaves open — transport, auth, search, `nextCron`, `expand`, `preload`, operational knobs. Know the boundary before you make it a requirement. | --- *Verified against [`resonate-specification@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3).* --- --- url: /server/outbox-and-delivery title: The outbox and message delivery --- # The outbox and message delivery The outbox is not a queue you bolt on after you get the state machine working. It is a field of [`ServerState`](https://github.com/resonatehq/resonate-specification/blob/main/spec/01-objects/state.lean) — the same struct that holds promises, tasks, and schedules — and its entries are written by the same handlers that write state transitions. If your outbox row doesn't commit atomically with the state change that caused it, you have a bug. ```lean -- spec/01-objects/state.lean structure ServerState where config : ServerConfig := {} promises : List PromiseObject tasks : List TaskObject ... outbox : List OutboxEntry := [] ``` That placement is intentional. The outbox is the server's declaration of what *must* be delivered to the outside world. It says what is pending; the transport says what has been sent. Keeping those two concerns separate is the entire architecture of reliable dispatch. ## Two message types, nothing else The spec defines exactly two messages: ```lean -- spec/01-objects/state.lean:78–81 inductive Message | execute (taskId : String) (version : Nat) | unblock (promise : PromiseRecord) ``` **`execute (taskId, version)`** tells a target address to acquire and run a specific task. The version is the current fencing token — not a hint to increment, just the version the worker must present to `task.acquire`. A worker that receives this, acquires successfully, and finds its version stale has already been superseded; it should stop. **`unblock (promise)`** tells a listener address that a promise they registered on has settled. It carries the full settled promise record — state, value, tags — so the recipient can act without a round-trip fetch. There is no third type. There is no "heartbeat message" or "cancel message" flowing through the outbox. ## `setMessage` is an upsert Every enqueue goes through one function: ```lean -- spec/01-objects/state.lean:166–170 def setMessage (address : String) (msg : Message) : M Unit := modify fun s => let entry := OutboxEntry.mk address msg let key := entry.key { s with outbox := entry :: s.outbox.filter (fun e => e.key != key) } ``` This is a keyed upsert: drop any existing entry with the same key, prepend the new one. Keys are defined as: ```lean -- spec/01-objects/state.lean:88–90 def OutboxEntry.key : OutboxEntry → String | { message := .execute taskId _, .. } => taskId | { address, message := .unblock p } => s!"{p.id}:notify:{address}" ``` **For execute:** keyed by `taskId` alone. Two executes for the same task collapse into one — the later write wins. This is safe: if the retry timeout fires and enqueues a second execute before the first is delivered, the worker will see only one (with the most current version). A stale duplicate from before the upsert has already been dropped. **For unblock:** keyed by `"{promiseId}:notify:{address}"`. A given (promise, listener) pair produces at most one pending unblock in the outbox. If the promise settles and the listener is registered twice (which `addListener` prevents, but the upsert is the backstop), only one outbox entry exists. ## `retryTimeout` — the re-dispatch cadence [`ServerConfig`](https://github.com/resonatehq/resonate-specification/blob/main/spec/01-objects/state.lean) carries one field that governs how quickly the server re-offers a task: ```lean structure ServerConfig where retryTimeout : Nat := 5000 -- milliseconds ``` `retryTimeout` is the interval at which a `pending` task re-emits its execute message if no worker has claimed it. Every handler that enqueues an execute also arms a retry-timeout at `now + retryTimeout`. That timeout fires `onTaskRetryTimeout`, which re-emits the execute and arms the timeout again — an automatic re-dispatch loop until a worker acquires the task or the promise is settled. ## Full enumeration of enqueue sites These are every spec handler that *directly* calls `setMessage`, verified against the Lean files. Two handlers also produce messages **transitively**: `task.fence` (T-04) delegates to the full `promiseCreate`/`promiseSettle` handlers, and `onScheduleTimeout`'s catchUp loop calls `promiseCreate` for each fired occurrence — so both inherit those handlers' enqueue behavior, and both must run inside the same transaction scope as the delegated handler's writes: ### Execute messages | Transition | Handler | Lean file | |---|---|---| | Promise created with `resonate:target` (no delay, or elapsed delay) | `promiseCreate` | [`P-02-promise.create.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-02-promise.create.lean) | | Worker releases an acquired task | `taskRelease` | [`T-08-task.release.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-08-task.release.lean) | | Halted task resumed by operator | `taskContinue` | [`T-10-task.continue.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-10-task.continue.lean) | | Pending task's retry timeout fires | `onTaskRetryTimeout` | [`02-timeouts.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/02-timeouts.lean) | | Acquired task's lease expires | `onTaskLeaseTimeout` | [`02-timeouts.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/02-timeouts.lean) | | Awaited promise settles, suspended task wakes | `enqueueResume` | [`00-resume.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/00-resume.lean) | ### Unblock messages | Transition | Handler | Lean file | |---|---|---| | Promise settled via `promise.settle` | `promiseSettle` | [`P-03-promise.settle.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-03-promise.settle.lean) | | Promise settled via `task.fulfill` | `taskFulfill` | [`T-07-task.fulfill.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-07-task.fulfill.lean) | | Promise timed out | `onPromiseTimeout` | [`02-timeouts.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/02-timeouts.lean) | All three settlement paths iterate `p.listeners` and call `setMessage address (.unblock p.toRecord)` for each registered listener. All three also call `enqueueResume` for each registered callback — which may produce additional execute messages if the awaiting task was suspended. Miss the `task.fulfill` path and a promise settled through T-07 notifies no one — no error, no signal, just listeners that never fire. One enqueue site that does **not** send a message: `promise.create` with a `resonate:delay` tag whose deadline has not yet elapsed. The task is created in `pending` and a retry timeout is armed, but no execute is sent immediately — the first dispatch waits for `onTaskRetryTimeout` to fire at the delay instant. This is a named tag behavior, not general spec machinery. One more: `task.create` creates the task in `acquired` (the calling worker already holds it) and writes no execute to the outbox. The execute appears later, when `task.release` or a timeout transitions the task back to `pending`. ## The resume chain `enqueueResume` handles the case where an awaited promise settles and the task that was waiting on it needs to be woken: ```lean -- spec/02-actions/00-resume.lean def enqueueResume (awaitedId awaiterId : String) (now : Nat) : M Unit := do let some t ← getTask awaiterId | pure () match t.state with | .suspended => let retryTimeout := (← get).config.retryTimeout let t := { t with state := .pending, resumes := [awaitedId] } setTask t setTaskTimeout t.id 0 (now + retryTimeout) match ← getPromise awaiterId with | none => pure () | some p => let target := (p.tags.get? "resonate:target").getD "" if target != "" then setMessage target (.execute t.id t.version) | .pending | .acquired | .halted => if t.resumes.contains awaitedId then pure () else setTask { t with resumes := t.resumes ++ [awaitedId] } | .fulfilled => pure () ``` Three things worth noting: **Version is not bumped.** `enqueueResume` sends `(.execute t.id t.version)` — the current version, not an incremented one. The spec comment is explicit: a resume re-emits the current version as a wake-up hint. The version only advances on `task.acquire`. See [tasks and the worker loop](/sdk/tasks-and-the-worker-loop). **Non-suspended states buffer.** If the task is `pending`, `acquired`, or `halted` when its awaited promise settles, `enqueueResume` doesn't send anything — it appends `awaitedId` to `t.resumes` (deduplicated). The task already has something driving it; the settled promise is recorded and will be visible when the driver next inspects state. If the task is `fulfilled`, it's a no-op. **Empty target sends nothing.** If the promise's `resonate:target` tag is absent or empty, `enqueueResume` transitions the task to `pending` and arms the retry timeout, but skips the execute. This means a polling worker that re-acquires via `task.acquire` (bypassing push entirely) will still find the task — the retry timeout will re-offer it — but no push message is sent. A suspended task carries no task timeout (invariant 6 in the [spec's task model](/spec/system-model/tasks)). The retry timeout is armed only when `enqueueResume` transitions the task to `pending`. Before that, the task is parked indefinitely, waiting on a settlement. This is correct and expected: the wakeup is event-driven, not timer-driven. If the awaited promise never settles (e.g., an eternal external promise), the suspended task waits forever — which is intentional. Timer promises exist precisely to bound that wait. See [timeouts and projection](/server/timeouts-and-projection). ## Implementation shape: atomic commit, then drain The outbox only works if the outbox row commits in the same transaction as the state change that caused it. If you write the state, commit, then write the outbox, a crash between the two leaves a task pending with no pending delivery — the task re-surfaces only if a timeout fires later, if one was armed. If you write the outbox and crash before committing the state change, the row is orphaned (benign but noisy). Atomicity is the correct direction. The reference server (Apache-2.0, [`resonate`](https://github.com/resonatehq/resonate)) implements this in its SQLite and Postgres persistence layers. Every handler runs inside a database transaction (`unchecked_transaction()` → `commit()` in `persistence_sqlite.rs`). The `INSERT INTO outgoing_execute` and `INSERT INTO outgoing_unblock` statements run inside that same transaction, alongside the promise/task/timeout updates. A background delivery loop then drains the outbox: ```rust // resonate/src/processing/processing_messages.rs let (execute_msgs, unblock_msgs) = storage .transact(move |db| db.take_outgoing(batch_size)) .await?; ``` `take_outgoing` uses `DELETE ... RETURNING` — it atomically removes and returns a batch of rows. This means a message is consumed exactly once by the delivery loop; if the loop crashes after deleting but before delivering, the message is lost. The transport must be prepared to re-send if an upstream retry fires — and the worker must tolerate duplicate executes. See [tasks and the worker loop](/sdk/tasks-and-the-worker-loop) for the version-based deduplication that makes receiving the same execute twice safe. The loop runs on a configurable `poll_interval` and `batch_size` from the server config. Neither is specified by the DAA protocol — they are reference-server operational parameters. The unblock delivery follows the same pattern. The spec defines what belongs in the outbox and when. It does not define how deliveries are retried if the network is down when the loop drains a batch. The reference server's delivery loop is fire-and-forget per batch: it deletes the rows, dispatches, and moves on. Retry, backoff, and transport-level deduplication are implementation concerns above the spec floor. The retry timeout (`onTaskRetryTimeout`) re-enqueues execute messages for tasks that were never acquired — so for execute, the spec itself provides a recovery path at the cost of `retryTimeout` latency. For unblock, a listener that misses its delivery window has no built-in recovery; the listener address is the worker's responsibility to keep alive. ## Wire shape The reference server serializes execute messages as: ```json { "kind": "execute", "head": { "serverUrl": "https://my-server.example.com" }, "data": { "task": { "id": "my-task-id", "version": 3 } } } ``` And unblock messages as: ```json { "kind": "unblock", "head": {}, "data": { "promise": { "id": "...", "state": "resolved", ... } } } ``` The wire encoding and address format are transport-specific and unspecified by the DAA protocol. For address semantics and the full picture of delivery on the wire, see [message passing](/spec/execution-model/message-passing). For how to implement the task side of this flow, see [implementing tasks](/server/implementing-tasks). --- *Verified against [resonate-specification`@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3).* --- --- url: /server/state-model title: State model: records, enums, and internal fields --- # State model: records, enums, and internal fields The [abstract machine](/server/abstract-machine) holds all its knowledge in a single `ServerState`. This chapter walks every component of that state: the records, their fields, which fields cross the wire, and which are internal implementation details the spec uses but never exposes to callers. The canonical source for everything here is [`spec/01-objects/types.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/01-objects/types.lean) (wire records and enums) and [`spec/01-objects/state.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/01-objects/state.lean) (internal objects, effects, and `ServerState`). Read these files directly when a detail matters for your implementation. ## Two levels: internal objects and wire records The spec defines two shapes for each entity: - **Internal object** (`PromiseObject`, `TaskObject`) — what the machine stores. Contains all fields including ones that are never serialized. - **Wire record** (`PromiseRecord`, `TaskRecord`) — what appears in request/response bodies. A strict subset of the internal object. The conversion is explicit. `PromiseObject.toRecord` strips `callbacks` and `listeners`. `TaskObject.toRecord` strips `resumes` (as a `List String`) and replaces it with `resumes.length` (a count). If you store more than the wire record in your database, those extra fields are implementation internals that callers will never see and must never rely on. ## Promises ### PromiseState ```text pending | resolved | rejected | rejectedCanceled | rejectedTimedout ``` Five variants. A promise starts `pending` and moves to exactly one terminal state. The three `rejected*` variants distinguish a caller-initiated rejection (`rejected`), an administrative cancellation (`rejectedCanceled`), and a timeout expiry (`rejectedTimedout`). `resolved` is the success terminal. See the [durable promise specification](/spec/programming-model/durable-promise-specification) for the semantics of each state. ### PromiseRecord (wire-visible) | Field | Type | Meaning | |---|---|---| | `id` | `String` | Unique promise identifier | | `state` | `PromiseState` | Current state | | `param` | `Value` | Input value set at creation; immutable | | `value` | `Value` | Output value; meaningful only when settled | | `tags` | `Tags` | Key-value pairs; `resonate:target` and `resonate:timer` drive machine behavior | | `timeoutAt` | `Nat` | Unix ms deadline; past this instant a pending promise is *projected* as already settled | | `createdAt` | `Nat` | Unix ms creation timestamp | | `settledAt` | `Option Nat` | Unix ms settlement timestamp; `none` while pending | `Value` is `{ headers : Tags, data : Option String }`. `Tags` is `List (String × String)`. Both appear on the wire as-is. ### PromiseObject (internal — adds two fields) | Field | Type | Wire-visible? | Meaning | |---|---|---|---| | *(all PromiseRecord fields)* | — | Yes | See above | | `callbacks` | `List String` | **No** | Awaiter promise IDs registered on this promise; flushed and replaced with resume messages when the promise settles | | `listeners` | `List String` | **No** | Addresses subscribed for an `unblock` message when the promise settles | `callbacks` holds promise IDs, not task IDs. When a task suspends waiting for a promise to settle, the server adds the *awaiter's promise ID* (which equals the awaiter's task ID) to the awaited promise's `callbacks` list. `listeners` holds delivery addresses (strings). Both lists are deduplicated: adding an entry that already exists is a no-op. ### External and timer promises Two predicate functions on `PromiseObject` matter throughout the machine: - **External promise**: `p.tags.has "resonate:target" || p.isTimer` — a promise that triggers work outside the machine (a worker execution or a timer). External promises always have a corresponding task. - **Timer promise**: `p.tags.get? "resonate:timer" == some "true"` — a special external promise that settles `resolved` (not `rejectedTimedout`) when its `timeoutAt` elapses. See [tasks](/spec/system-model/tasks) for how `resonate:target` causes a task and an execute message to be created alongside the promise. ## Tasks ### TaskState ```text pending | acquired | suspended | halted | fulfilled ``` Five variants. `fulfilled` is the only terminal state; `halted` is an administrative state that a task enters via `task.halt` and leaves via `task.continue` (which returns it to `pending`). A task in `halted` is out of circulation: it holds no lease and no retry timeout. Your monitoring should surface halted tasks explicitly — they will not self-recover. ### TaskRecord (wire-visible) | Field | Type | Wire-visible? | Meaning | |---|---|---|---| | `id` | `String` | Yes | Task identifier; equals its promise's id | | `state` | `TaskState` | Yes | Current state | | `version` | `Nat` | Yes | Fencing token; increments on each `task.acquire` | | `resumes` | `Nat` | Yes | **Count only** — number of buffered settled-awaited IDs | | `ttl` | `Option Nat` | Yes | Lease deadline (Unix ms); `none` when not acquired | | `pid` | `Option String` | Yes | Worker process ID holding the lease; `none` when not acquired | ### TaskObject (internal — `resumes` is richer) | Field | Type | Wire-visible? | Meaning | |---|---|---|---| | *(all TaskRecord fields except `resumes`)* | — | Yes | See above | | `resumes` | `List String` | **No** (exposed as `.length`) | Settled promise IDs that triggered this task's re-pending while it was suspended, pending, acquired, or halted; deduplicated | The `resumes` divergence is the sharpest difference between the two levels. On the wire, `resumes` is a count — a worker can observe that multiple settlements fired while it was away but cannot see which ones without looking up the awaited promises individually. Internally, the machine stores the actual IDs so that `enqueueResume` can deduplicate. ### Version and lease **Version** (`version : Nat`) is the fencing token. It increments on every `pending→acquired` transition — in `task.acquire` (see [`T-03-task.acquire.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-03-task.acquire.lean)) and in the re-acquire branch of `task.create` ([`T-02-task.create.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-02-task.create.lean)). No other operation changes it. A task returning to `pending` after a lease expiry or a `task.release` keeps its current version; the next `task.acquire` bumps it. Every mutating operation on a task (`fence`, `suspend`, `fulfill`, `release`) must present the current version and receives `409` if the version has moved on. This is how a stale worker's writes are rejected. **Lease** is `ttl`+`pid` together. `ttl` is the Unix ms deadline after which the server considers the worker dead; `pid` identifies which worker holds it. Both are `none` when a task is not acquired. Both are set on `task.acquire` and cleared on `task.suspend` (the task parks without a lease; it is woken by settlement, not by a clock). A lease timeout fires `onTaskLeaseTimeout`, which returns the task to `pending`, clears `ttl` and `pid`, and re-enqueues an execute message — all without advancing the version. The machine carries a task back to `pending` at its current version when the lease lapses. The version advances on the *next* `task.acquire`. A test that asserts "version incremented the instant the lease expired" will fail against a correct implementation. Assert instead that once any worker re-acquires, the version is strictly greater than what the previous holder saw. ## Schedules A `Schedule` is a recurring cron rule. The machine stores it as a single record with no internal-only variant: | Field | Type | Meaning | |---|---|---| | `id` | `String` | Schedule identifier | | `cron` | `String` | Cron expression (implementation-defined interpretation via `opaque nextCron`) | | `promiseId` | `String` | Template for promise IDs created at each fire; expanded via `opaque expand` | | `promiseTimeout` | `Nat` | `timeoutAt` offset (ms) applied to each created promise | | `promiseParam` | `Value` | `param` for each created promise | | `promiseTags` | `Tags` | `tags` for each created promise | | `nextRunAt` | `Nat` | Unix ms of next scheduled fire | | `lastRunAt` | `Option Nat` | Unix ms of last fire; `none` before the first | | `createdAt` | `Nat` | Unix ms creation timestamp | Each schedule carries its own timeout entry. When that timeout fires, `onScheduleTimeout` calls `catchUp` — which loops, creating promises for every missed fire since the last run, advancing `nextRunAt` after each one, until the schedule is current. See [timeouts and projection](/server/timeouts-and-projection) for the full catchUp semantics. ## Timeout lists `ServerState` has three timeout lists. Each entry is a `(id, timeout)` pair the environment is expected to fire when `now ≥ timeout`: | List | Entry type | `kind` field | When fired | |---|---|---|---| | `promiseTimeouts` | `PromiseTimeout` | — | Settles a pending promise as `rejectedTimedout` (or `resolved` for timers) | | `taskTimeouts` | `TaskTimeout` | `0` = retry, `1` = lease | `0`: re-enqueues an execute for a pending task; `1`: expires an acquired task's lease | | `scheduleTimeouts` | `ScheduleTimeout` | — | Fires a schedule, creating promises for all missed intervals (catchUp) | The `kind` field on `TaskTimeout` is the only place where a single list encodes two distinct behaviors. A pending task has a retry timeout (`kind=0`): if no worker acquires it within `retryTimeout` ms (default 5000), the server re-enqueues the execute message. An acquired task has a lease timeout (`kind=1`): if no heartbeat refreshes it in time, the server evicts the worker. The two kinds are always exclusive — a pending task carries exactly one `kind=0` entry, an acquired task exactly one `kind=1` entry, and other states carry none; `delTaskTimeout` (which removes all entries for an id regardless of kind) always runs before `setTaskTimeout` in every handler that transitions between these states, so both kinds are never present at once. See [timeouts and projection](/server/timeouts-and-projection) for how projection interacts with both. ## Outbox `outbox : List OutboxEntry` is the machine's pending delivery queue. Each entry is: ```lean structure OutboxEntry where address : String message : Message -- execute (taskId version) | unblock (promise : PromiseRecord) ``` Two message types exist: - **`execute taskId version`** — dispatches a task to a worker at `address`. Carries the task's *current* version (not incremented by the enqueue; the next acquire will increment it). - **`unblock promise`** — notifies a listener at `address` that a promise has settled. Carries the settled `PromiseRecord`. Outbox entries are keyed: `setMessage` deduplicates by `taskId` for execute messages, and by `"{promiseId}:notify:{address}"` for unblock messages. A handler that enqueues the same message twice leaves only one entry. Your delivery layer reads and drains the outbox; the spec says nothing about delivery guarantees or ordering beyond the keying rule. See [outbox and delivery](/server/outbox-and-delivery) for implementation strategies. ## Cross-reference The objects defined here appear throughout the prose specification: - Promise lifecycle and promise states: [durable promise specification](/spec/programming-model/durable-promise-specification) - Task lifecycle, invariants, and version semantics: [tasks](/spec/system-model/tasks) - How timeout projection makes a pending-but-expired promise read as already settled: [timeouts and projection](/server/timeouts-and-projection) The abstract machine chapter preceding this one covers `M`, effects, and handler structure: [abstract machine](/server/abstract-machine). --- *Verified against [`resonate-specification@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3).* --- --- url: /server/storage-strategies title: Storage strategies: backing a DAA server --- # Storage strategies: backing a DAA server The [abstract machine](/server/abstract-machine) defines what a server must do. This chapter is about what you must give it to stand on. Every DAA server stores the same objects — promises, tasks, schedules, callbacks, listeners, outbox messages — and every mutating handler touches them with the same four requirements. Get those requirements right and the mapping to your storage technology is a detail. Get any of them wrong and the machine's safety guarantees evaporate. ## Drop-in vs. not drop-in — read this first The three public implementations are introduced, with maturity and license labels, in the [track overview](/server) — the fact that matters for this chapter: the reference server and `resonate-on-scylladb` speak the same HTTP/JSON envelope (existing SDKs point at either unchanged), while `resonate-on-nats` speaks NATS pub/sub instead of HTTP and is **not** a drop-in. State that kind of asymmetry clearly in your own implementation's documentation before your users discover it under load. ## What your storage must give you Four requirements follow directly from the machine. ### 1. Durability before acknowledgment A handler that returns `200` or `201` has committed. If the process dies the instant after that response leaves the wire, the committed state must survive and be visible to the next request. This rules out in-memory-only stores for any production path — ephemeral state is only safe for testing, and even then your test harness must account for it. ### 2. Atomic compare-and-swap for task mutations Task mutations — acquire, release, fulfill, suspend — all carry a `version` (fencing token). The version advances only on each fresh `task.acquire`; every subsequent mutation must be gated on the version the worker believes is current. A mutation that presents a stale version must fail without committing. This is the entire mechanism that prevents two workers from simultaneously driving the same execution. Your storage must expose either: - **Row-level transactions with a conditional update** (`UPDATE ... WHERE version = $expected`), or - **A native CAS primitive** that fails atomically if the expected revision does not match. A `SELECT` followed by an application-level check followed by an `UPDATE` does not satisfy this requirement. The gap between the read and the write is a window for a concurrent acquire to slip through. ### 3. Multi-record atomicity for handler effects Many handlers touch more than one row. The most load-bearing case is `task.fulfill`, which must settle the corresponding promise and complete the task — and must do both or neither. A crash between the two writes leaves the machine in a state it cannot recover from on its own. The same requirement applies to settlement scrub (processing callbacks and listeners when a promise settles) and to the transactional outbox: outbox messages written in the same transaction as the state change they describe, so no state change is ever committed without a corresponding message — and no message is ever committed without a corresponding state change. See [Outbox and delivery](/server/outbox-and-delivery) for the full delivery contract. ### 4. Efficient timeout scans The machine is full of deadlines: promise `timeout_at`, task retry timeouts, task lease expirations, schedule fire times. Your background worker needs to find expired entries cheaply and process them in order. If your timeout scan is a full table scan it will eventually become your bottleneck. A time-ordered index on `timeout_at` is the minimum. Wide-column stores without secondary indexes solve this with bucket-partitioned time tables; that pattern trades query flexibility for predictable scan cost. --- ## Case study 1: Relational SQL — how the reference server does it The reference server (`resonatehq/resonate`, Rust, Apache-2.0) defines all storage operations behind a single `Db` trait (`src/persistence/mod.rs`). The enum `Storage` wraps three backends — `Sqlite`, `Postgres`, `Mysql` — and exposes two entry points to handlers: ```text Storage::transact(f) // wraps f in a transaction; for all mutating handlers Storage::query(f) // read-only; no transaction overhead ``` Every mutating handler calls `transact`. The closure receives a `&dyn Db` reference and makes all its reads and writes through it. The database transaction commits when the closure returns `Ok`; it rolls back on any error. Handlers never commit partial state. ### Schema shape The SQLite backend (`src/persistence/persistence_sqlite.rs`) reveals the full object model. Tables of interest: ```text promises — id, state, param_*, value_*, tags, timeout_at, created_at, settled_at tasks — id (FK → promises.id), state, version promise_timeouts — timeout_at (indexed ASC), id task_timeouts — timeout_at (indexed ASC), id, timeout_type, process_id, ttl callbacks — awaited_id, awaiter_id, ready listeners — promise_id, address outgoing_execute — id, version, address outgoing_unblock — promise_id, address schedules — id, cron, promise_id, promise_timeout, … schedule_timeouts — timeout_at (indexed ASC), id ``` Tasks and promises share the same primary key — a task is an extension of its promise, not a separate entity. The `tasks.version` column is the fencing token, used in two distinct patterns. Acquire transitions (`task.acquire`, and `task.create`'s re-acquire branch) are the only mutations that increment it: `UPDATE tasks SET state = 'acquired', version = version + 1, pid = ?, ttl = ? WHERE id = ? AND version = ?`. Every other version-fenced mutation (`fence`, `suspend`, `fulfill`, `release`) checks without bumping: `UPDATE tasks SET state = ?, … WHERE id = ? AND version = ?`. Either way, stale version → no rows updated → the handler returns `409`. SQLite runs with WAL mode, `busy_timeout = 5000`, and `foreign_keys = ON`. WAL allows concurrent readers while a writer holds the lock; `busy_timeout` absorbs transient write contention without an immediate error. ### The outbox `outgoing_execute` and `outgoing_unblock` are the outbox tables. The reference server drains them with `take_outgoing`, which issues `DELETE ... RETURNING` — it claims and removes a batch in one statement, so there is no window where a message is "claimed" but not yet removed. Delivery is at-most-once from the storage side; the server retries from the transport side on failure. See [Outbox and delivery](/server/outbox-and-delivery). ### Serialization errors Postgres (and MySQL) can return serialization errors (`SQLSTATE 40001`) or deadlock errors (`40P01`) on concurrent transactions. The `From` implementation converts these to `StorageError::Serialization` — documented in `persistence/mod.rs` as "retries exhausted, nothing was committed." Note: the handler layer currently surfaces this as a generic `500`; a distinct retriable `503` is the documented design intent but is not yet implemented in `server.rs`. SQLite avoids this category with its single-writer model. The MySQL backend maps storage-level constraint violations (oversized field values against `VARCHAR(255)` columns) to `StorageError::InvalidInput`, which becomes `400`. If you are implementing a MySQL backend, size constraints are a real sharp edge: a client that sends a promise id longer than 255 bytes will receive `400`, not `500`. Test this path explicitly. --- ## Case study 2: Wide-column NoSQL — resonate-on-scylladb `resonatehq/resonate-on-scylladb` is a complete Go reimplementation of the server protocol backed by ScyllaDB. It speaks the same HTTP/JSON envelope as the reference server, so existing SDKs connect to it with no changes. The schema collapses promises and tasks into a single wide row (`internal/dbms/schema.cql`): ```text promises (PRIMARY KEY (origin, id)) — state, param_*, value_*, tags, timeout_at, created_at, settled_at — task_state, task_version, task_ttl, task_pid, task_resumes — task_timeout_retry, task_timeout_lease — callbacks (SET), listeners (SET) ``` Co-locating task and promise in the same row has a direct payoff: a `task.fulfill` that settles the promise and completes the task is a single-row write. There is no multi-table transaction to coordinate, and no partial-commit window. ### CAS via Lightweight Transactions ScyllaDB does not support multi-row ACID transactions. Instead it offers Lightweight Transactions (LWT) — Paxos-based CAS on a single partition. The server uses this for every conditional operation. `task.create` for a fresh promise issues: ```cql INSERT INTO promises (id, origin, …, task_state, task_version, …) VALUES (…) IF NOT EXISTS ``` `task.create` for a re-acquire on an existing pending task issues an `UPDATE ... IF task_state = 'pending'` conditional. If the condition fails the LWT returns `applied = false` and the handler returns `409` to the client — the same semantics the reference server achieves with a conditional SQL update inside a transaction. The `IF NOT EXISTS` / `IF ` idiom is the CAS primitive. It is atomic within a single partition (one `origin + id` pair). Multi-partition atomicity is not available; the schema is designed so that each handler's critical path fits within one partition. ### Timeout tables ScyllaDB lacks efficient secondary indexes across arbitrary partition keys, so timeouts use dedicated time-bucket tables: ```text promise_timeouts (PRIMARY KEY ((bucket, shard), timeout_at, origin, promise_id)) task_timeouts (PRIMARY KEY ((bucket, shard), timeout_at, timeout_type, origin, task_id)) schedule_timeouts (PRIMARY KEY ((bucket, shard), timeout_at, origin, schedule_id, create_token)) ``` `bucket` is derived from `timeout_at` by rounding to a configurable width (default `1h`). A background worker scans forward from `now - lookback` across the current and near-future buckets. This gives O(expired entries in window) scan cost rather than O(total rows). Because ScyllaDB has no transactions spanning the timeout entry and the main promise row, `task.create` pre-inserts the timeout rows *before* the LWT on the main row. If the LWT fails, the server rolls back the orphaned timeout entries. This is a visible pattern in the source (`handler_task.go`, step 2a/2b): pre-insert → LWT → on conflict, clean up orphans. --- ## Case study 3: Message broker — resonate-on-nats `resonatehq/resonate-on-nats` is a Go reimplementation backed by NATS JetStream KV. It is a **different transport model** from the two cases above: SDKs communicate via NATS subjects, not HTTP. Existing HTTP-based SDKs cannot use it without a NATS-aware client. This is the not-drop-in server. ### State layout: one KV entry per origin The server partitions state by `origin` — the `resonate:origin` tag value on a promise. All promises, tasks, schedules, callbacks, listeners, and outbox messages for a given origin are serialized into a single JSON blob and stored under one JetStream KV key. A server instance subscribes to a subset of origins (partitions) and processes messages for those origins serially. ```text KV key: base64url(origin) KV value: { promises: {…}, tasks: {…}, schedules: {…}, callbacks: {…}, listeners: {…}, messages: […], timeouts: […] } ``` Outbox messages sit inside the same blob as the state they describe. Writing the blob commits both simultaneously — no separate outbox table, no partial-commit risk. ### Optimistic CAS via JetStream revision JetStream KV exposes a per-key revision counter. The server uses it as a CAS token: ```go // Create (revision 0 → key does not yet exist) newRev, err = kv.Create(ctx, kvKey(origin), data) // Update (must present current revision) newRev, err = kv.Update(ctx, kvKey(origin), data, revision) // Both return ErrKeyExists / ErrKeyWrongLastSequence on a lost race → ErrCASConflict ``` On `ErrCASConflict` the server retries: reload state from KV, replay the handler logic, attempt the write again. This is optimistic concurrency — reads are never locked, writes win or retry. Serial-per-origin processing makes conflicts rare in practice (one actor per origin at a time), but the retry path must be correct because concurrent instances can race on the same origin during failover. This CAS is origin-scoped, not row-scoped. Every write to *any* promise or task within an origin touches the same KV key. That is the tradeoff: no multi-record coordination problem (the whole origin atomically), but contention within a hot origin hits a single CAS bottleneck. --- ## Building a new substrate: what you need If you are implementing storage from scratch, verify each of the following before considering your layer complete: 1. **Durability** — writes acknowledged to the handler are visible after a process crash and restart. Test by killing the process mid-write and reading back. 2. **Version-fenced task mutations** — `task.acquire`, `task.fulfill`, `task.release`, `task.suspend` must all fail with an observable conflict signal (return value, error code, or exception) when the presented version is not the current version. No read-then-write gap. 3. **Idempotent creates** — `promise.create` and `task.create` must be safe to call twice with the same id. The second call must return the existing record (with a `was_created = false` or equivalent signal), not an error and not a duplicate row. 4. **Atomic scrub + outbox** — settling a promise, processing its callbacks and listeners, and writing the resulting outbox messages must all commit or all fail. A settlement with no outbox entry produces a lost notification. An outbox entry with no settlement produces a phantom notify. 5. **Timeout scan** — you must be able to find all promises, tasks, and schedules whose `timeout_at <= now` without scanning the full dataset. A time-ordered index or a bucket-partitioned time table both work. 6. **Serialization error propagation** — on a lost CAS or a transaction conflict, fail the whole handler with nothing committed and surface a retriable error to the caller — never a silent empty response, and never a success. Neither the Lean model nor the reference server defines a distinct retriable status today (the reference server surfaces these as `500`; a distinct `503` is its documented design intent) — what matters is that your error is distinguishable and the caller can safely retry. See [the status-code table](/server/envelope-and-transport). Developers who implement storage in layers typically get durability and idempotent creates right early on. The first real pain arrives when a timeout scan starts reading the full table at scale, and the second arrives when a promise settlement commits without writing its callback notifications. Build both correctly from the start rather than retrofitting them. Once your storage layer passes basic handler tests, the conformance question becomes: does your server behave identically to the reference server across the full operation space — including corner cases like born-expired promises, concurrent acquires, and lease-lapsed re-claims? See [Testing and conformance](/server/testing-and-conformance) for how to structure that verification. ### Related chapters - [Abstract machine](/server/abstract-machine) — the state transitions your storage must durably record - [Outbox and delivery](/server/outbox-and-delivery) — the full transactional outbox contract and delivery semantics - [Envelope and transport](/server/envelope-and-transport) — how the HTTP/JSON envelope maps to storage reads and writes - [Testing and conformance](/server/testing-and-conformance) — how to know your storage layer got it right --- *Verified against [resonate-specification`@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3). Reference server source verified against `resonatehq/resonate` `@c8d7c7b`. ScyllaDB and NATS implementations verified from public repository READMEs, CQL schema, and Go source (`internal/dbms/schema.cql`, `internal/core/handler_task.go`, `internal/store.go`) fetched via GitHub API.* --- --- url: /server/testing-and-conformance title: Testing and conformance: what you can verify today --- # Testing and conformance: what you can verify today The conformance question is: does your server behave identically to the normative abstract machine for every valid input sequence? This chapter explains the tools and techniques you can reach for today — all of them anchored on public artifacts — and names what does not yet exist publicly so you know what you are not getting. See also: [Unspecified surface](/server/unspecified-surface) — before you write conformance tests, know which behaviors are yours to choose. ## The oracle: your primary reference `resonatehq/resonate` ships [`src/oracle.rs`](https://github.com/resonatehq/resonate/blob/main/src/oracle.rs) (Apache-2.0). It is a pure in-memory implementation of every protocol handler — the same `promise.get`, `task.acquire`, `task.suspend`, settlement scrub, and timeout-projection logic the full SQLite/Postgres/MySQL server runs, extracted into a self-contained struct with no I/O: ```rust pub struct Oracle { promises: HashMap, tasks: HashMap, schedules: HashMap, p_timeouts: Vec, t_timeouts: Vec, s_timeouts: Vec, outgoing: Vec<(String, Value)>, } impl Oracle { pub fn apply(&mut self, req: &RequestEnvelope) -> ResponseEnvelope { ... } } ``` `apply` dispatches a `RequestEnvelope` to the appropriate handler and returns a `ResponseEnvelope`. Every handler is deterministic: given the same state, the same request, and the same `now`, the output is identical every time. The `now` value comes from `req.head.debug_time` if set — overriding the wall clock — which is how seeded simulation works. The oracle-diff technique is direct: feed the same sequence of `RequestEnvelope` values to an oracle instance *and* to your server. Compare the `ResponseEnvelope` returned by each operation. Then call `debug.snap` on both after each step and diff the snapshots. Any divergence is a bug — either in your implementation or in a model you've built on top of it. The oracle is the tiebreaker. This is the same technique the [SDK testing guide](/sdk/testing-against-the-spec) teaches for SDK conformance; the direction is reversed (you are the server, not the SDK), but the oracle is the same artifact. The [`resonatehq/resonate-specification`](https://github.com/resonatehq/resonate-specification) repo is a valid Lean 4 project. `lake build` compiles it. You can call handlers directly from a Lean test or extract the generated `.olean` files for programmatic use — giving you a second, formally-checkable oracle layer independent of the Rust oracle. The Lean spec is normative; `oracle.rs` is an authoritative implementation of it. When they agree, you have high confidence. ## Reference-server debug endpoints The reference server exposes five debug endpoints, implemented in `src/server.rs`; the `Oracle` struct in `src/oracle.rs` implements the same five independently, for use in oracle-diff testing. They are reference-server tooling — they are not defined by the Lean spec — but they work against any server that implements them, and they are the primitives that make seeded simulation tractable. **`debug.start` / `debug.stop`** Use them to bracket a test scenario: send `debug.start` before the first operation, `debug.stop` after the last assertion. In the oracle they are explicit no-ops returning `200 {}`; in the running reference server they pause and resume the background processing loops (timeout firing, message delivery) so a test can control when those effects happen. **`debug.reset`** Clears all state — promises, tasks, schedules, all timeout queues, the outbox. Returns `200 {}`. Call this between test cases to get a clean slate without restarting the process. **`debug.snap`** Returns the complete current state as a deterministic snapshot: ```json { "promises": [...], // sorted by id "callbacks": [...], // sorted by awaited, then awaiter "listeners": [...], // sorted by promise_id, then address "tasks": [...], // sorted by id "promiseTimeouts": [...], // sorted by id "taskTimeouts": [...], // sorted by id "messages": [...] // outbox entries, in emission order } ``` The sorting is deliberate: snapshots are structurally comparable without additional normalization. Diff two snapshots and every difference is meaningful, not an ordering artifact. **`debug.tick`** The most powerful of the five. Send `{"kind": "debug.tick", "head": {"corrId": "...", "resonate:debug_time": T}, "data": {"time": T}}`. The server advances virtual time to `T` and synchronously fires every timeout that would have matured by `T` (the oracle does this against its in-memory map; the running reference server runs the same timeout processing against persistent storage): - Promise timeouts: settles pending promises (timer → `resolved`, else → `rejectedTimedout`), triggers settlement scrub, enqueues `unblock` messages, runs `enqueueResume` on callbacks. - Lease timeouts: returns acquired tasks to `pending`, re-arms retry timeout, emits `execute`. - Retry timeouts: re-emits `execute` for pending tasks. - Schedule timeouts: runs the `catchUp` loop — fires every missed cron occurrence, advances `nextRunAt`. This lets your test advance from `T=0` to `T=50000` in a single call and observe exactly the state the spec says should result, without waiting on real-time. If `head."resonate:debug_time"` is present on a `debug.tick`, it must equal `data.time` — a mismatch returns `400`. Omitting it is not an error, but then virtual time is not applied to the operation. Set it consistently, because `resonate:debug_time` is also what `resolve_time()` uses as `now` for every other operation — so a time-controlled test sequence must set it on every request, not just `debug.tick`. ## The structural invariants: assert after every operation The [task model](/spec/system-model/tasks) and [recovery protocol](/spec/execution-model/recovery-protocol#server-invariants) name structural invariants the server must hold after every transition. These are your most valuable assertions because they are total — they say nothing about a specific scenario, only about the state that any valid sequence of operations must produce. A violation pins the bug to the exact step that broke the invariant. The seven task invariants: 1. **Every task has a corresponding promise** — no orphan tasks. 2. **Every pending task has a retry timeout** — a claimable-but-unclaimed task will eventually be re-offered. 3. **Every acquired task has a lease** — a worker that dies is detected and replaced. 4. **Every suspended task has at least one callback** on a promise it awaits — it can be woken. 5. **No suspended task has an already-consumed callback** — a settled promise can't leave a task parked forever. 6. **No suspended task has a timeout** — suspension is open-ended; it waits on a settlement, not a clock. 7. **No fulfilled task has a timeout** — a terminal task holds no obligations. The Lean spec is the normative source for these; the [prose spec pages](/spec/system-model/tasks) restate them in readable form. Verify your assertions against the Lean files — several invariant identifiers in the spec source invert the described condition (a `no-timeout`-style name guards "must have a timeout"). Assert the described condition, not the name. The oracle's `debug.snap` response gives you everything you need to check all seven after every operation in a test loop. ## The ScyllaDB implementation as a worked example [`resonatehq/resonate-on-scylladb`](https://github.com/resonatehq/resonate-on-scylladb) — source-available under BUSL-1.1 (free for development and evaluation; commercial license for production; converts to Apache 2.0 on 2030-07-01) — demonstrates what a mature conformance-testing posture looks like against a non-reference implementation. Its README describes three test profiles, each runnable as a Docker Compose command: **Diff tests.** Feed the same operation sequences to the ScyllaDB implementation and to the reference server, compare responses and snapshots. This is oracle-diff with the reference server as oracle — the same technique described above. It exercises the full protocol surface under realistic latency. **Kill tests.** Inject random process kills during operation sequences and verify that the named invariants all hold after recovery. The invariant set lives in `internal/test/invariants.go` — 53 named invariants at the time of writing, checked on every snapshot (the set grows with the project). **Linearizability tests.** Run concurrent clients and record the history of operations and responses, then pass the history through [Porcupine](https://github.com/anishathalye/porcupine) — a Go implementation of Wing-Gong linearizability checking. A protocol that is explainable by some sequential execution of the abstract machine will produce a linearizable history. If Porcupine finds no valid sequential explanation, the implementation has a consistency bug. This three-layer approach — oracle-diff, fault injection against named invariants, linearizability verification — is the shape worth aiming for. ## What is not public today The conformance harness — the tooling that replays operation sequences against an arbitrary server and checks invariants after every step — exists and is used internally. It is private by design. No public pass/fail conformance suite exists today. Whether one is published is an open question. Do not wait for it; build your own against the oracle-diff and invariant-assertion techniques above. --- *Verified against [`resonate-specification@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3).* --- --- url: /server/timeouts-and-projection title: Timeouts and projection --- # Timeouts and projection The server maintains three timeout lists and four internal transitions that fire against them. None of these are user-visible API calls; they are background machinery the server runs on its own clock. They are also where the most important correctness property in the machine lives: the **projection convention**, which means that what a handler returns about a promise and what is actually written in storage can legally diverge — and your implementation must handle both sides correctly. ## The three timeout lists `ServerState` carries three separate timeout lists ([state.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/01-objects/state.lean), lines 62–76): ```lean structure PromiseTimeout where id : String timeout : Nat structure TaskTimeout where id : String kind : Nat -- 0 = pending retry, 1 = lease expiration timeout : Nat structure ScheduleTimeout where id : String timeout : Nat ``` **`promiseTimeouts`** is armed when a promise is created — but **only if the promise is external**. A promise is external when it carries a `resonate:target` tag (meaning a task backs it) or a `resonate:timer = "true"` tag ([state.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/01-objects/state.lean), lines 34–35): ```lean def PromiseObject.external (p : PromiseObject) : Bool := p.tags.has "resonate:target" || p.isTimer ``` Promises that carry neither tag — pure callback rendezvous points created by the SDK to coordinate internal workflows — are never added to the promise timeout list. Their only settlement path is an explicit `promise.settle` call from a worker. One exception to the external-only rule: `task.create` ([`T-02-task.create.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-02-task.create.lean), fresh-promise branch) calls `setPromiseTimeout` **unconditionally**, with no externality check — every task-backed promise created through T-02 gets a timeout entry. The tag-gated arming described above is `promise.create`'s (P-02) behavior. **`taskTimeouts`** carries two logically distinct entries, distinguished by `kind`: - `kind = 0`: pending-retry timeout. Set when a task is first created or returned to `pending`. Fires `onTaskRetryTimeout`, which re-issues the execute message. - `kind = 1`: lease-expiration timeout. Set when a task is acquired. Fires `onTaskLeaseTimeout`, which reverts the task to pending and re-issues the execute message. **`scheduleTimeouts`** holds one entry per schedule, set to `nextRunAt`. Fires `onScheduleTimeout`. ## The four internal transitions ### onPromiseTimeout [02-timeouts.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/02-timeouts.lean), lines 8–41. Fires when a promise timeout entry matures. Guard: if the promise does not exist, or is not `pending`, this is a no-op. Otherwise: 1. Settle the promise — `resolved` if `p.isTimer`, `rejectedTimedout` otherwise. `settledAt` is set to `p.timeoutAt` (the original deadline), not `now`. 2. Delete the promise timeout entry. 3. If a task exists for this promise id: transition it to `fulfilled`, clear `pid`/`ttl`/`resumes`, and delete its task timeout entries. 4. **Settlement scrub**: scan every pending promise in the store and remove this promise's id from their `callbacks` list. A timed-out promise can never resume an awaiter; callbacks pointing to it would stall forever without this step. 5. Send an `unblock` message (carrying the settled promise record) to every registered listener address via the outbox. 6. Call `enqueueResume` for every registered callback (awaiter promise id) — transitioning suspended awaiters back to `pending` and re-issuing their execute messages. Step 4 is the **settlement scrub**; steps 5–6 are the settlement propagation (listeners, then callbacks). The same three-step block runs identically in `promiseSettle` and `taskFulfill`. `onPromiseTimeout` is the only writer that executes it for expired promises. ### onTaskRetryTimeout [02-timeouts.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/02-timeouts.lean), lines 43–57. Fires when a `pending` task's retry timeout (kind 0) matures. Guard: task must exist and be in `pending`. Otherwise no-op. 1. Delete the existing kind-0 timeout entry. 2. Set a new kind-0 timeout at `now + retryTimeout` (default 5000 ms from `ServerConfig.retryTimeout`). 3. Re-send an `execute` message to the task's target address (taken from `resonate:target` on the promise). This is the mechanism that re-offers unclaimed work. A pending task with no worker will be re-dispatched every `retryTimeout` milliseconds until it is acquired or its promise settles. ### onTaskLeaseTimeout [02-timeouts.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/02-timeouts.lean), lines 59–75. Fires when an acquired task's lease expires (kind-1 timeout). Guard: task must exist and be in `acquired`. Otherwise no-op. 1. Revert the task to `pending` — clear `pid` and `ttl`. 2. Delete the existing kind-1 timeout entry. 3. Set a new kind-0 (pending-retry) timeout at `now + retryTimeout`. 4. Re-send an `execute` message to the target address. **The version is not incremented here.** It advances on the next `task.acquire`. A worker that held the lease at version `n` and lost it will find its next mutating operation rejected with `409` once another worker acquires at version `n+1`. See [Implementing tasks](/server/implementing-tasks). ### onScheduleTimeout [02-timeouts.lean](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/02-timeouts.lean), lines 88–97. Fires when a schedule's fire time arrives. Runs the `catchUp` loop (which fires `promiseCreate` for every missed occurrence), writes the updated schedule, disarms the current timeout entry, and arms a new one at `nextRunAt`. See [Implementing schedules](/server/implementing-schedules) for the full catch-up semantics. ## The projection convention This is the hardest property to reason about correctly, and the one most likely to bite you if you skip it. Several handlers return a promise's state computed from the stored record and the current clock — **without writing the computed state to storage**. This is the projection convention. It appears in `promise.get`, `promise.create` (when the promise already exists), and `promise.settle`. All three follow the same pattern: > If a stored promise has `state = pending` and `timeoutAt ≤ now`, the handler returns a projected settled record — `resolved` for timer promises (`resonate:timer = "true"`), `rejectedTimedout` for all others, with `settledAt = p.timeoutAt` — **without writing that record to storage**. The actual write to storage happens only when `onPromiseTimeout` fires. ```text Promise created: { state: pending, timeoutAt: T } | ... time passes, no timeout handler fired yet ... | now = T+1 | ▼ promise.get returns: { state: rejectedTimedout, settledAt: T } storage still holds: { state: pending, timeoutAt: T } | now = T+N | ▼ onPromiseTimeout fires: writes { state: rejectedTimedout, settledAt: T } sends unblock to listeners enqueues resumes for callbacks ``` ### Where settlement is actually written Settlement is written to storage in exactly three places, all of which run the same settlement scrub afterward: 1. **`promiseSettle`** ([P-03](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-03-promise.settle.lean)) — direct settle by an API caller, when `p.timeoutAt > now`. If the promise is already expired (`p.timeoutAt ≤ now`), `promiseSettle` returns `200` with the projected state and does **not** write. 2. **`taskFulfill`** ([T-07](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-07-task.fulfill.lean)) — task completes and settles its promise atomically, when `p.state == pending && p.timeoutAt > now`. If the promise is expired, `taskFulfill` returns `409` — the task cannot fulfill against an already-projected promise. 3. **`onPromiseTimeout`** — the background transition that is the only writer for expired promises. This is what actually delivers the settlement scrub, listener notifications, and callback resumes for timed-out promises. ### Implementation consequences **Your storage layer must not assume read-your-writes symmetry on promise state.** A query for "all settled promises with `settledAt < T`" will miss promises that have logically expired but haven't been processed by `onPromiseTimeout` yet. A query for "all pending promises" may include promises that would project as settled to any handler that reads them. Both queries are internally consistent — they reflect different moments in the machine — but you must be explicit about which you're asking and why. **The timeout machinery is a correctness component, not a janitor.** If `onPromiseTimeout` never fires: - `promise.get` returns correct results — projection handles it. - Listeners never receive their `unblock` message. - Awaiting tasks never receive their resume. - Work silently stalls — no error, no crash, nothing happening. An implementation that skips or indefinitely delays promise timeout processing passes every read-path test and fails every durability test. Build the timeout dispatch loop with the same care as the write handlers, and test it explicitly: create a promise with a short `timeoutAt`, let it expire, verify that the listener receives `unblock` and the awaiting task transitions to `pending`. If your storage layer builds an index over the `state` field for efficient lookups and you answer `promise.get` from that index, expired promises will return `pending` when they should return `rejectedTimedout`. Projection must happen in handler logic — after the read, before the response — not in the index. Any pending promise whose `timeoutAt ≤ now` must be projected at the point of response construction, even if the storage record still says `pending`. ## Cross-references - [Implementing promises](/server/implementing-promises) — `promiseSettle`, `promise.create`, and where promise timeouts are armed - [Implementing tasks](/server/implementing-tasks) — `taskFulfill`, version increment on re-acquire, lease and retry timeout lifecycle - [Implementing schedules](/server/implementing-schedules) — `onScheduleTimeout` and the catch-up loop - [Outbox and delivery](/server/outbox-and-delivery) — `unblock` and `execute` messages emitted by timeout transitions - [State model](/server/state-model) — `PromiseTimeout`, `TaskTimeout`, `ScheduleTimeout` struct definitions --- *Verified against [resonate-specification`@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3).* --- --- url: /server/unspecified-surface title: Unspecified surface: what the spec deliberately leaves open --- # Unspecified surface: what the spec deliberately leaves open The Lean spec defines a precise, bounded surface; everything outside it is deliberately left open. This chapter lists every such area — with what the spec says (or explicitly doesn't), what the reference server does (labeled as such, verified in source), and what you are free to choose. Check your design against this list before you present any implementation decision as "spec-required." See also: [Testing and conformance](/server/testing-and-conformance) — how to build conformance tests that stay on the specified side of this boundary. ## The three search handlers: P-06, T-11, S-04 **What the spec says.** The Lean files are unambiguous: ```lean -- P-06-promise.search.lean def promiseSearch (_req : PromiseSearchReq) (_now : Nat) : M PromiseSearchRes := do return { status := 501 } -- T-11-task.search.lean def taskSearch (_req : TaskSearchReq) (_now : Nat) : M TaskSearchRes := do return { status := 501 } -- S-04-schedule.search.lean def scheduleSearch (_req : ScheduleSearchReq) (_now : Nat) : M ScheduleSearchRes := do return { status := 501 } ``` The Lean handlers for P-06, T-11, and S-04 ignore their request arguments entirely and return `501`. Filter semantics, pagination, sort order, cursor format — none of this is specified. Sources: [`spec/02-actions/P-06-promise.search.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/P-06-promise.search.lean), [`T-11-task.search.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/T-11-task.search.lean), [`S-04-schedule.search.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/02-actions/S-04-schedule.search.lean). **What the reference server does.** The reference server implements all three with cursor-based pagination, optional state filter, and tag containment filter (all provided tags must be present on the returned object). Default page size is 100 for promises and tasks, 10 for schedules; maximum is 1000. Results are sorted by `id` ascending; the cursor is the `id` of the last returned item. This is reference-server convention, not spec behavior. **What you are free to choose.** Any filter semantics, any pagination scheme, any sort order, any cursor format. You may return `501` and be fully conformant. If you implement search, document what your implementation provides — no client should assume they will get reference-server semantics from a non-reference server. ## `nextCron` and `expand`: declared opaque **What the spec says.** [`spec/01-objects/state.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/01-objects/state.lean) declares both functions with no body: ```lean opaque nextCron : (cron : String) → (after : Nat) → Nat opaque expand : (template id : String) → (timestamp : Nat) → String ``` `nextCron` computes the next cron fire time after a given epoch-millisecond timestamp. `expand` turns a promise-id template and a schedule id into a concrete promise id at a given timestamp. The `opaque` keyword in Lean means: the function exists and has this type, but its definition is hidden — the spec makes no claims about the output for any input. Every handler that uses them (`schedule.create` for `nextCron`, the `catchUp` loop for both) is parameterized by whatever your implementation provides. **What the reference server does.** `oracle.rs` calls `util::compute_next_cron()` for cron advancement and uses a simple template-substitution expansion: `promise_id_template.replace("{{.id}}", &schedule_id).replace("{{.timestamp}}", ¤t_timeout.to_string())`. The cron parser is a standard Go/Rust library call; the expansion is literal string substitution with two named placeholders. **What you are free to choose.** Any standards-compliant cron parser and any cron dialect. Any template expansion format — your promise-id templates can use different placeholder syntax. The only constraint is internal consistency: whatever `expand` produces must be a valid promise id, and `nextCron` must return a timestamp strictly after its `after` argument (otherwise the `catchUp` loop does not terminate). The `catchUp` transition in `02-timeouts.lean` advances `nextRunAt` by calling `nextCron s.cron cronTime` in a loop until it surpasses `now`. If your `nextCron` implementation ever returns a value ≤ `after`, this loop does not terminate. Validate that your cron library always steps forward. ## `preload`: present in wire types, semantics unspecified **What the spec says.** Four response types carry a `preload` field in [`spec/01-objects/types.lean`](https://github.com/resonatehq/resonate-specification/blob/main/spec/01-objects/types.lean): ```lean structure TaskCreateRes where ... preload : List PromiseRecord := [] -- line 180 structure TaskAcquireRes where ... preload : List PromiseRecord := [] -- line 194 structure TaskFenceRes where ... preload : List PromiseRecord := [] -- line 216 structure TaskSuspendRes where ... preload : List PromiseRecord := [] -- line 241 ``` Every handler in the spec that returns one of these types leaves `preload` at the empty-list default. No Lean handler assigns a non-empty list to it. The field exists in the type; its semantics do not. **What the reference server does.** The reference server populates `preload` in its `preload()` helper (`oracle.rs` lines 2109–2129): it reads the `resonate:branch` tag from the acquired promise's tags and returns all other promises that share the same branch value. The intent is to give the worker a warm cache of sibling promises it will probably need, saving round trips. This optimization is entirely reference-server behavior — not a protocol requirement. **What you are free to choose.** Return an empty list and you are fully conformant. Populate it with any set of promise records that would be useful to the receiving worker. No client should assume that `preload` is always empty, always populated, or populated by any particular selection criterion. If your implementation populates it, document what you include and why. ## Authentication and authorization **What the spec says.** Nothing. The abstract machine has no auth model. Every handler takes a request and a current time; there is no principal, no token, no permission check anywhere in the Lean spec. The machine is not multi-tenant and does not distinguish callers. **What the reference server does.** The reference server provides optional API-key authentication on HTTP requests. It is not part of the protocol and is not modeled. **What you are free to choose.** Any authentication and authorization scheme, or none. JWT, mTLS, API keys, IP allowlists, internal-network-only deployment — all are valid. None is required. ## Transport encoding **What the spec says.** The abstract machine models handlers as pure functions `Req → now → M Res`. It does not model HTTP. It does not define JSON. The envelope shape — the `kind`/`head`/`data` structure, the `corrId` field, the `version` string, the HTTP method and path — is not in any Lean file. **What the reference server does.** The reference server uses an HTTP/JSON transport with a specific envelope format. All existing Resonate SDKs speak this envelope. See [Envelope and transport](/server/envelope-and-transport) for the full definition. **What you are free to choose.** More than the Lean model constrains, less than the full protocol allows: the [prose spec's message-passing protocol](/spec/execution-model/message-passing#address-schemes) names HTTP as the required transport for a conformant implementation, with other schemes (NATS, Kafka, poll/SSE, gRPC, Unix sockets) optional additions. Within that, serialization details and any extra transports are yours. `resonate-on-nats` demonstrates the trade: it speaks NATS pub/sub instead of HTTP, which is exactly why existing SDKs cannot talk to it without a matching transport layer — plan for that if you deviate. ## Concurrency and isolation **What the spec says.** The abstract machine is a sequential state machine. Its computation type is `StateM ServerState` — a single-threaded state thread. There is no concurrency primitive, no lock, no transaction isolation level anywhere in the Lean model. **What this means for your implementation.** Your server must behave *as if* it were executing operations one at a time in some sequential order. Concurrent requests are fine as long as the observable results — responses, state snapshots — are explainable by *some* valid sequential execution of the abstract machine. This is the standard linearizability criterion. See [Abstract machine](/server/abstract-machine) for the state model your concurrent implementation must serialize against. **What you are free to choose.** Any concurrency model: single-threaded event loop, thread-per-request with mutex, MVCC, Paxos-based LWT (as `resonate-on-scylladb` uses), JetStream KV CAS (as `resonate-on-nats` uses). The test that matters is whether concurrent executions produce a linearizable history against the abstract machine — not whether you used a specific locking primitive to achieve it. Every task mutation is version-fenced. If two workers race to acquire the same task, one wins the version increment and the other gets `409 Conflict`. You do not need to serialize the entire server to prevent double-acquisition; you need an atomic compare-and-swap on the task's version field. That is the minimal isolation unit the protocol requires. --- *Verified against [`resonate-specification@83d64c3`](https://github.com/resonatehq/resonate-specification/tree/83d64c3).* --- --- url: /spec/execution-model/coordination-protocol title: Distributed Coordination Protocol --- The **Distributed Coordination Protocol** is responsible for the coordination of distributed function executions. ## Eventual resumption Eventual resumption occurs when the calling execution E1 awaits a promise P2 that is not yet resolved. The calling execution E1 is suspended until the promise P2 is resolved or rejected. ![Distributed Coordination Protocol — eventual resumption sequence](/images/spec/distributed-coordination-protocol-eventual-resumption-sequence.svg) 1. When the calling execution E1, identified by Durable Promise P1 and hosted on worker W1, invokes a function remotely, W1 sends a request to create a Durable Promise (P2) to the server S. 2. Upon receiving the request, the server S creates the durable promise P2. 3. The Server S sends a response to W1 which forwards P2 to E1. 4. The server S sends an Invoke(P2) message to process group GB, here delivered to worker W2. 5. Worker W2 spawns execution E2. 6. When the calling execution E1 awaits promise P2, W1 sends a request to register a callback Resume(P1, P2) deliverable to GA with preference W1 on P2 to server S. 7. Upon receiving the request, since P2 is still pending, server S registers the callback C1 with promise P2. 8. Server S sends a response indicating success to worker W1, which suspends E1. 9. When the called execution E2 returns, the worker W2 sends a request to settle durable promise P2 to the server S. 10. Upon receiving the request, the server S settles the durable promise P2. 11. Server S sends a response containing P2 to W2. 12. The server S sends a Resume(P1, P2) message to process group GA with the preference of W1, here delivered to worker W1, which resumes E1. ## Immediate resumption Immediate resumption occurs when the calling execution E1 awaits a promise P2 that is already resolved. ![Distributed Coordination Protocol — immediate resumption sequence](/images/spec/distributed-coordination-protocol-immediate-resumption-sequence.svg) 1. When the calling execution E1, identified by durable promise P1 and hosted on worker W1, invokes a function remotely, W1 sends a request to create a durable promise P2 to the server S. 2. Upon receiving the request, the server S creates the durable promise P2. 3. Server S sends a response to W1, which forwards P2 to E1. 4. The server S sends an Invoke(P2) message to process group GB, here delivered to worker W2. 5. Worker W2 spawns execution E2. 6. When the called execution E2 returns, the worker W2 sends a request to settle durable promise P2 to the server S. 7. Upon receiving the request, the server S settles the durable promise P2. 8. Server S sends a response containing P2 to W2. 9. When the calling execution E1 awaits promise P2, W1 sends a request to register a callback Resume(P1, P2) deliverable to GA with preference W1 on P2 to server S. 10. Upon receiving the request, since P2 is already completed, server S **does not** register a callback. 11. Server S sends a response indicating the callback was not registered because the promise is already completed to worker W1, which resumes E1. --- --- url: /spec/execution-model title: Execution model --- The execution model lays out the algorithms, protocols, and object specifications that make Distributed Async Await work at a platform level. The execution model is structured into three protocols: - **[Coordination Protocol](/spec/execution-model/coordination-protocol)** — coordination of distributed function executions across partial order. - **[Recovery Protocol](/spec/execution-model/recovery-protocol)** — detection and recovery of crash failures across partial failure. - **[Message Passing Protocol](/spec/execution-model/message-passing)** — the underlying transport-layer contract for sending and receiving between processes. --- --- url: /spec/execution-model/message-passing title: Message Passing Protocol --- The **Message Passing Protocol** defines how components in Distributed Async Await communicate — the envelope every message uses, the addresses messages are sent to, and the transport variants the protocol rides on. The Message Passing Protocol is transport-agnostic. The same envelope flows over HTTP, NATS, Kafka, or an SSE poll connection — only the framing changes. The specification defines minimal requirements and minimal guarantees so a wide array of transports can implement it. ## Message events Messages are the means by which processes communicate — the primary mechanism for exchanging information and coordinating actions in a distributed system. Messages are sent between processes; consequently, messages are addressed to processes. One (successful) message exchange consists of a *send event* at the sending process and a *receive event* at the receiving process. ![Send and Receive event diagram](/images/spec/send-receive-event.svg) The most fine-grained model of message exchange between two processes consists of **four events**: a send event at the sending process, a receive event at the network, followed by a send event at the network and a receive event at the receiving process. At this level of abstraction, message exchange between processes and the network is reliable; the network itself may reorder, drop, or duplicate messages. The more common model of message exchange between two processes consists of **two events**: a send event at the sending process and a receive event at the receiving process. At this level of abstraction, message exchange between processes is unreliable — messages may be reordered, dropped, or duplicated. The only guarantee is that if a process experiences a receive event, there is a corresponding send event at a sending process (which may be the same process). ## Send Every component may issue a `Send(address, message)` command that emits a `Send(address, message)` event. ```text command Send(address : Address, message : T) ``` The Send command is the foundation of Distributed Async Await's failover mechanics: Send's anycast semantics enable both routing and transparent rerouting in case of failure. This allows distributed executions to adapt dynamically to changes in process topology. ### Invoke and Resume — the canonical interaction The canonical interaction in Distributed Async Await is invoking and awaiting an asynchronous function remotely (see the [Coordination Protocol](/spec/execution-model/coordination-protocol)). It involves two messages: - **Invoke** — the request to invoke an execution. - **Resume** — the request to resume an awaiting execution. Both are delivered as the on-the-wire envelope described below. ## Wire envelope All communication uses a uniform request/response envelope. The protocol is transport-agnostic — the envelope is the same regardless of whether the transport is HTTP, NATS, Kafka, or another binding. ### Request ```text { kind: string, // operation identifier, e.g. "promise.create" head: { corrId: string, // correlation ID for request/response matching version: string, // protocol version auth?: string // optional bearer token }, data: { ... } // operation-specific payload } ``` ### Response ```text { kind: string, // echoes the request kind head: { corrId: string, // echoes the request correlation ID status: integer, // status code (see below) version: string // protocol version }, data: { ... } // operation-specific payload, or error string on failure } ``` The `corrId` is echoed unchanged in the response so a client multiplexing multiple in-flight requests over a single connection can pair each response to its request. The `version` field carries the protocol version the request is built against; the server may reject unsupported versions with `400`. On error (status `>= 400`), the response `data` is a string describing the error. ### Status codes | Code | Meaning | |------|---------| | 200 | Success | | 300 | Continue — for `task.suspend`, an awaited promise has already settled; the worker should resume immediately without suspending | | 400 | Bad request (validation error or unsupported protocol version) | | 401 | Unauthorized (missing or invalid auth token) | | 403 | Forbidden (valid token but insufficient access) | | 404 | Not found | | 409 | Conflict (version mismatch, invalid state transition, or fence check failed) | | 422 | Unprocessable entity (e.g., awaiter not found, missing required tag) | | 429 | Too many requests (rate limited) | | 500 | Internal server error | | 501 | Not implemented | ## Reserved tags Promises carry a `tags` map. The protocol reserves the `resonate:` namespace for tags that drive routing, scheduling, and the execution tree. | Tag | Effect | |-----|--------| | `resonate:target` | Delivery address for execute messages. Triggers task creation on `promise.create`. | | `resonate:timer` | When `"true"`, timeout transitions to `resolved` instead of `rejected_timedout`. | | `resonate:origin` | Identifies the root promise that initiated the execution. All promises in an execution tree share the same `resonate:origin` value. | | `resonate:parent` | Identifies the direct parent promise in the execution tree. | | `resonate:branch` | Identifies the current execution branch. Set when a promise in an execution tree has a `resonate:target` tag. Drives preload semantics. | | `resonate:delay` | When present on a `promise.create` request that also carries a `resonate:target` tag, the server parses the value as a timestamp (natural number). If the delay value is greater than `now`, the server arms the task's retry timeout at `delay` rather than emitting an immediate `execute` message, effectively scheduling the execution for a future time. | User-defined tags outside the `resonate:` namespace are opaque to the protocol and may be used freely. ## Addressing An **address** is a URI that determines which transport to use, where to deliver the message, and the delivery semantics. Addresses appear on the `resonate:target` tag of a promise (for execute messages) and on listener registrations (for unblock messages). ### Address format ```text scheme[+connection]://[cast@]host[:port][/path][?query] ``` The scheme selects the transport. The optional `+connection` suffix selects a named connection for that transport — when absent, the transport uses the connection named `default`. The remainder of the URI is transport-specific. ### Address schemes | Scheme | Transport | Description | |---|---|---| | `http://`, `https://` | HTTP | Deliver via HTTP POST to the given URL | | `poll://` | Poll | Deliver via Server-Sent Events to a connected poll client | | `nats://` | NATS | Deliver via NATS publish to a derived subject | | `kafka://` | Kafka | Deliver via Kafka produce to the given topic | HTTP is the required transport. Every conformant implementation must support `http://` for both receive and send. ### Poll address ```text poll://cast@group[/id] ``` | Component | Required | Description | |---|---|---| | `cast` | Yes | Delivery mode: `uni` (unicast) or `any` (anycast) | | `group` | Yes | Logical group name that workers register under | | `id` | No | Specific worker id within the group | The poll transport does not push messages over the network. Workers connect to the server via HTTP GET and receive messages as Server-Sent Events; the server holds the connection open and writes messages as `data: {json}\n\n` frames. Examples: - `poll://any@workers` — deliver to any one connected worker in the `workers` group - `poll://uni@workers/worker-1` — deliver only to the worker with id `worker-1` - `poll://any@my-service/abc123` — prefer worker `abc123`, fall back to any worker in `my-service` (sticky routing with fallback) ## Delivery semantics The delivery semantics of `Send` depend on the transport and the address. | Mode | Definition | |---|---| | **Unicast** | The message is delivered to exactly one specific worker. If that worker is not available, delivery fails (the message remains in the outbox for retry). | | **Anycast** | The message is delivered to exactly one worker from a group. The server selects the recipient. If a preferred worker is specified and available, the server picks it; otherwise, a random worker in the group. | | **Broadcast** | The message is delivered to all workers listening on the address. Each worker receives a copy. | ### Per-transport delivery | Transport | Recv | Send | Delivery model | |---|---|---|---| | HTTP | yes | yes | Point-to-point — one URL, one endpoint | | Poll | no | yes | Unicast (`uni@`) or anycast (`any@`) by address | | NATS | yes | yes | Queue-group subscription = unicast; plain subscription = broadcast | | Kafka | yes | yes | Kafka consumer-group semantics | Poll is a `send`-only transport — workers receive messages over SSE, but the requests they originate still ride a `recv`-capable transport (typically HTTP POST). ## Address resolution This specification does not determine *how* and *when* addresses are resolved. Specifically, it does not determine whether addresses are resolved on send (early binding) or on recv (late binding). Conformant implementations may choose either model. --- --- url: /spec/execution-model/recovery-protocol title: Recovery Protocol --- The **Distributed Recovery Protocol** is responsible for the detection and recovery of crash failures. ## What recovery means Recovery is the mechanism by which a [logical execution](/spec/system-model/executions) survives the [failure](/spec/system-model/processes#failure) of a physical process. Distributed Async Await assumes a fail-stop process failure model: a process may non-deterministically terminate without warning. When that happens, the logical execution suspends and a successor physical execution can resume the work. Recovery is the second of the two foundational protocols in Distributed Async Await: - The [Coordination Protocol](/spec/execution-model/coordination-protocol) ensures coordination across **partial order** — the non-deterministic interleaving inherent to concurrent systems. - The Recovery Protocol ensures recovery across **partial failure** — the non-deterministic termination inherent to distributed systems. ## Three load-bearing pieces The recovery semantics described below rest on three load-bearing pieces: 1. **Durable Promises persist execution state.** The state required to recreate a function execution is stored externally to any single process. See the [Durable Promise Specification](/spec/programming-model/durable-promise-specification). 2. **Function executions are interruption-tolerant.** Re-running a durable function from the beginning, with deduplication via durable promises, converges to the same result as an uninterrupted execution. See the [Durable Function Specification](/spec/programming-model/durable-function-specification). 3. **Tasks claim work and report liveness.** Workers claim tasks from the durable system, heartbeat to indicate progress, and surrender claims on crash. See the [Tasks](/spec/system-model/tasks) page for the full lifecycle. ## Lease and heartbeat Liveness is enforced by a lease. When a worker acquires a task, the server records a deadline by which the worker must signal it is still making progress. Each successful heartbeat extends that deadline. ```text acquire → lease = now + leaseTimeout, version++ heartbeat → lease = now + leaseTimeout release → lease deleted, transition back to pending (version unchanged) lease expiry → transition back to pending (version unchanged), re-enqueue execute ``` Heartbeat is **process-level, not task-level**: a worker sends one heartbeat for all the acquired tasks it holds, and the server refreshes every matching `(id, version)` pair in a single round-trip. Workers typically heartbeat at roughly half the lease interval to absorb transient network delay. If a worker crashes — or loses the ability to make progress — its lease expires. The server transitions the task back to `pending` (without changing the version), deletes the lease, and re-enqueues the execute message. A successor worker calls `task.acquire`, which increments the version, and any in-flight operation from the prior worker will then fail with 409 on its next mutating call. The version is the optimistic concurrency control (OCC) token: every mutating task operation must present the current version, and the server returns `409 Conflict` on mismatch. This guarantees a worker that has lost the lease cannot still effect a settle or a fulfill — the new version on the task means the late operation collides with whatever the successor is doing. ## The settlement chain Every promise settlement — whether triggered by `promise.settle`, `task.fulfill`, `task.fence`, or a server-driven timeout — proceeds through the same three stages, each producing a distinct kind of effect. We call this the **settlement chain** because the stages form a linear causal order: ```text settle → resume → execute ``` 1. **Settle (intra-object).** The composite settles itself. The promise transitions to its terminal state, `value` and `settledAt` are recorded, the promise timeout is deleted, the task (if present) transitions to `fulfilled`, and the callbacks and listeners are consumed. *External side effect:* one **unblock** message is sent to each listener address. 2. **Resume (inter-object, internal side effect).** For each callback that was registered on the settled promise, enqueue a resume on the awaiter's task — mutating another composite's task state within the server. 3. **Execute (external side effect).** When the resume transitions an awaiter task from `suspended` to `pending`, the server sends an **execute** message to that task's `resonate:target` address. The chain is linear, not recursive. Settling composite X does not settle any other composite. It may resume awaiter tasks, which may eventually lead those composites to settle through their own future operations — but not as part of this settlement. The essential guarantee: **when a promise settles, all dependents are reliably notified.** ## Properties a conformant implementation must guarantee A Recovery Protocol implementation must satisfy: - **Liveness under failure.** A logical execution's progress is bounded by the availability of *some* worker, not any specific worker. If a worker crashes, a successor must be able to resume. - **Safety under recovery.** A re-invoked durable function must not externalize observable side effects twice. Side-effect deduplication is provided by the durable promises representing each step. - **Bounded re-execution.** A worker that has lost the ability to make progress must yield its task claim within a bounded time so a successor can take over without indefinite waiting. The lease enforces this bound. ## Server invariants Recovery rests on structural invariants the server maintains over its promise, task, and callback stores at all times. A violation indicates a bug in the implementation, not a recoverable runtime condition. ### Promise invariants | Invariant | Condition | |---|---| | `orphan_invokes` | Every pending promise with `resonate:target` must have a task. | | `missing_ptimeout` | Every pending promise must have a promise timeout entry. | | `stale_ptimeout` | No promise timeout should exist for a non-pending promise. | | `timedout_settled_at_mismatch` | Every `rejected_timedout` promise must have `settledAt == timeoutAt`. | | `listener_for_settled_promise` | No listener should exist for a non-pending promise. | ### Task invariants | Invariant | Condition | |---|---| | `orphan_tasks` | Every task must have a corresponding promise. | | `pending_task_no_ttimeout` | Every pending task must have a retry timeout. | | `acquired_task_no_lease` | Every acquired task must have a lease. | | `suspended_no_callback` | Every suspended task must have at least one callback registered on an awaited promise. | | `suspended_with_consumed_callbacks` | No suspended task should have any consumed callbacks. | | `suspended_task_has_ttimeout` | No suspended task should have any timeout. | | `fulfilled_task_has_ttimeout` | No fulfilled task should have any timeout. | ### Callback invariant | Invariant | Condition | |---|---| | `callback_awaiter_no_target` | Every callback's awaiter promise must have a `resonate:target` tag. | ## See also - [Coordination Protocol](/spec/execution-model/coordination-protocol) — the sibling protocol covering partial order. - [Message Passing Protocol](/spec/execution-model/message-passing) — the underlying transport layer including the wire envelope and address scheme. - [Tasks](/spec/system-model/tasks) — the lifecycle and operations recovery operationalizes. - [Processes](/spec/system-model/processes) — the failure model assumed by recovery. --- --- url: /spec/glossary title: Glossary --- ### Activation A single physical attempt to run a function. A logical execution may consist of multiple activations across recoveries — each one starts from the beginning, replays past completed steps using recorded results, and attempts to make further progress. Synonym for [Physical Execution](#physical-process). See [Function Executions](/spec/system-model/executions). ### Address A URI identifying where a message should be delivered. Addresses appear on the `resonate:target` tag of a promise (for execute messages) and on listener registrations (for unblock messages). The address scheme selects the [transport](/spec/execution-model/message-passing#address-schemes) (for example `http://`, `poll://`, `nats://`, `kafka://`). ### Agent A program that generates and executes code on behalf of a user — including, increasingly, code that calls Distributed Async Await APIs. From the perspective of the protocol, an agent is just another process. See [Process](#process). ### Anycast A delivery mode in which a message is sent to exactly one worker drawn from a group, with optional preference for a named worker. Contrast with [Unicast](#unicast). See [Message Passing Protocol](/spec/execution-model/message-passing#delivery-semantics). ### Application Node A process that hosts function executions and communicates with the server to coordinate them. The first execution on an application node is its [Root Promise](/spec/programming-model/#root-promises). [Reference SDKs](https://docs.resonatehq.io/develop) typically expose this as the entry point invoked by a top-level `run` call. ### Async Await Async Await is a programming model based on functions and promises that allows for concurrent execution of code within a single process. ### Async Call Graph A [Call Graph](#call-graph) in which the edges represent asynchronous invocations — that is, the caller continues executing concurrently with the callee, and the caller awaits a promise to receive the callee's result. ### Call Graph A Call Graph is a directed graph where the nodes are functions and the edges are calls between functions. See also: - [Async Call Graph](#async-call-graph) ### Callback A registration on a pending promise that, when the promise settles, enqueues a resume on the awaiter's task. Callbacks are the link between settlement and the [Settlement Chain's](#settlement-chain) resume step. See [Recovery Protocol](/spec/execution-model/recovery-protocol#the-settlement-chain). ### Concurrency The property of a system in which multiple executions may take a step at the same point in time. Concurrency introduces *partial order* — non-deterministic interleaving of steps across executions. See also [Coordination](#coordination). ### Coordination The mechanism by which concurrent executions agree on a partial order of operations. In Distributed Async Await, coordination is provided by [Promises](#promise) — a caller and callee execute concurrently, and the caller awaits a promise to receive the callee's result. See also [Concurrency](#concurrency). ### Definition The static description of a computation — the program text. A definition becomes an [Execution](#execution) when it is invoked. See also [Execution](#execution). ### Distribution The property of a system in which executions span multiple [Processes](#process), each with its own state and its own potential for [Failure](#interruption). Distribution introduces *partial failure* — non-deterministic termination of a subset of processes. See also [Recovery](#recovery). ### Durable Executions A Durable Execution is a programming abstraction with an interruption-agnostic definition resulting in an interruption-transparent execution. ### Envelope The uniform request/response container all protocol messages use: a `kind` (operation identifier), a `head` (correlation id, protocol version, optional auth or status), and a `data` payload (operation-specific). See [Message Passing Protocol](/spec/execution-model/message-passing#wire-envelope). ### Execution The dynamic instance of a [Definition](#definition) — the running program, with all its in-flight state. An execution has a well-defined lifecycle: *invoke* (entry) and *return* (exit). See also [Definition](#definition). ### Fence A version-guarded operation that atomically runs either a `promise.create` or a `promise.settle` only if the task is still acquired at the presented version. Used when a worker needs to confirm it is the unique current claimant before externalizing a side effect. See [Tasks](/spec/system-model/tasks#operations). ### Function A unit of computation defined by its inputs, its outputs, and its body. In Distributed Async Await, functions compose recursively — a function may call other functions, spanning a [Call Graph](#call-graph). Together with [Promises](#promise), functions are one of two universal abstractions on which the programming model is built. ### Heartbeat A periodic process-level signal a worker sends to extend the [Lease](#lease) on every task it currently holds. The cadence is implementation-defined; workers typically heartbeat at roughly half the lease interval. See [Recovery Protocol](/spec/execution-model/recovery-protocol#lease-and-heartbeat). ### Idempotency Key An optional opaque value attached to a [Promise](#promise) operation that allows the server to deduplicate a retry from an unknown-outcome request. Two keys are tracked per promise: `ikc` (idempotency key for create) and `iku` (idempotency key for update). See the [Durable Promise Specification](/spec/programming-model/durable-promise-specification#state-transitions) state-transition table. ### Interruption The term _interruption_ refers to a voluntary (system-triggered) or involuntary (environment-triggered) termination mid execution. ### Interruption-agnostic A property of a [Definition](#definition): the definition (program, code) does not contain interruption detection or interruption mitigation logic. The author writes ordinary, synchronous-looking code; the protocol takes responsibility for handling interruptions transparently. ### Interruption-tolerant A property of an [Execution](#execution): an execution that experiences an interruption and subsequently recovers is equivalent to some execution that does not experience an interruption. Formally, `(⟨p⟩, →(+interruption)) ≃ (⟨p⟩, →(-interruption))`. See the [Durable Function Specification](/spec/programming-model/durable-function-specification) for the formal treatment. ### Lease A deadline by which a [Worker](#worker) must signal liveness on an acquired [Task](#task). Each successful [Heartbeat](#heartbeat) extends the lease. If the lease expires, the server releases the task back to `pending` (version unchanged) and re-enqueues the execute message; a successor worker's `task.acquire` will then increment the [Version](#version). See [Recovery Protocol](/spec/execution-model/recovery-protocol#lease-and-heartbeat). ### Listener An external address registered to receive an *unblock* message when a [Promise](#promise) settles. Distinct from a [Callback](#callback): a listener is consumed by external delivery, a callback is consumed by internal resume. See [Message Passing Protocol](/spec/execution-model/message-passing). ### Logical Process A [Process](#process) identified by its logical identity — what the system reasons *about*. One logical process is realized by one or more [Physical Processes](#physical-process) over time; when a physical process crashes, the logical process suspends, and a successor physical process resumes it. ### Messages Messages are the means by which processes communicate with each other. ### Physical Process A concrete OS-level process that runs on a specific machine for a finite lifetime. A physical process realizes a [Logical Process](#logical-process); a single logical process may be realized by a sequence of physical processes across crashes and restarts. ### Process Processes are the **fundamental unit of locality** in a system, each with a unique identity and a well-defined lifecycle. See also: - [Logical Process](#logical-process) - [Physical Process](#physical-process) ### Process Group A logical group name that workers register under. Used by the [Anycast](#anycast) and [Unicast](#unicast) addressing modes to identify the recipient set for a message. See [Message Passing Protocol](/spec/execution-model/message-passing#poll-address). ### Programming Model The developer-facing surface of a system: the abstractions that authors compose to express computation. Distributed Async Await's programming model is built on [Functions](#function) and [Promises](#promise) — the same primitives as ordinary async/await, lifted to span [Distribution](#distribution). ### Promise A representation of a future value. A promise is either *pending* or *completed*; a completed promise is *resolved* (success), *rejected* (failure), *rejectedCanceled* (canceled), or *rejectedTimedout* (timed out). Promises are the universal abstraction for [Coordination](#coordination) — a caller awaits a promise to receive a callee's result. In Distributed Async Await, promises are *durable*: they persist independently of the processes that reference them. See the [Durable Promise Specification](/spec/programming-model/durable-promise-specification) for the formal API and state-transition table. ### Recovery The mechanism by which a [Logical Process](#logical-process) survives the [Failure](#interruption) of a [Physical Process](#physical-process). Recovery is the response to *partial failure*, just as [Coordination](#coordination) is the response to *partial order*. See the [Recovery Protocol](/spec/execution-model/recovery-protocol) for details. ### Reserved Tag A tag in the `resonate:` namespace whose meaning is defined by the protocol — for example `resonate:target` (delivery address), `resonate:timer` (timeout-as-resolved), `resonate:origin` (root promise of an execution tree). See [Message Passing Protocol](/spec/execution-model/message-passing#reserved-tags). ### Server The component that owns durable promise and task state, runs the [Settlement Chain](#settlement-chain), enforces protocol [Invariants](/spec/execution-model/recovery-protocol#server-invariants), and routes messages between [Workers](#worker). Reference implementation: the open-source [Resonate Server](https://github.com/resonatehq/resonate). ### Settlement Chain The linear sequence of effects triggered by every promise settlement: *settle* → *resume* → *execute*. Settling a promise in one composite may cause resumes on awaiter tasks, which may cause execute messages to other workers — but the chain is linear, not recursive. See [Recovery Protocol](/spec/execution-model/recovery-protocol#the-settlement-chain). ### Task The unit of work the server delivers to a worker. A task and its associated promise share an identifier; the promise owns the *value*, the task owns the *claim*. Tasks have a lifecycle (`pending`/`acquired`/`suspended`/`halted`/`fulfilled`), a [Version](#version), and a [Lease](#lease). See [Tasks](/spec/system-model/tasks). ### Unicast A delivery mode in which a message is sent to exactly one specific worker by id. If that worker is not available, delivery fails and the message remains in the outbox for retry. Contrast with [Anycast](#anycast). See [Message Passing Protocol](/spec/execution-model/message-passing#delivery-semantics). ### Version The optimistic concurrency control (OCC) token on a [Task](#task). The version increments only on the `pending → acquired` transition (`task.acquire`); transitions back to `pending` (release, resume, lease expiry) do not change the version. All mutating task operations must present the current version; a mismatch returns `409 Conflict`. See [Tasks](/spec/system-model/tasks#lifecycle). ### Worker A process that claims [Tasks](#task) from the [Server](#server), executes the associated function, and reports liveness via [Heartbeat](#heartbeat). A worker can host multiple acquired tasks at once. Reference SDKs ([TypeScript](https://docs.resonatehq.io/develop/typescript), [Python](https://docs.resonatehq.io/develop/python), [Rust](https://docs.resonatehq.io/develop/rust), [Go](https://docs.resonatehq.io/develop/go), [Java](https://docs.resonatehq.io/develop/java)) embed worker logic alongside the application code. --- --- url: /spec title: Specification --- Specification section Specification section The Distributed Async Await protocol is specified twice, on purpose. [`resonatehq/resonate-specification`](https://github.com/resonatehq/resonate-specification) is the executable abstract machine in Lean 4 — the normative, machine-checkable definition of the protocol's core handlers and state transitions. These pages are the prose specification: the human-readable companion that explains the same protocol. Where prose and Lean disagree, the Lean model wins. For working code that implements the protocol, see the reference implementations: the [Resonate Server](https://github.com/resonatehq/resonate) and the [TypeScript](https://docs.resonatehq.io/develop/typescript), [Python](https://docs.resonatehq.io/develop/python), [Rust](https://docs.resonatehq.io/develop/rust), [Go](https://docs.resonatehq.io/develop/go), and [Java](https://docs.resonatehq.io/develop/java) SDKs. The specification is structured into three parts: - **[Programming model](/spec/programming-model)** — the intended structure of distributed programs and the developer interface. - **[Execution model](/spec/execution-model)** — how distributed programs reliably make progress. - **[System model](/spec/system-model)** — the environment in which distributed programs exist, and a way of thinking about them. A [glossary](/spec/glossary) defines the load-bearing terms. --- --- url: /spec/programming-model/durable-function-specification title: Durable Function Specification --- Durability is a property of a function execution that allows it to be resumed after an interruption. ## Durable executions A durable execution is a programming abstraction with an interruption-agnostic *definition* resulting in an interruption-transparent *execution*. The defining characteristic of durable executions is that they are both interruption-agnostic and interruption-transparent — being only one is not sufficient. ### Interruption The term _interruption_ refers to a voluntary (system-triggered) or involuntary (environment-triggered) termination mid execution. A voluntary termination is also referred to as an _interrupt_; an involuntary termination is also referred to as a _failure_. ### Interruption-agnostic definition The term _interruption-agnostic definition_ refers to a definition (program, code) that does not acknowledge the possibility of interruptions. The definition does not contain interruption detection or interruption mitigation. ### Interruption-transparent execution The term _interruption-tolerant execution_ refers to an execution that does not externalize (make observable) the presence of interruptions. An execution that experiences an interruption and subsequently recovers is equivalent to some execution that does not experience an interruption. Interruption tolerance can be defined formally as: ``` (⟨p⟩, →(+interruption)) ≃ (⟨p⟩, →(-interruption)) ``` **In words.** A program `p` is interruption-tolerant if, starting from an initial configuration `⟨p⟩`, an execution in the presence of interruptions `(⟨p⟩, →(+interruption))` is equivalent to some execution in the absence of interruptions `(⟨p⟩, →(-interruption))`. ## Preconditions for the equivalence The equivalence above is conditional. For an execution to be interruption-tolerant in practice, the function must satisfy three constraints. ### 1. Determinism Same inputs always produce the same control-flow path. A durable function may be replayed from the beginning after a crash; if a second execution makes different decisions than the first — picks a different branch, reads a different timestamp, observes a different random value — replay is no longer equivalent to the original execution. In practice, determinism requires interception of every source of non-determinism the function consumes, including: - **Time** — wall-clock timestamps must be retrieved through a durable primitive so the recorded value is replayed, not re-sampled. - **Randomness** — pseudo-random values must be retrieved through a durable primitive for the same reason. - **External I/O** — any call whose result depends on the outside world (network requests, file reads, queue lookups) must be wrapped as a step whose result is recorded and replayed. The protocol does not prescribe the surface of these primitives; reference implementations expose them as context-bound calls (for example, `ctx.run(...)` for arbitrary I/O steps, plus typed helpers for clock and randomness). ### 2. Idempotency Side effects must be safe to retry. A durable function may execute the same step more than once across the lifecycle of a logical execution — once on the original physical execution, again on a successor after recovery. The function's externalized effects must converge regardless of how many times each step runs. The protocol provides one half of idempotency for free: every step is associated with a deterministically-derived [Durable Promise](/spec/programming-model/durable-promise-specification) id, and the state-transition table guarantees that subsequent attempts to settle a promise with a matching idempotency key are deduplicated rather than repeated. The other half — that the *content* of a step is itself replay-safe — is the function author's responsibility. ### 3. Activation lifetime A function execution cannot outlive the physical process that hosts it. This is a consequence of the [process lifecycle model](/spec/system-model/processes): events outside the `(init, term)` interval cannot be emitted. Long-running work — sleep, await for external input, multi-day workflows — must be expressed through protocol primitives (durable sleep, durable promises awaiting external settlement) so that the *logical* execution can outlive any one physical execution while no individual physical execution does. ## Why the constraints matter The three constraints are not arbitrary. They are the preconditions under which the formal interruption-tolerance equivalence holds: | Constraint | What breaks without it | |---|---| | Determinism | Replay produces a different control-flow path; the recovered execution is not equivalent to the original | | Idempotency | A retried step externalizes a side effect twice; the recovered execution is observably different from one without interruption | | Activation lifetime | A "long-running" execution cannot be recovered at all because no physical successor can take over from the dead one | Together they define what it means to write a function the protocol can recover. This page captures the constraints any conformant durable function must satisfy and the formal equivalence that defines interruption tolerance. A more rigorous treatment of step composition, durable side-effect semantics, and the formal relationship between durable function state and durable promise state is in progress. For the operational shape of durable functions in working systems, see the [Durable Promise Specification](/spec/programming-model/durable-promise-specification) (the state on which durable functions execute) and the [TypeScript](https://docs.resonatehq.io/develop/typescript), [Python](https://docs.resonatehq.io/develop/python), [Rust](https://docs.resonatehq.io/develop/rust), [Go](https://docs.resonatehq.io/develop/go), and [Java](https://docs.resonatehq.io/develop/java) SDKs (how each language realizes durable functions). --- --- url: /spec/programming-model/durable-promise-specification title: Durable Promise Specification --- This page specifies the full durable promise contract, including surface the Lean 4 abstract machine ([resonatehq/resonate-specification](https://github.com/resonatehq/resonate-specification)) does not model: the `ikc`/`iku` idempotency keys, the `strict` flag, and the state-transition table built on them. That extended surface comes from the legacy durable-promise contract and is implemented by the reference server. Where this page and the Lean model overlap and disagree, the Lean model wins. Promises are **fundamental units of coordination**. Distributed Async Await proposes Durable Promises — promises that persist in storage and enable coordination across process boundaries. Within the Distributed Async Await specification, each execution — whether a function execution or an action taking place in the physical world — pairs to a promise. Distributed Async Await requires that promise data objects are stored to disk, thus making the promises durable, giving them the name **Durable Promises**. A promise (also called future, awaitable, or deferred) is a representation of a future value. A promise is either pending or completed. A pending promise has no value yet; a completed promise carries one of four outcomes: resolved (success), rejected (failure), rejectedCanceled (canceled by a downstream caller), or rejectedTimedout (expired before settlement). ![Promise lifecycle diagram](/images/spec/promise-lifecycle.svg) A promise is a coordination primitive. In a typical scenario, a downstream function execution creates a promise and awaits its completion. An upstream function execution settles the promise. On completion, the downstream execution resumes with the value of the promise. ## API **Durable Promise Application Programming Interface** Logically, the Application Programming Interface (API) is divided into two parts: the Downstream API and the Upstream API. ![Promise API illustration](/images/spec/promise-api-illustration.svg) ### Downstream API - **Create** A downstream component may create a promise. ``` Create(promise-id, idempotency-key, param, header, timeout, strict) ``` - **Cancel** A downstream component can cancel an existing promise. ``` Cancel(promise-id, idempotency-key, value, header, strict) ``` - **Callback** A downstream component can register a callback on an existing promise. ``` Callback(id, promise-id, root-promise-id, timeout, recv) ``` A `recv` specifies the transport on which the callback will occur. Below is a non-exhaustive list of supported receivers. | Type | Data | Shorthand | | ------ | ---------------------------------------------------- | -------------------- | | `poll` | `{"group": "string", "id": "string"}` | `poll://group:id` | | `http` | `{"headers": {"string": "string"}, "url": "string"}` | `http://example.com` | ### Upstream API - **Settle** An upstream component settles an existing promise with a terminal state. A single `promise.settle` operation carries a `state` parameter that selects the outcome. ``` Settle(promise-id, idempotency-key, state, value, header, strict) ``` Where `state` is one of `resolved`, `rejected`, or `rejectedCanceled`. (By convention, `rejectedTimedout` is reserved for server-initiated timeout settlement.) The legacy API exposed these as separate operations — `Resolve(...)` and `Reject(...)` upstream, `Cancel(...)` downstream; the current wire protocol unifies all three into the single `promise.settle` handler. ### OpenAPI spec example ```yaml openapi: 3.0.0 info: title: Durable Promise API version: x.x.x license: name: Apache 2.0 url: https://opensource.org/license/apache-2-0 servers: - url: https://your_public_api_url description: - url: http://localhost: description: paths: /promises: get: tags: [Promises] summary: Search promises operationId: searchPromises parameters: - in: header name: request-id schema: { type: string } - name: id in: query schema: { type: string } - name: state in: query schema: { type: string, enum: [pending, resolved, rejected] } - name: tags in: query style: deepObject explode: true schema: type: object additionalProperties: { type: string } - name: limit in: query schema: { type: integer } - name: cursor in: query schema: { type: string } responses: 200: description: OK content: application/json: schema: type: object properties: promises: type: array items: $ref: "#/components/schemas/Promise" cursor: type: string post: tags: [Promises] summary: Create promise operationId: createPromise parameters: - in: header name: request-id schema: { type: string } - in: header name: idempotency-key schema: { type: string } - in: header name: strict schema: { type: boolean } requestBody: required: true content: application/json: schema: type: object required: [id, timeout] properties: id: { type: string } timeout: { type: integer, format: int64 } param: { $ref: "#/components/schemas/Value" } tags: type: object additionalProperties: { type: string } responses: 200: description: OK — returned for both new and existing promises; duplicate create is idempotent content: application/json: schema: { $ref: "#/components/schemas/Promise" } 400: { description: Invalid request } 403: { description: Forbidden } /promises/{id}: get: tags: [Promises] summary: Read promise operationId: readPromise parameters: - in: path name: id required: true schema: { type: string } - in: header name: request-id schema: { type: string } responses: 200: description: OK content: application/json: schema: { $ref: "#/components/schemas/Promise" } 400: { description: Invalid request } 404: { description: Promise not found } patch: tags: [Promises] summary: Complete promise operationId: completePromise parameters: - in: path name: id required: true schema: { type: string } - in: header name: request-id schema: { type: string } - in: header name: idempotency-key schema: { type: string } - in: header name: strict schema: { type: boolean } requestBody: required: true content: application/json: schema: type: object required: [state] properties: state: type: string enum: [RESOLVED, REJECTED, REJECTED_CANCELED] value: { $ref: "#/components/schemas/Value" } responses: 200: description: OK content: application/json: schema: { $ref: "#/components/schemas/Promise" } 400: { description: Invalid request } 403: { description: Forbidden } 404: { description: Promise not found } components: schemas: Promise: type: object required: [id, state, timeout, param, value, tags] properties: id: { type: string } state: type: string enum: [PENDING, RESOLVED, REJECTED, REJECTED_CANCELED, REJECTED_TIMEDOUT] timeout: { type: integer, format: int64 } param: { $ref: "#/components/schemas/Value" } value: { $ref: "#/components/schemas/Value" } tags: type: object additionalProperties: { type: string } idempotencyKeyForCreate: { type: string, readOnly: true } idempotencyKeyForComplete: { type: string, readOnly: true } createdOn: { type: integer, format: int64 } completedOn: { type: integer, format: int64 } Value: type: object properties: headers: type: object additionalProperties: { type: string } data: { type: string } Task: type: object required: [id, version, state] properties: id: { type: string } state: type: string enum: [pending, acquired, suspended, halted, fulfilled] version: { type: integer } timeout: { type: integer, format: int64 } processId: { type: string } createdOn: { type: integer, format: int64 } completedOn: { type: integer, format: int64 } ``` A task and its associated promise share an identifier; they are paired but distinct. A task carries a `version` that increments only on the `pending → acquired` transition (`task.acquire`); transitions back to `pending` (release, resume, lease expiry) do not change the version. All mutating task operations require the caller to present the current `version` — a mismatch returns conflict (status `409`). For the full task lifecycle, operations, and invariants, see [Tasks](/spec/system-model/tasks). ## State transitions | | Current State | Action | Next State | Output | | --- | ---------------------- | --------------------- | ---------------------- | -------------------- | | 1 | Init | Create(id, -, T) | Pending(id, -, -) | OK | | 2 | Init | Create(id, -, F) | Pending(id, -, -) | OK | | 3 | Init | Create(id, ikc, T) | Pending(id, ikc, -) | OK | | 4 | Init | Create(id, ikc, F) | Pending(id, ikc, -) | OK | | 5 | Init | Resolve(id, -, T) | Init | KO, Already Init | | 6 | Init | Resolve(id, -, F) | Init | KO, Already Init | | 7 | Init | Resolve(id, iku, T) | Init | KO, Already Init | | 8 | Init | Resolve(id, iku, F) | Init | KO, Already Init | | 9 | Init | Reject(id, -, T) | Init | KO, Already Init | | 10 | Init | Reject(id, -, F) | Init | KO, Already Init | | 11 | Init | Reject(id, iku, T) | Init | KO, Already Init | | 12 | Init | Reject(id, iku, F) | Init | KO, Already Init | | 13 | Init | Cancel(id, -, T) | Init | KO, Already Init | | 14 | Init | Cancel(id, -, F) | Init | KO, Already Init | | 15 | Init | Cancel(id, iku, T) | Init | KO, Already Init | | 16 | Init | Cancel(id, iku, F) | Init | KO, Already Init | | 17 | Pending(id, -, -) | Create(id, -, T) | Pending(id, -, -) | KO, Already Pending | | 18 | Pending(id, -, -) | Create(id, -, F) | Pending(id, -, -) | KO, Already Pending | | 19 | Pending(id, -, -) | Create(id, ikc, T) | Pending(id, -, -) | KO, Already Pending | | 20 | Pending(id, -, -) | Create(id, ikc, F) | Pending(id, -, -) | KO, Already Pending | | 21 | Pending(id, -, -) | Resolve(id, -, T) | Resolved(id, -, -) | OK | | 22 | Pending(id, -, -) | Resolve(id, -, F) | Resolved(id, -, -) | OK | | 23 | Pending(id, -, -) | Resolve(id, iku, T) | Resolved(id, -, iku) | OK | | 24 | Pending(id, -, -) | Resolve(id, iku, F) | Resolved(id, -, iku) | OK | | 25 | Pending(id, -, -) | Reject(id, -, T) | Rejected(id, -, -) | OK | | 26 | Pending(id, -, -) | Reject(id, -, F) | Rejected(id, -, -) | OK | | 27 | Pending(id, -, -) | Reject(id, iku, T) | Rejected(id, -, iku) | OK | | 28 | Pending(id, -, -) | Reject(id, iku, F) | Rejected(id, -, iku) | OK | | 29 | Pending(id, -, -) | Cancel(id, -, T) | RejectedCanceled(id, -, -) | OK | | 30 | Pending(id, -, -) | Cancel(id, -, F) | RejectedCanceled(id, -, -) | OK | | 31 | Pending(id, -, -) | Cancel(id, iku, T) | RejectedCanceled(id, -, iku) | OK | | 32 | Pending(id, -, -) | Cancel(id, iku, F) | RejectedCanceled(id, -, iku) | OK | | 33 | Pending(id, ikc, -) | Create(id, -, T) | Pending(id, ikc, -) | KO, Already Pending | | 34 | Pending(id, ikc, -) | Create(id, -, F) | Pending(id, ikc, -) | KO, Already Pending | | 35 | Pending(id, ikc, -) | Create(id, ikc, T) | Pending(id, ikc, -) | OK, Deduplicated | | 36 | Pending(id, ikc, -) | Create(id, ikc, F) | Pending(id, ikc, -) | OK, Deduplicated | | 37 | Pending(id, ikc, -) | Create(id, ikc\*, T) | Pending(id, ikc, -) | KO, Already Pending | | 38 | Pending(id, ikc, -) | Create(id, ikc\*, F) | Pending(id, ikc, -) | KO, Already Pending | | 39 | Pending(id, ikc, -) | Resolve(id, -, T) | Resolved(id, ikc, -) | OK | | 40 | Pending(id, ikc, -) | Resolve(id, -, F) | Resolved(id, ikc, -) | OK | | 41 | Pending(id, ikc, -) | Resolve(id, iku, T) | Resolved(id, ikc, iku) | OK | | 42 | Pending(id, ikc, -) | Resolve(id, iku, F) | Resolved(id, ikc, iku) | OK | | 43 | Pending(id, ikc, -) | Reject(id, -, T) | Rejected(id, ikc, -) | OK | | 44 | Pending(id, ikc, -) | Reject(id, -, F) | Rejected(id, ikc, -) | OK | | 45 | Pending(id, ikc, -) | Reject(id, iku, T) | Rejected(id, ikc, iku) | OK | | 46 | Pending(id, ikc, -) | Reject(id, iku, F) | Rejected(id, ikc, iku) | OK | | 47 | Pending(id, ikc, -) | Cancel(id, -, T) | RejectedCanceled(id, ikc, -) | OK | | 48 | Pending(id, ikc, -) | Cancel(id, -, F) | RejectedCanceled(id, ikc, -) | OK | | 49 | Pending(id, ikc, -) | Cancel(id, iku, T) | RejectedCanceled(id, ikc, iku) | OK | | 50 | Pending(id, ikc, -) | Cancel(id, iku, F) | RejectedCanceled(id, ikc, iku) | OK | | 51 | Resolved(id, -, -) | Create(id, -, T) | Resolved(id, -, -) | KO, Already Resolved | | 52 | Resolved(id, -, -) | Create(id, -, F) | Resolved(id, -, -) | KO, Already Resolved | | 53 | Resolved(id, -, -) | Create(id, ikc, T) | Resolved(id, -, -) | KO, Already Resolved | | 54 | Resolved(id, -, -) | Create(id, ikc, F) | Resolved(id, -, -) | KO, Already Resolved | | 55 | Resolved(id, -, -) | Resolve(id, -, T) | Resolved(id, -, -) | KO, Already Resolved | | 56 | Resolved(id, -, -) | Resolve(id, -, F) | Resolved(id, -, -) | KO, Already Resolved | | 57 | Resolved(id, -, -) | Resolve(id, iku, T) | Resolved(id, -, -) | KO, Already Resolved | | 58 | Resolved(id, -, -) | Resolve(id, iku, F) | Resolved(id, -, -) | KO, Already Resolved | | 59 | Resolved(id, -, -) | Reject(id, -, T) | Resolved(id, -, -) | KO, Already Resolved | | 60 | Resolved(id, -, -) | Reject(id, -, F) | Resolved(id, -, -) | KO, Already Resolved | | 61 | Resolved(id, -, -) | Reject(id, iku, T) | Resolved(id, -, -) | KO, Already Resolved | | 62 | Resolved(id, -, -) | Reject(id, iku, F) | Resolved(id, -, -) | KO, Already Resolved | | 63 | Resolved(id, -, -) | Cancel(id, -, T) | Resolved(id, -, -) | KO, Already Resolved | | 64 | Resolved(id, -, -) | Cancel(id, -, F) | Resolved(id, -, -) | KO, Already Resolved | | 65 | Resolved(id, -, -) | Cancel(id, iku, T) | Resolved(id, -, -) | KO, Already Resolved | | 66 | Resolved(id, -, -) | Cancel(id, iku, F) | Resolved(id, -, -) | KO, Already Resolved | | 67 | Resolved(id, -, iku) | Create(id, -, T) | Resolved(id, -, iku) | KO, Already Resolved | | 68 | Resolved(id, -, iku) | Create(id, -, F) | Resolved(id, -, iku) | KO, Already Resolved | | 69 | Resolved(id, -, iku) | Create(id, ikc, T) | Resolved(id, -, iku) | KO, Already Resolved | | 70 | Resolved(id, -, iku) | Create(id, ikc, F) | Resolved(id, -, iku) | KO, Already Resolved | | 71 | Resolved(id, -, iku) | Resolve(id, -, T) | Resolved(id, -, iku) | KO, Already Resolved | | 72 | Resolved(id, -, iku) | Resolve(id, -, F) | Resolved(id, -, iku) | KO, Already Resolved | | 73 | Resolved(id, -, iku) | Resolve(id, iku, T) | Resolved(id, -, iku) | OK, Deduplicated | | 74 | Resolved(id, -, iku) | Resolve(id, iku, F) | Resolved(id, -, iku) | OK, Deduplicated | | 75 | Resolved(id, -, iku) | Resolve(id, iku\*, T) | Resolved(id, -, iku) | KO, Already Resolved | | 76 | Resolved(id, -, iku) | Resolve(id, iku\*, F) | Resolved(id, -, iku) | KO, Already Resolved | | 77 | Resolved(id, -, iku) | Reject(id, -, T) | Resolved(id, -, iku) | KO, Already Resolved | | 78 | Resolved(id, -, iku) | Reject(id, -, F) | Resolved(id, -, iku) | KO, Already Resolved | | 79 | Resolved(id, -, iku) | Reject(id, iku, T) | Resolved(id, -, iku) | KO, Already Resolved | | 80 | Resolved(id, -, iku) | Reject(id, iku, F) | Resolved(id, -, iku) | OK, Deduplicated | | 81 | Resolved(id, -, iku) | Reject(id, iku\*, T) | Resolved(id, -, iku) | KO, Already Resolved | | 82 | Resolved(id, -, iku) | Reject(id, iku\*, F) | Resolved(id, -, iku) | KO, Already Resolved | | 83 | Resolved(id, -, iku) | Cancel(id, -, T) | Resolved(id, -, iku) | KO, Already Resolved | | 84 | Resolved(id, -, iku) | Cancel(id, -, F) | Resolved(id, -, iku) | KO, Already Resolved | | 85 | Resolved(id, -, iku) | Cancel(id, iku, T) | Resolved(id, -, iku) | KO, Already Resolved | | 86 | Resolved(id, -, iku) | Cancel(id, iku, F) | Resolved(id, -, iku) | OK, Deduplicated | | 87 | Resolved(id, -, iku) | Cancel(id, iku\*, T) | Resolved(id, -, iku) | KO, Already Resolved | | 88 | Resolved(id, -, iku) | Cancel(id, iku\*, F) | Resolved(id, -, iku) | KO, Already Resolved | | 89 | Resolved(id, ikc, -) | Create(id, -, T) | Resolved(id, ikc, -) | KO, Already Resolved | | 90 | Resolved(id, ikc, -) | Create(id, -, F) | Resolved(id, ikc, -) | KO, Already Resolved | | 91 | Resolved(id, ikc, -) | Create(id, ikc, T) | Resolved(id, ikc, -) | KO, Already Resolved | | 92 | Resolved(id, ikc, -) | Create(id, ikc, F) | Resolved(id, ikc, -) | OK, Deduplicated | | 93 | Resolved(id, ikc, -) | Create(id, ikc\*, T) | Resolved(id, ikc, -) | KO, Already Resolved | | 94 | Resolved(id, ikc, -) | Create(id, ikc\*, F) | Resolved(id, ikc, -) | KO, Already Resolved | | 95 | Resolved(id, ikc, -) | Resolve(id, -, T) | Resolved(id, ikc, -) | KO, Already Resolved | | 96 | Resolved(id, ikc, -) | Resolve(id, -, F) | Resolved(id, ikc, -) | KO, Already Resolved | | 97 | Resolved(id, ikc, -) | Resolve(id, iku, T) | Resolved(id, ikc, -) | KO, Already Resolved | | 98 | Resolved(id, ikc, -) | Resolve(id, iku, F) | Resolved(id, ikc, -) | KO, Already Resolved | | 99 | Resolved(id, ikc, -) | Reject(id, -, T) | Resolved(id, ikc, -) | KO, Already Resolved | | 100 | Resolved(id, ikc, -) | Reject(id, -, F) | Resolved(id, ikc, -) | KO, Already Resolved | | 101 | Resolved(id, ikc, -) | Reject(id, iku, T) | Resolved(id, ikc, -) | KO, Already Resolved | | 102 | Resolved(id, ikc, -) | Reject(id, iku, F) | Resolved(id, ikc, -) | KO, Already Resolved | | 103 | Resolved(id, ikc, -) | Cancel(id, -, T) | Resolved(id, ikc, -) | KO, Already Resolved | | 104 | Resolved(id, ikc, -) | Cancel(id, -, F) | Resolved(id, ikc, -) | KO, Already Resolved | | 105 | Resolved(id, ikc, -) | Cancel(id, iku, T) | Resolved(id, ikc, -) | KO, Already Resolved | | 106 | Resolved(id, ikc, -) | Cancel(id, iku, F) | Resolved(id, ikc, -) | KO, Already Resolved | | 107 | Resolved(id, ikc, iku) | Create(id, -, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 108 | Resolved(id, ikc, iku) | Create(id, -, F) | Resolved(id, ikc, iku) | KO, Already Resolved | | 109 | Resolved(id, ikc, iku) | Create(id, ikc, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 110 | Resolved(id, ikc, iku) | Create(id, ikc, F) | Resolved(id, ikc, iku) | OK, Deduplicated | | 111 | Resolved(id, ikc, iku) | Create(id, ikc\*, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 112 | Resolved(id, ikc, iku) | Create(id, ikc\*, F) | Resolved(id, ikc, iku) | KO, Already Resolved | | 113 | Resolved(id, ikc, iku) | Resolve(id, -, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 114 | Resolved(id, ikc, iku) | Resolve(id, -, F) | Resolved(id, ikc, iku) | KO, Already Resolved | | 115 | Resolved(id, ikc, iku) | Resolve(id, iku, T) | Resolved(id, ikc, iku) | OK, Deduplicated | | 116 | Resolved(id, ikc, iku) | Resolve(id, iku, F) | Resolved(id, ikc, iku) | OK, Deduplicated | | 117 | Resolved(id, ikc, iku) | Resolve(id, iku\*, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 118 | Resolved(id, ikc, iku) | Resolve(id, iku\*, F) | Resolved(id, ikc, iku) | KO, Already Resolved | | 119 | Resolved(id, ikc, iku) | Reject(id, -, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 120 | Resolved(id, ikc, iku) | Reject(id, -, F) | Resolved(id, ikc, iku) | KO, Already Resolved | | 121 | Resolved(id, ikc, iku) | Reject(id, iku, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 122 | Resolved(id, ikc, iku) | Reject(id, iku, F) | Resolved(id, ikc, iku) | OK, Deduplicated | | 123 | Resolved(id, ikc, iku) | Reject(id, iku\*, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 124 | Resolved(id, ikc, iku) | Reject(id, iku\*, F) | Resolved(id, ikc, iku) | KO, Already Resolved | | 125 | Resolved(id, ikc, iku) | Cancel(id, -, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 126 | Resolved(id, ikc, iku) | Cancel(id, -, F) | Resolved(id, ikc, iku) | KO, Already Resolved | | 127 | Resolved(id, ikc, iku) | Cancel(id, iku, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 128 | Resolved(id, ikc, iku) | Cancel(id, iku, F) | Resolved(id, ikc, iku) | OK, Deduplicated | | 129 | Resolved(id, ikc, iku) | Cancel(id, iku\*, T) | Resolved(id, ikc, iku) | KO, Already Resolved | | 130 | Resolved(id, ikc, iku) | Cancel(id, iku\*, F) | Resolved(id, ikc, iku) | KO, Already Resolved | | 131 | Rejected(id, -, -) | Create(id, -, T) | Rejected(id, -, -) | KO, Already Rejected | | 132 | Rejected(id, -, -) | Create(id, -, F) | Rejected(id, -, -) | KO, Already Rejected | | 133 | Rejected(id, -, -) | Create(id, ikc, T) | Rejected(id, -, -) | KO, Already Rejected | | 134 | Rejected(id, -, -) | Create(id, ikc, F) | Rejected(id, -, -) | KO, Already Rejected | | 135 | Rejected(id, -, -) | Resolve(id, -, T) | Rejected(id, -, -) | KO, Already Rejected | | 136 | Rejected(id, -, -) | Resolve(id, -, F) | Rejected(id, -, -) | KO, Already Rejected | | 137 | Rejected(id, -, -) | Resolve(id, iku, T) | Rejected(id, -, -) | KO, Already Rejected | | 138 | Rejected(id, -, -) | Resolve(id, iku, F) | Rejected(id, -, -) | KO, Already Rejected | | 139 | Rejected(id, -, -) | Reject(id, -, T) | Rejected(id, -, -) | KO, Already Rejected | | 140 | Rejected(id, -, -) | Reject(id, -, F) | Rejected(id, -, -) | KO, Already Rejected | | 141 | Rejected(id, -, -) | Reject(id, iku, T) | Rejected(id, -, -) | KO, Already Rejected | | 142 | Rejected(id, -, -) | Reject(id, iku, F) | Rejected(id, -, -) | KO, Already Rejected | | 143 | Rejected(id, -, -) | Cancel(id, -, T) | Rejected(id, -, -) | KO, Already Rejected | | 144 | Rejected(id, -, -) | Cancel(id, -, F) | Rejected(id, -, -) | KO, Already Rejected | | 145 | Rejected(id, -, -) | Cancel(id, iku, T) | Rejected(id, -, -) | KO, Already Rejected | | 146 | Rejected(id, -, -) | Cancel(id, iku, F) | Rejected(id, -, -) | KO, Already Rejected | | 147 | Rejected(id, -, iku) | Create(id, -, T) | Rejected(id, -, iku) | KO, Already Rejected | | 148 | Rejected(id, -, iku) | Create(id, -, F) | Rejected(id, -, iku) | KO, Already Rejected | | 149 | Rejected(id, -, iku) | Create(id, ikc, T) | Rejected(id, -, iku) | KO, Already Rejected | | 150 | Rejected(id, -, iku) | Create(id, ikc, F) | Rejected(id, -, iku) | KO, Already Rejected | | 151 | Rejected(id, -, iku) | Resolve(id, -, T) | Rejected(id, -, iku) | KO, Already Rejected | | 152 | Rejected(id, -, iku) | Resolve(id, -, F) | Rejected(id, -, iku) | KO, Already Rejected | | 153 | Rejected(id, -, iku) | Resolve(id, iku, T) | Rejected(id, -, iku) | KO, Already Rejected | | 154 | Rejected(id, -, iku) | Resolve(id, iku, F) | Rejected(id, -, iku) | OK, Deduplicated | | 155 | Rejected(id, -, iku) | Resolve(id, iku\*, T) | Rejected(id, -, iku) | KO, Already Rejected | | 156 | Rejected(id, -, iku) | Resolve(id, iku\*, F) | Rejected(id, -, iku) | KO, Already Rejected | | 157 | Rejected(id, -, iku) | Reject(id, -, T) | Rejected(id, -, iku) | KO, Already Rejected | | 158 | Rejected(id, -, iku) | Reject(id, -, F) | Rejected(id, -, iku) | KO, Already Rejected | | 159 | Rejected(id, -, iku) | Reject(id, iku, T) | Rejected(id, -, iku) | OK, Deduplicated | | 160 | Rejected(id, -, iku) | Reject(id, iku, F) | Rejected(id, -, iku) | OK, Deduplicated | | 161 | Rejected(id, -, iku) | Reject(id, iku\*, T) | Rejected(id, -, iku) | KO, Already Rejected | | 162 | Rejected(id, -, iku) | Reject(id, iku\*, F) | Rejected(id, -, iku) | KO, Already Rejected | | 163 | Rejected(id, -, iku) | Cancel(id, -, T) | Rejected(id, -, iku) | KO, Already Rejected | | 164 | Rejected(id, -, iku) | Cancel(id, -, F) | Rejected(id, -, iku) | KO, Already Rejected | | 165 | Rejected(id, -, iku) | Cancel(id, iku, T) | Rejected(id, -, iku) | KO, Already Rejected | | 166 | Rejected(id, -, iku) | Cancel(id, iku, F) | Rejected(id, -, iku) | OK, Deduplicated | | 167 | Rejected(id, -, iku) | Cancel(id, iku\*, T) | Rejected(id, -, iku) | KO, Already Rejected | | 168 | Rejected(id, -, iku) | Cancel(id, iku\*, F) | Rejected(id, -, iku) | KO, Already Rejected | | 169 | Rejected(id, ikc, -) | Create(id, -, T) | Rejected(id, ikc, -) | KO, Already Rejected | | 170 | Rejected(id, ikc, -) | Create(id, -, F) | Rejected(id, ikc, -) | KO, Already Rejected | | 171 | Rejected(id, ikc, -) | Create(id, ikc, T) | Rejected(id, ikc, -) | KO, Already Rejected | | 172 | Rejected(id, ikc, -) | Create(id, ikc, F) | Rejected(id, ikc, -) | OK, Deduplicated | | 173 | Rejected(id, ikc, -) | Create(id, ikc\*, T) | Rejected(id, ikc, -) | KO, Already Rejected | | 174 | Rejected(id, ikc, -) | Create(id, ikc\*, F) | Rejected(id, ikc, -) | KO, Already Rejected | | 175 | Rejected(id, ikc, -) | Resolve(id, -, T) | Rejected(id, ikc, -) | KO, Already Rejected | | 176 | Rejected(id, ikc, -) | Resolve(id, -, F) | Rejected(id, ikc, -) | KO, Already Rejected | | 177 | Rejected(id, ikc, -) | Resolve(id, iku, T) | Rejected(id, ikc, -) | KO, Already Rejected | | 178 | Rejected(id, ikc, -) | Resolve(id, iku, F) | Rejected(id, ikc, -) | KO, Already Rejected | | 179 | Rejected(id, ikc, -) | Reject(id, -, T) | Rejected(id, ikc, -) | KO, Already Rejected | | 180 | Rejected(id, ikc, -) | Reject(id, -, F) | Rejected(id, ikc, -) | KO, Already Rejected | | 181 | Rejected(id, ikc, -) | Reject(id, iku, T) | Rejected(id, ikc, -) | KO, Already Rejected | | 182 | Rejected(id, ikc, -) | Reject(id, iku, F) | Rejected(id, ikc, -) | KO, Already Rejected | | 183 | Rejected(id, ikc, -) | Cancel(id, -, T) | Rejected(id, ikc, -) | KO, Already Rejected | | 184 | Rejected(id, ikc, -) | Cancel(id, -, F) | Rejected(id, ikc, -) | KO, Already Rejected | | 185 | Rejected(id, ikc, -) | Cancel(id, iku, T) | Rejected(id, ikc, -) | KO, Already Rejected | | 186 | Rejected(id, ikc, -) | Cancel(id, iku, F) | Rejected(id, ikc, -) | KO, Already Rejected | | 187 | Rejected(id, ikc, iku) | Create(id, -, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 188 | Rejected(id, ikc, iku) | Create(id, -, F) | Rejected(id, ikc, iku) | KO, Already Rejected | | 189 | Rejected(id, ikc, iku) | Create(id, ikc, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 190 | Rejected(id, ikc, iku) | Create(id, ikc, F) | Rejected(id, ikc, iku) | OK, Deduplicated | | 191 | Rejected(id, ikc, iku) | Create(id, ikc\*, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 192 | Rejected(id, ikc, iku) | Create(id, ikc\*, F) | Rejected(id, ikc, iku) | KO, Already Rejected | | 193 | Rejected(id, ikc, iku) | Resolve(id, -, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 194 | Rejected(id, ikc, iku) | Resolve(id, -, F) | Rejected(id, ikc, iku) | KO, Already Rejected | | 195 | Rejected(id, ikc, iku) | Resolve(id, iku, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 196 | Rejected(id, ikc, iku) | Resolve(id, iku, F) | Rejected(id, ikc, iku) | OK, Deduplicated | | 197 | Rejected(id, ikc, iku) | Resolve(id, iku\*, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 198 | Rejected(id, ikc, iku) | Resolve(id, iku\*, F) | Rejected(id, ikc, iku) | KO, Already Rejected | | 199 | Rejected(id, ikc, iku) | Reject(id, -, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 200 | Rejected(id, ikc, iku) | Reject(id, -, F) | Rejected(id, ikc, iku) | KO, Already Rejected | | 201 | Rejected(id, ikc, iku) | Reject(id, iku, T) | Rejected(id, ikc, iku) | OK, Deduplicated | | 202 | Rejected(id, ikc, iku) | Reject(id, iku, F) | Rejected(id, ikc, iku) | OK, Deduplicated | | 203 | Rejected(id, ikc, iku) | Reject(id, iku\*, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 204 | Rejected(id, ikc, iku) | Reject(id, iku\*, F) | Rejected(id, ikc, iku) | KO, Already Rejected | | 205 | Rejected(id, ikc, iku) | Cancel(id, -, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 206 | Rejected(id, ikc, iku) | Cancel(id, -, F) | Rejected(id, ikc, iku) | KO, Already Rejected | | 207 | Rejected(id, ikc, iku) | Cancel(id, iku, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 208 | Rejected(id, ikc, iku) | Cancel(id, iku, F) | Rejected(id, ikc, iku) | OK, Deduplicated | | 209 | Rejected(id, ikc, iku) | Cancel(id, iku\*, T) | Rejected(id, ikc, iku) | KO, Already Rejected | | 210 | Rejected(id, ikc, iku) | Cancel(id, iku\*, F) | Rejected(id, ikc, iku) | KO, Already Rejected | | 211 | RejectedCanceled(id, -, -) | Create(id, -, T) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 212 | RejectedCanceled(id, -, -) | Create(id, -, F) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 213 | RejectedCanceled(id, -, -) | Create(id, ikc, T) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 214 | RejectedCanceled(id, -, -) | Create(id, ikc, F) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 215 | RejectedCanceled(id, -, -) | Resolve(id, -, T) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 216 | RejectedCanceled(id, -, -) | Resolve(id, -, F) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 217 | RejectedCanceled(id, -, -) | Resolve(id, iku, T) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 218 | RejectedCanceled(id, -, -) | Resolve(id, iku, F) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 219 | RejectedCanceled(id, -, -) | Reject(id, -, T) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 220 | RejectedCanceled(id, -, -) | Reject(id, -, F) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 221 | RejectedCanceled(id, -, -) | Reject(id, iku, T) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 222 | RejectedCanceled(id, -, -) | Reject(id, iku, F) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 223 | RejectedCanceled(id, -, -) | Cancel(id, -, T) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 224 | RejectedCanceled(id, -, -) | Cancel(id, -, F) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 225 | RejectedCanceled(id, -, -) | Cancel(id, iku, T) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 226 | RejectedCanceled(id, -, -) | Cancel(id, iku, F) | RejectedCanceled(id, -, -) | KO, Already RejectedCanceled | | 227 | RejectedCanceled(id, -, iku) | Create(id, -, T) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 228 | RejectedCanceled(id, -, iku) | Create(id, -, F) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 229 | RejectedCanceled(id, -, iku) | Create(id, ikc, T) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 230 | RejectedCanceled(id, -, iku) | Create(id, ikc, F) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 231 | RejectedCanceled(id, -, iku) | Resolve(id, -, T) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 232 | RejectedCanceled(id, -, iku) | Resolve(id, -, F) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 233 | RejectedCanceled(id, -, iku) | Resolve(id, iku, T) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 234 | RejectedCanceled(id, -, iku) | Resolve(id, iku, F) | RejectedCanceled(id, -, iku) | OK, Deduplicated | | 235 | RejectedCanceled(id, -, iku) | Resolve(id, iku\*, T) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 236 | RejectedCanceled(id, -, iku) | Resolve(id, iku\*, F) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 237 | RejectedCanceled(id, -, iku) | Reject(id, -, T) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 238 | RejectedCanceled(id, -, iku) | Reject(id, -, F) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 239 | RejectedCanceled(id, -, iku) | Reject(id, iku, T) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 240 | RejectedCanceled(id, -, iku) | Reject(id, iku, F) | RejectedCanceled(id, -, iku) | OK, Deduplicated | | 241 | RejectedCanceled(id, -, iku) | Reject(id, iku\*, T) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 242 | RejectedCanceled(id, -, iku) | Reject(id, iku\*, F) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 243 | RejectedCanceled(id, -, iku) | Cancel(id, -, T) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 244 | RejectedCanceled(id, -, iku) | Cancel(id, -, F) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 245 | RejectedCanceled(id, -, iku) | Cancel(id, iku, T) | RejectedCanceled(id, -, iku) | OK, Deduplicated | | 246 | RejectedCanceled(id, -, iku) | Cancel(id, iku, F) | RejectedCanceled(id, -, iku) | OK, Deduplicated | | 247 | RejectedCanceled(id, -, iku) | Cancel(id, iku\*, T) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 248 | RejectedCanceled(id, -, iku) | Cancel(id, iku\*, F) | RejectedCanceled(id, -, iku) | KO, Already RejectedCanceled | | 249 | RejectedCanceled(id, ikc, -) | Create(id, -, T) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 250 | RejectedCanceled(id, ikc, -) | Create(id, -, F) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 251 | RejectedCanceled(id, ikc, -) | Create(id, ikc, T) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 252 | RejectedCanceled(id, ikc, -) | Create(id, ikc, F) | RejectedCanceled(id, ikc, -) | OK, Deduplicated | | 253 | RejectedCanceled(id, ikc, -) | Create(id, ikc\*, T) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 254 | RejectedCanceled(id, ikc, -) | Create(id, ikc\*, F) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 255 | RejectedCanceled(id, ikc, -) | Resolve(id, -, T) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 256 | RejectedCanceled(id, ikc, -) | Resolve(id, -, F) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 257 | RejectedCanceled(id, ikc, -) | Resolve(id, iku, T) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 258 | RejectedCanceled(id, ikc, -) | Resolve(id, iku, F) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 259 | RejectedCanceled(id, ikc, -) | Reject(id, -, T) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 260 | RejectedCanceled(id, ikc, -) | Reject(id, -, F) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 261 | RejectedCanceled(id, ikc, -) | Reject(id, iku, T) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 262 | RejectedCanceled(id, ikc, -) | Reject(id, iku, F) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 263 | RejectedCanceled(id, ikc, -) | Cancel(id, -, T) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 264 | RejectedCanceled(id, ikc, -) | Cancel(id, -, F) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 265 | RejectedCanceled(id, ikc, -) | Cancel(id, iku, T) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 266 | RejectedCanceled(id, ikc, -) | Cancel(id, iku, F) | RejectedCanceled(id, ikc, -) | KO, Already RejectedCanceled | | 267 | RejectedCanceled(id, ikc, iku) | Create(id, -, T) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 268 | RejectedCanceled(id, ikc, iku) | Create(id, -, F) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 269 | RejectedCanceled(id, ikc, iku) | Create(id, ikc, T) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 270 | RejectedCanceled(id, ikc, iku) | Create(id, ikc, F) | RejectedCanceled(id, ikc, iku) | OK, Deduplicated | | 271 | RejectedCanceled(id, ikc, iku) | Create(id, ikc\*, T) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 272 | RejectedCanceled(id, ikc, iku) | Create(id, ikc\*, F) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 273 | RejectedCanceled(id, ikc, iku) | Resolve(id, -, T) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 274 | RejectedCanceled(id, ikc, iku) | Resolve(id, -, F) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 275 | RejectedCanceled(id, ikc, iku) | Resolve(id, iku, T) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 276 | RejectedCanceled(id, ikc, iku) | Resolve(id, iku, F) | RejectedCanceled(id, ikc, iku) | OK, Deduplicated | | 277 | RejectedCanceled(id, ikc, iku) | Resolve(id, iku\*, T) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 278 | RejectedCanceled(id, ikc, iku) | Resolve(id, iku\*, F) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 279 | RejectedCanceled(id, ikc, iku) | Reject(id, -, T) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 280 | RejectedCanceled(id, ikc, iku) | Reject(id, -, F) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 281 | RejectedCanceled(id, ikc, iku) | Reject(id, iku, T) | RejectedCanceled(id, ikc, iku) | OK, Deduplicated | | 282 | RejectedCanceled(id, ikc, iku) | Reject(id, iku, F) | RejectedCanceled(id, ikc, iku) | OK, Deduplicated | | 283 | RejectedCanceled(id, ikc, iku) | Reject(id, iku\*, T) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 284 | RejectedCanceled(id, ikc, iku) | Reject(id, iku\*, F) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 285 | RejectedCanceled(id, ikc, iku) | Cancel(id, -, T) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 286 | RejectedCanceled(id, ikc, iku) | Cancel(id, -, F) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 287 | RejectedCanceled(id, ikc, iku) | Cancel(id, iku, T) | RejectedCanceled(id, ikc, iku) | OK, Deduplicated | | 288 | RejectedCanceled(id, ikc, iku) | Cancel(id, iku, F) | RejectedCanceled(id, ikc, iku) | OK, Deduplicated | | 289 | RejectedCanceled(id, ikc, iku) | Cancel(id, iku\*, T) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 290 | RejectedCanceled(id, ikc, iku) | Cancel(id, iku\*, F) | RejectedCanceled(id, ikc, iku) | KO, Already RejectedCanceled | | 291 | RejectedTimedout(id, -, -) | Create(id, -, T) | RejectedTimedout(id, -, -) | KO, Already RejectedTimedout | | 292 | RejectedTimedout(id, -, -) | Create(id, -, F) | RejectedTimedout(id, -, -) | KO, Already RejectedTimedout | | 293 | RejectedTimedout(id, -, -) | Create(id, ikc, T) | RejectedTimedout(id, -, -) | KO, Already RejectedTimedout | | 294 | RejectedTimedout(id, -, -) | Create(id, ikc, F) | RejectedTimedout(id, -, -) | KO, Already RejectedTimedout | | 295 | RejectedTimedout(id, -, -) | Resolve(id, -, T) | RejectedTimedout(id, -, -) | KO, Already RejectedTimedout | | 296 | RejectedTimedout(id, -, -) | Resolve(id, -, F) | RejectedTimedout(id, -, -) | OK, Deduplicated | | 297 | RejectedTimedout(id, -, -) | Resolve(id, iku, T) | RejectedTimedout(id, -, -) | KO, Already RejectedTimedout | | 298 | RejectedTimedout(id, -, -) | Resolve(id, iku, F) | RejectedTimedout(id, -, -) | OK, Deduplicated | | 299 | RejectedTimedout(id, -, -) | Reject(id, -, T) | RejectedTimedout(id, -, -) | KO, Already RejectedTimedout | | 300 | RejectedTimedout(id, -, -) | Reject(id, -, F) | RejectedTimedout(id, -, -) | OK, Deduplicated | | 301 | RejectedTimedout(id, -, -) | Reject(id, iku, T) | RejectedTimedout(id, -, -) | KO, Already RejectedTimedout | | 302 | RejectedTimedout(id, -, -) | Reject(id, iku, F) | RejectedTimedout(id, -, -) | OK, Deduplicated | | 303 | RejectedTimedout(id, -, -) | Cancel(id, -, T) | RejectedTimedout(id, -, -) | KO, Already RejectedTimedout | | 304 | RejectedTimedout(id, -, -) | Cancel(id, -, F) | RejectedTimedout(id, -, -) | OK, Deduplicated | | 305 | RejectedTimedout(id, -, -) | Cancel(id, iku, T) | RejectedTimedout(id, -, -) | KO, Already RejectedTimedout | | 306 | RejectedTimedout(id, -, -) | Cancel(id, iku, F) | RejectedTimedout(id, -, -) | OK, Deduplicated | | 307 | RejectedTimedout(id, ikc, -) | Create(id, -, T) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 308 | RejectedTimedout(id, ikc, -) | Create(id, -, F) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 309 | RejectedTimedout(id, ikc, -) | Create(id, ikc, T) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 310 | RejectedTimedout(id, ikc, -) | Create(id, ikc, F) | RejectedTimedout(id, ikc, -) | OK, Deduplicated | | 311 | RejectedTimedout(id, ikc, -) | Create(id, ikc\*, T) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 312 | RejectedTimedout(id, ikc, -) | Create(id, ikc\*, F) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 313 | RejectedTimedout(id, ikc, -) | Resolve(id, -, T) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 314 | RejectedTimedout(id, ikc, -) | Resolve(id, -, F) | RejectedTimedout(id, ikc, -) | OK, Deduplicated | | 315 | RejectedTimedout(id, ikc, -) | Resolve(id, iku, T) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 316 | RejectedTimedout(id, ikc, -) | Resolve(id, iku, F) | RejectedTimedout(id, ikc, -) | OK, Deduplicated | | 317 | RejectedTimedout(id, ikc, -) | Reject(id, -, T) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 318 | RejectedTimedout(id, ikc, -) | Reject(id, -, F) | RejectedTimedout(id, ikc, -) | OK, Deduplicated | | 319 | RejectedTimedout(id, ikc, -) | Reject(id, iku, T) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 320 | RejectedTimedout(id, ikc, -) | Reject(id, iku, F) | RejectedTimedout(id, ikc, -) | OK, Deduplicated | | 321 | RejectedTimedout(id, ikc, -) | Cancel(id, -, T) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 322 | RejectedTimedout(id, ikc, -) | Cancel(id, -, F) | RejectedTimedout(id, ikc, -) | OK, Deduplicated | | 323 | RejectedTimedout(id, ikc, -) | Cancel(id, iku, T) | RejectedTimedout(id, ikc, -) | KO, Already RejectedTimedout | | 324 | RejectedTimedout(id, ikc, -) | Cancel(id, iku, F) | RejectedTimedout(id, ikc, -) | OK, Deduplicated | --- --- url: /spec/programming-model title: Programming model --- Distributed Async Await, like async/await, builds on two abstractions: functions and promises. Functions are a universal abstraction for computation; promises are a universal abstraction for coordination. ## Async Await ### Sequential programming - Combines invocation and synchronization. - Only one function execution enabled at one time. In a traditional programming model, when a function (the _caller_) invokes another function (the _callee_), the callee begins and the caller suspends until the callee returns. Upon return, the caller resumes execution with the return value of the callee. Executions proceed sequentially — at any point in time, exactly one execution may take a step. ### Concurrent programming - Separates invocation and synchronization. - More than one function execution enabled at one time. In async/await, when the caller invokes the callee, the callee begins and the caller continues execution with a promise representing the callee. When the caller requires the return value of the callee, the caller awaits the promise — suspending execution until the callee returns, at which point the caller resumes with the return value of the callee. Executions proceed concurrently — at any point in time, more than one execution may take a step. ### The role of promises Promises are a universal abstraction for coordination. They allow callers and callees to execute concurrently while still enabling the caller to coordinate by awaiting the completion and receiving the return value of the callee. ![Execution locality of sequential vs concurrent](/images/spec/traditional-vs-async-await.svg) ## Distributed Async Await Distributed Async Await extends async/await to a concurrent and distributed programming model: like async/await, Distributed Async Await is based on functions and promises, but adds **Distributed Coordination** and **Distributed Recovery**. ### Distributed coordination Distributed Async Await allows callers to invoke and await callees across processes. Promises are no longer local objects — promises coordinate across the network, enabling the familiar semantics of async/await to span across systems. ### Distributed recovery Distributed Async Await defines recovery semantics in case of a process interruption. ## Call graphs Call graphs are a powerful tool for visualizing the execution of a program. Functions compose recursively — a function may call other functions, spanning a call graph. A call graph is a directed graph where the nodes are functions and the edges are calls between functions. The call graph provides a flexible lens onto a computation: it is zoomable, allowing one to expand a caller into its callees (treating a function as a composite) or collapse those callees into the caller (treating the function as atomic). This flexibility enables reasoning at different levels of abstraction. ![Generic Call Graph](/images/spec/generic-call-graph.svg) Within the context of the Distributed Async Await specification, and all implementations of it, a call graph is the full set of promises and executions from ephemeral edge to ephemeral edge — a map of the durable world. In the Distributed Async Await specification, all execution invocations pair with a promise. ![Promise execution pair](/images/spec/promise-execution-pairs.svg) An execution can be either a function execution or some action that must be taken outside of the system, such as a human clicking a button. ![Promise execution promise action pairs](/images/spec/promise-execution-promise-action-pairs.svg) A call graph can illustrate locality — that is, an execution can display its locality in relation to other executions. This is called a **distributed call graph**. ![Execution locality](/images/spec/execution-locality.svg) A call graph is zoomable — it can show the full set of promises and executions, or it can show just the root executions. Consider the following pseudo-code, where `foo()` and `bar()` are in process a, and `baz()` is in process b. The notation `async_local` / `async_remote` (and `async_r` used elsewhere in this specification) is **pseudo-code chosen for the spec**. It is not the surface API of any implementation. Reference implementations expose the same semantics through their own idioms — for example, `yield* ctx.run(fn)` for local invocation and `yield* ctx.beginRpc(fn, args)` for remote invocation in the TypeScript SDK. See the [TypeScript](https://docs.resonatehq.io/develop/typescript), [Python](https://docs.resonatehq.io/develop/python), [Rust](https://docs.resonatehq.io/develop/rust), [Go](https://docs.resonatehq.io/develop/go), and [Java](https://docs.resonatehq.io/develop/java) SDKs for the per-language mapping. process a: ```text func foo(): r = await (async_local bar()) return r func bar(): r = await (async_remote baz()) return r ``` process b: ```text func baz(): return 1 ``` If the call graph zooms in on all the promises, function executions, and localities (details, concurrency, and distribution), it would look like this: ![Detailed distributed concurrent Call Graph](/images/spec/detailed-distributed-concurrent-call-graph.svg) This same call structure can assume promises and the call graph can just show the function executions. ![Distributed concurrent Call Graph](/images/spec/distributed-concurrent-call-graph.svg) To show just the distributed nature of the application, we can zoom out and show just the root executions of each process. ![Distributed Call Graph](/images/spec/distributed-call-graph.svg) ## Local and remote When a function calls another function within the same process, it is called a **Local Function Invocation (LFI)**. When a function calls another function in a different application node, it is considered a **Remote Function Invocation (RFI)**. An application can make use of both LFIs and RFIs, and thus a call graph can span just one or many application nodes. ## Root promises There are two types of root promises. The first is the root promise of an application node (process), and the second is the root promise of a call graph. An application-node root promise (also known as the process) is the promise associated with the first execution in an application node — the execution invoked by `resonate.run()` (reference SDK notation). A call-graph root promise is the promise associated with the first execution in a call graph. This execution is also invoked by `resonate.run()` (reference SDK notation), but it is on the "edge" — typically the entry point into the application. ## Sub-specifications --- --- url: /spec/system-model/executions title: Function Executions --- Executions are the **fundamental unit of computation** in a system. On an execution level, a system is a collection of executions; each execution has a unique identity. An execution has a well-defined lifecycle, represented by the lifecycle events _invoke_ and _return_, that governs its participation in the system: **_invoke_** marks the initialization of the execution and represents its entry into the system. **_return_** marks the termination of the execution and represents its exit from the system. An execution is bound to a single process. An execution cannot emit events outside its (_invoke_, _return_) interval. As a consequence, an execution cannot be restarted. Instead, a new execution must be created that may be considered the logical successor or the logical equivalent of the terminated execution. ![Execution lifecycle diagram](/images/spec/execution-lifecycle.svg) An execution may raise events such as send or receive events only within its (_invoke_, _return_) interval. ## Long-running executions The term *long running* is not a well-defined term in the context of distributed systems — "long" is relative. This system model argues that you should not think about "long" in terms of time. Instead, *long* refers to the potential for a logical execution to span multiple physical locations. An execution is long-running if it potentially runs across multiple processes. ## Physical vs logical Distributed Async Await makes the distinction between physical executions and logical executions. One logical execution consists of multiple concurrent and distributed physical executions — physical executions may execute independently and on different resources. For example, logical execution `le1` starts as physical execution `pe1` on physical process `p1`. Physical process `p1` then experiences a crash failure. Logical execution `le1` is then resumed as physical execution `pe2` on physical process `p2`. Physical executions never "restart". Each physical execution is distinct. A logical execution can contain many physical executions, each one starting from the beginning and attempting to make progress. The logical execution completes when a physical execution reaches the return statement of the function. ## Concurrency The defining characteristic of concurrency is non-deterministic *partial order*; to mitigate the challenges of partial order, we leverage **coordination** to ensure consistency. ## Distribution The defining characteristic of distribution is non-deterministic *partial failure*; to mitigate the challenges of partial failure, we leverage **recovery** to ensure completion. ### Distribution across space An execution is distributed across space when the same logical execution spans multiple physical executions concurrently — at the same point in time. ### Distribution across time An execution is distributed across time when the same logical execution spans multiple physical executions sequentially — at different points in time. --- --- url: /spec/system-model title: System model --- The system model introduces a **way of thinking** about distributed systems that is foundational to the Distributed Async Await specification. It defines the rules that govern the behavior of distributed software systems. Establishing a clear and concise **definition of a distributed system** is foundational to a first-principles approach to a coherent system model. A distributed system is a collection of concurrent components — *processes* — that communicate by exchanging messages over the network. Each component has exclusive access to its local state as well as its local channel to the network. Components do not share state and cannot directly observe each other. ![Distributed System diagram](/images/spec/distributed-system.svg) The behavior of a distributed system emerges from the behavior of its processes. The system model is structured into three parts: - **[Processes](/spec/system-model/processes)** — the fundamental unit of locality, with a well-defined lifecycle and a fail-stop failure model. - **[Function Executions](/spec/system-model/executions)** — the fundamental unit of computation, bound to a process, with a logical layer that can survive physical-process crashes. - **[Tasks](/spec/system-model/tasks)** — the unit of work the server delivers to a worker, with a lease-based liveness model that operationalizes recovery. The communication layer that ties these together — message events, addressing, the wire envelope — lives in the [Message Passing Protocol](/spec/execution-model/message-passing). --- --- url: /spec/system-model/processes title: Processes --- Processes are the **fundamental unit of locality** in a system. On a process level, a system is a collection of processes; each process has a unique identity. A process has a well-defined lifecycle, represented by the lifecycle events _init_ and _term_, that governs its participation in the system: **_init_** marks the initialization of the process and represents its entry into the system. **_term_** marks the termination of the process and represents its exit from the system. A process cannot emit events outside its (_init_, _term_) interval. As a consequence, a process cannot announce its termination after its termination. Instead, if desired, the process must announce its intent to terminate. In the case of failure, the process fails silently. ![Process lifecycle diagram](/images/spec/process-lifecycle.svg) A process cannot be restarted. Instead, a new process must be created that may be considered the logical successor or the logical equivalent of the terminated process. A _term_ event may denote both a normal planned termination and an abnormal unplanned termination — fail-stop. The behavior of a process is represented by its trace, a totally ordered sequence of events such as sending or receiving a message. Within a process, events are totally ordered. However, across processes, events are only partially ordered. Processes emit events through [executions](/spec/system-model/executions). ## Failure Distributed Async Await assumes a fail-stop, also called crash-stop, failure model for processes: when a process experiences a failure, the process stops performing any internal or external steps at an arbitrary moment in time and never again performs any further internal or external steps. This specification does not make statements about process creation. Specifically, when processes leave the system (e.g. in the event of failures), an external mechanism is responsible to add new processes if necessary. A process may non-deterministically terminate. Specifically, upon termination, the process does not raise additional events and therefore cannot announce its termination. In case of a failure — that is, an unexpected and unwanted termination event — a process is considered crashed. The process cannot be restarted. Instead, a new process must be created that may be considered the logical successor or the logical equivalent of the terminated process. The notion of logical successor or logical equivalent introduces two distinct levels of reasoning: the logical layer and the physical layer. ### Logical layer The logical layer is populated by logical processes. A logical process has a logical id. A logical process is composed of one or more physical processes. ### Physical layer The physical layer is populated by physical processes. A physical process has a physical id. One or more physical processes are composed into one logical process. A logical process can experience additional lifetime events: suspension and resumption. When a physical process terminates, the associated logical process suspends (commonly referred to as a crash). When another physical process initiates, the associated logical process resumes (commonly referred to as restart or recovery). --- --- url: /spec/system-model/tasks title: Tasks --- A task is the unit of work the server delivers to a worker. Where a [Promise](/spec/programming-model/durable-promise-specification) represents a future value, a task represents the *responsibility for producing that value* — claimed by a worker, kept alive by a lease, and surrendered on crash so a successor can take over. A task and its associated promise share an identifier. They are paired but distinct: the promise owns the value; the task owns the claim. Tasks exist only when work needs to be delivered to a worker — that is, when the promise carries a `resonate:target` tag specifying a delivery address. ## Lifecycle A task moves through five states. The version increments only on the `pending → acquired` transition (`task.acquire`); it is the optimistic concurrency control (OCC) token every mutating operation must present. ```text create (via promise.create with resonate:target) | v [pending] ---(acquire, version++) ---> [acquired] <--- create (via task.create) ^ | ^ | <------(release)----- (suspend) | | | | v | | [suspended] | | +-------------(resume)----------------+ | | [pending] ---(halt)---> [halted] | [acquired] --(halt)---> [halted] | [suspended]-(halt)---> [halted] | +<-----------(continue)--- [halted] [acquired] ---(fulfill)---> [fulfilled] ``` | State | Meaning | |---|---| | `pending` | The task is available to be claimed. The server has enqueued an `execute` message to the `resonate:target` address. | | `acquired` | A worker has claimed the task and is making progress. The worker holds a lease that must be refreshed via heartbeat. | | `suspended` | The worker has declared it is awaiting one or more other promises. The server holds the task here until any awaited promise settles. | | `halted` | Admin-blocked state. A halted task can be returned to `pending` via `task.continue`. | | `fulfilled` | Terminal. The associated promise has settled and the task is complete. | `task.create` creates the task directly in `acquired` state. `promise.create` with a `resonate:target` tag creates the task in `pending` state. The version field increments only on the `pending → acquired` transition (`task.acquire`); it does not change on transitions back to `pending` (release, resume, lease expiry). All mutating task operations require the caller to present the current version. A version mismatch results in conflict (status 409) — the task has moved on without you, and the operation must be re-fetched and retried. ## Lease and heartbeat An acquired task carries a **lease**: a deadline by which the worker must signal liveness. The lease is set when the task is acquired and refreshed by every successful heartbeat. If the lease expires, the server releases the task back to `pending` (without incrementing the version), re-enqueues the execute message, and allows a different worker to claim it — at which point `task.acquire` will increment the version. ```text acquire → set lease = now + leaseTimeout, version++ heartbeat → set lease = now + leaseTimeout release → delete lease, transition back to pending (version unchanged) lease expiry → transition back to pending (version unchanged), re-enqueue execute ``` Heartbeat is **process-level, not task-level**. A worker sends one heartbeat for all the acquired tasks it holds; the server refreshes every matching `(id, version)` pair in a single round-trip. The heartbeat cadence is implementation-defined. A typical worker heartbeats at roughly half the lease interval to absorb transient network delay without losing the lease. ## Operations The server exposes the following task operations. Each operation takes the task's current `id` and `version` (where applicable) and returns the new state or an error. | Operation | Purpose | |---|---| | `task.get` | Retrieve a task by id (read-only). | | `task.create` | Create a task and its promise atomically; the task starts in `acquired`. The action must carry a `resonate:target` tag. | | `task.acquire` | Transition `pending → acquired`. Only the worker presenting the current version succeeds; concurrent workers see 409. | | `task.heartbeat` | Refresh the lease on one or more acquired tasks. Each `(id, version)` pair is processed independently; mismatches are silently skipped. | | `task.suspend` | Declare the worker is awaiting one or more promises. The server registers callbacks atomically and transitions the task to `suspended`. If any awaited promise has already settled, the server returns 300 — the worker should resume immediately without suspending. | | `task.fulfill` | Settle the task's promise and transition the task to `fulfilled` in one atomic operation. The action's promise id must equal the task id. | | `task.fence` | Run a `promise.create` or `promise.settle` guarded by the task's fencing token. The guard checks that the task is `acquired`, the associated promise is `pending` and not timed out, and the presented version matches — any failure returns 409. The `req.action` field selects which promise operation to execute. | | `task.release` | Surrender the claim. The task transitions back to `pending` (version unchanged), the server re-enqueues the `execute` message to the `resonate:target` address, and the task becomes available to other workers. The version increments when a successor calls `task.acquire`. | | `task.halt` | Administratively block the task from being claimed or resumed. | | `task.continue` | Reverse `task.halt`. | | `task.search` | Enumerate tasks (filtered by state, target, etc.). | ## Asymmetry with promises Task operations are **not idempotent** in the way promise operations are. A promise carries idempotency keys (`ikc`, `iku`) that allow safe retry after an unknown-outcome failure (see the [Durable Promise Specification](/spec/programming-model/durable-promise-specification#state-transitions)). A task carries only its version — retrying a successful `task.acquire` with the same version succeeds again at first but fails as soon as the successor's `task.acquire` increments the version. This asymmetry reflects the role of each: a promise represents *state* whose final value is what matters; a task represents *progress*, where each transition is a decisive moment of claim ownership. The version protects against duplicate progress; idempotency keys protect against duplicate state. ## Invariants A conformant server must maintain the following invariants over its task store: - **`orphan_tasks`** — every task must have a corresponding promise. - **`pending_task_no_ttimeout`** — every pending task must have a retry timeout. - **`acquired_task_no_lease`** — every acquired task must have a lease. - **`suspended_no_callback`** — every suspended task must have at least one callback registered on an awaited promise. - **`suspended_with_consumed_callbacks`** — no suspended task should have any consumed callbacks. - **`suspended_task_has_ttimeout`** — no suspended task should have any timeout. - **`fulfilled_task_has_ttimeout`** — no fulfilled task should have any timeout. Violations indicate a bug in the implementation, not a recoverable runtime condition. ## See also - [Processes](/spec/system-model/processes) — the failure model that motivates leases. - [Recovery Protocol](/spec/execution-model/recovery-protocol) — how tasks operationalize recovery. - [Coordination Protocol](/spec/execution-model/coordination-protocol) — how `task.suspend` and the settlement chain coordinate awaiting executions. - [Durable Promise Specification](/spec/programming-model/durable-promise-specification) — the state a task ultimately settles.