Retry is not a policy

One provider told an Efferent run to retry after 48489 seconds — thirteen and a half hours, the distance to its midnight quota reset.

A generic exponential retry policy would either ignore that instruction and hammer the same wall, or honor it and park the agent until tomorrow. Both are mechanically faithful and operationally absurd.

“Has retries” is not a resilience property. The policy lives in the classification, the timeout boundary, and — once tokens stream — the point after which replay becomes more dangerous than failure.

The current taxonomy is deliberately small#

The previous runtime grew a five-class forensic taxonomy. The current line ports only the distinction its behavior actually uses:

export type ErrorClass = "transient" | "permanent"

Transient means the same request may succeed soon: a short 429, a 5xx, a transport failure, a timeout, or an HTTP 200 carrying no usable content. Permanent means retrying the same call unchanged cannot help: ordinary 4xx responses, malformed output, and long quota resets.

Anything unfamiliar defaults to permanent. That bias is important. A false transient classification multiplies traffic and latency; a false permanent classification surfaces one error to a human. The safer unknown is the one that stops.

export const classifyLlmError = (error: unknown): ErrorClass => {
  const e = error as ProviderError | null
  if (e === null || typeof e !== "object") return "permanent"

  if (e._tag === "HttpResponseError") {
    const status = e.response?.status ?? 0
    if (status === 429) {
      const wait = retryAfterMillis(e.response?.headers)
      return wait !== undefined && wait > 60_000
        ? "permanent" // quota wall, not a blip
        : "transient"
    }
    return status >= 500 ? "transient" : "permanent"
  }

  return e._tag === "UnknownError" || e._tag === "HttpRequestError"
    ? "transient"
    : "permanent"
}

The one-minute ceiling does not mean Efferent sleeps exactly as long as a short Retry-After; the fast schedule remains its own 1s → 2s → 4s exponential series with jitter. The header is used to recognize a quota wall, not to let an upstream choose how long a local fiber disappears.

Bound the request before retrying it#

Official provider clients do not all impose a request timeout. Efferent wraps every settled call in a five-minute Effect timeout, then applies three transient-only retries:

effect.pipe(
  Effect.timeoutFail({
    duration: Duration.millis(300_000),
    onTimeout: () => timeoutError(label, "generateText", 300_000),
  }),
  Effect.retry({
    schedule: fastRetries,
    while: (error) => classifyLlmError(error) === "transient",
  }),
)

Five minutes looks indulgent until the transport is non-streaming and the gateway’s model spends two minutes reasoning before returning its first body byte. A 120-second timeout classified healthy slow inference as failure and retried it into the same load. Timeout values describe the transport you actually use, not a universal notion of patience.

A successful HTTP response can still be a failed turn#

The nastiest provider failure in an agent loop is HTTP 200 with no text, reasoning, or tool call. If accepted, the loop reads “no tool calls” as completion and ends mid-task. Nothing throws; the agent simply fake-finishes.

Efferent rejects empty content before it reaches loop policy:

export const rejectEmptyResponse = (label: string) =>
  <A extends { readonly content: ReadonlyArray<unknown> }, E, R>(effect: Effect.Effect<A, E, R>) =>
    effect.pipe(
      Effect.filterOrFail(
        (response) => response.content.some(isContentPart),
        () => emptyError(label, "generateText"), 
      ),
    )

The synthesized error is transient because an empty body usually reflects a flaky provider or compatibility gateway, not a bad conversation. This is an example of classification by consequence rather than status code: a technically successful response is operationally retryable failure.

Streams change what retry means#

A settled call is all-or-nothing. If it fails, no response has reached the consumer, so replay is safe.

A stream is not. Once one text delta or tool call has flowed into the loop and UI, restarting the request can duplicate prose, tool calls, or both. The stream retry schedule therefore consults a Ref:

const contentSeen = yield* Ref.make(false)

return stream.pipe(
  Stream.mapEffect((part) =>
    isContentPartEncoded(part)
      ? Ref.set(contentSeen, true).pipe(Effect.as(part))
      : Effect.succeed(part),
  ),
  Stream.retry(
    fastRetries.pipe(
      Schedule.whileInputEffect((error) =>
        Ref.get(contentSeen).pipe(
          Effect.map(
            (seen) => !seen && classifyLlmError(error) === "transient", 
          ),
        ),
      ),
    ),
  ),
)

Before first content, retry still means “try to connect and begin.” After first content, retry means “replay visible output,” so the same transient failure becomes final. The error class did not change; the safety of the verb did.

That is why “retry transient errors” is incomplete advice. You also need to know whether the operation is still replayable.

Stream timeouts are per idle gap#

The settled path times the whole request. The streaming path times each wait for the next part. A healthy ten-minute response should not fail merely because it is long; a stream that stops emitting for five minutes should.

Stream.timeoutFail resets as parts flow, bounding both time-to-first-part and mid-stream silence. The same duration has different semantics because the observable progress signal differs.

The empty-response guard also changes shape for streams. The finish part is withheld until at least one content part has appeared. A finish-without-content becomes the same transient empty error, and because no content escaped, the schedule may replay safely.

One configured fallback rung#

After the primary exhausts its fast retries, the router may try one different human-configured fallbackModel — but only for a transient error:

Option.filter(
  fallback,
  (candidate) =>
    formatModelSelection(candidate) !== formatModelSelection(primary) &&
    classifyLlmError(error) === "transient",
)

Permanent failures pass through. An invalid request is likely invalid on the fallback too; a decode defect belongs to the conversation; an auth error needs a human. The router also refuses an identical fallback, preventing a typo from buying a doomed extra call.

The rung is intentionally one level deep. There is no fallback-of-fallback search and no agent-selected emergency model. If both configured choices fail, the error surfaces.

Fallback remains observable: provider spans and request metrics label the resolved model and whether the call was the fallback attempt. The finish part is also stamped with the actual model so the persisted conversation can answer who produced a message without consulting telemetry.

The policy is the boundary#

The costs are honest. Regex and status-based classification will trail provider behavior. A permanent default can surface an outage another retry might have survived. A fallback provider starts with a cold prompt cache. After streamed content begins, a momentary socket failure ends a turn that might have recovered if output buffering had delayed visibility.

Those are preferable to the hidden alternatives: thirteen-hour fibers, duplicate tool calls, fake-completed turns, and retry storms against invalid requests.

The useful review checklist is therefore not “how many retries?” It is:

• What observations make a failure transient here?
• What is the maximum time one attempt may occupy?
• Has any irreversible or externally visible output escaped?
• What counts as success beyond the status code?
• Who chose the fallback, and how many rungs may run?

Retry is the final verb. The engineering is everything that decides whether that verb is still safe.