Observability is built in; where it goes is a layer

Here’s the question that started this. A single agent run cost $2.14 and took fifty seconds, and I wanted to know where the seconds and the cents went — which turn, which model call, which tool. The usual way to answer that is to bolt on observability: install a vendor’s OpenTelemetry SDK, build a tracer, reach for it (or an ambient global) at every call site you care about, wrap the interesting ones in tracer.startActiveSpan(…), and remember to .end() each one on every exit path — including the one where it threw. Then you carry that machinery, and its cost, whether or not anyone is looking.

In an Effect program you don’t do any of that, and the reason is the point of this post. Tracing, metrics, and logs aren’t a library you add. They’re built into the runtime the same way typed errors and structured concurrency are. You don’t instrument an Effect program — you describe what’s worth observing as ordinary values, and a layer at the edge decides whether, and where, anyone records them.

Everything below is real code from efferent, the agent system I’m building on Effect. The receipts come from the live engine, providers, and Smith composition root.

One paragraph of Effect for readers who haven’t met it. An Effect<A, E, R> is a description of a program — inert until run — that succeeds with an A, can fail with a typed E, and needs the services in R to execute. A Layer is a recipe that builds those services, supplied once at the program’s edge, which is what makes swapping an implementation (real exporter, file, nothing) a one-line change at one location. Hold onto that last clause — it’s the whole argument.

The first built-in: a span is a combinator, not an object#

A span is the unit of a trace — a named, timed interval with attributes, nested under whatever span was open when it started. In the SDK world a span is an object you create, hold, and end. In Effect it’s a combinator you wrap around a description. The agent loop wraps each turn, and the whole run:

const outcome = yield* (useStreaming ? streamed : settled).pipe(
  Effect.withSpan('engine.turn', {
    attributes: { 'engine.turn': state.turnIndex },
  }),
)

// …and the loop itself, at the bottom of the file:
}).pipe(Effect.withSpan('engine.run'))

Effect.withSpan takes a description and returns a new description that, when run, opens the span, runs the inner effect, and closes the span — on success, on failure, and on interruption, because the span’s lifetime is the effect’s lifetime and the runtime owns both. There is no .end() to forget. Cancel the run halfway through and its span closes with the cancellation; the finally you’d have to write by hand is structural.

Two things follow that don’t in the SDK model. First, the span context is ambient, carried on the fiber, not passed as an argument. A tool handler thirty calls deep opens a child span without anyone threading a tracer down to it — the child nests under the parent automatically because “the current span” is fiber state. Second, you can reach the active span and annotate it mid-effect, which is how the run’s verdict lands on the run span after the loop that computed it has finished:

// The run span states its verdict AND its trajectory vitals — a
// "completed" run with 25 steps, five failed calls, and three
// corrections is a low-quality success visible at a glance.
yield* Effect.annotateCurrentSpan({
  'engine.outcome': outcome,
  'engine.reason': reason,
  'engine.turns': final.turnIndex,
  'engine.usage.total_tokens': final.usage.totalTokens,
  'engine.tool_calls': final.toolCalls,
  'engine.tool_calls.failed': final.toolFailures,
  'engine.corrections': final.corrections,
  error: outcome !== 'ok',
})

annotateCurrentSpan is a write to whatever span is currently open on the fiber. No handle, no global lookup — the runtime has been carrying the span the whole time. Which spans an agent should open, and what belongs on them, is the next post; this one is about the fact that opening one is free of ceremony.

The second built-in: a metric is a value you define once and record into#

Effect’s Metric is a first-class value: you define it once as a module-level constant and record into it at the chokepoints. Every LLM request in efferent crosses one seam — the provider router — so that seam is where tokens and latency are counted, once, for every agent in the system:

/**
 * The routed-call metrics — every LLM request crosses this seam, so this
 * is the ONE place tokens and latency are counted. Tagged per resolved
 * model; exported by TracingLive's metric reader.
 */
const llmInputTokens = Metric.counter('llm.usage.input_tokens', {
  description: 'prompt tokens consumed by routed LLM calls',
  incremental: true,
})
const llmRequests = Metric.counter('llm.requests', {
  description: 'routed LLM calls by final outcome (after retries)',
  incremental: true,
})
/** Wall-clock per routed call INCLUDING retries, in millisecond buckets
 *  spanning a fast cached turn (100ms) to the 300s request timeout. */
const llmDuration = Metric.timerWithBoundaries(
  'llm.request.duration',
  [100, 250, 500, 1_000, 2_500, 5_000, 10_000, 30_000, 60_000, 120_000, 300_000],
)

A Metric.counter is not a vendor handle — it is Effect’s own metric value, recorded into the runtime registry. That registry is useful even in tests (Metric.value can inspect it); the edge layer decides whether a reader exports it anywhere. Production code only ever records; where the numbers go is a layer choice, exactly like spans.

Recording is where the one real discipline of metrics lives: tags. A tag is a dimension you can later group by, and tags multiply — every distinct combination of tag values is a separate time series your backend must store. Effect makes tagging composable (Metric.tagged returns a new tagged metric), so the cardinality of a metric is visible at the record site, in code you can review:

const byModel = (metric, label: string) => Metric.tagged(metric, 'llm.model', label)

// at the call site, on the way out of the provider call:
Effect.tapBoth({
  onFailure: () =>
    Metric.increment(
      Metric.tagged(byModel(llmRequests, label), 'outcome', 'error'), 
    ),
  onSuccess: () =>
    Metric.increment(Metric.tagged(byModel(llmRequests, label), 'outcome', 'ok')),
}),
Metric.trackDuration(byModel(llmDuration, label)),

Every tag in the system is a small closed set: llm.model is bounded by the models you’ve configured, outcome is ok | error, the loop’s engine.outcome is ok | partial, its engine.kind correction tag is malformed | degenerate-nudge, and the judge gate’s verdict counter is sound | unsound | crash — three values by construction. The product stays in the dozens of series, not the millions. Tag a counter with a request id or a free-text error message and you’ve built a cardinality bomb that looks fine in dev and takes the metrics backend down a week later; the closed-set rule is the entire defense, and this model at least makes the rule reviewable at each record site.

The third built-in: logs are structured, and their destination is swappable too#

Effect’s logger is built in — Effect.logInfo, logWarning, logError emit structured records that carry fiber context, not strings on stdout. The agent loop logs at exactly the moments you’d want a paper trail: recovering from a malformed model response, breaking a degenerate tool-call loop, a transient provider failure exhausting its retries.

The interesting part is, again, the edge. efferent’s TUI owns the terminal — any console write corrupts the alternate screen — so for a while the interactive path silenced logs entirely, which meant a mid-run failure left no trace anywhere: a session that “just died” was undiagnosable. The fix wasn’t touching a single log statement; it was one layer:

export const FileLoggerLive = (path: string): Layer.Layer<never> =>
  Logger.replaceScoped(
    Logger.defaultLogger,
    Effect.gen(function* () {
      const fs = yield* FileSystem.FileSystem
      yield* fs
        .makeDirectory(dirname(path), { recursive: true })
        .pipe(Effect.orElseSucceed(() => undefined))
      return yield* PlatformLogger.toFile(Logger.logfmtLogger, path, { flag: 'a' })
      // An unopenable log file must never fail the boot — fall back to silent
      // (the pre-file behavior), the session is more important than its log.
    }).pipe(Effect.orElseSucceed(() => Logger.none)),
  )

The TUI now routes the default logger to an append-only logfmt file; the headless mode routes it to stderr (stdout is the event stream, and polluting it corrupts the protocol) and keeps the same file sink for crash forensics; tests keep the default. Same log statements everywhere — the destination is decided per app surface, at composition, by swapping one layer. And note the fallback: if the log file can’t be opened, the logger degrades to silent rather than failing the boot. Observability code must never become the outage.

The layer: composed once per app, and “off” is structural#

Here’s the payoff. The instrumented kernel — the spans, the metric records, the logs — does not name a backend anywhere. It can’t: a withSpan or a Metric.update is just a description. Something has to provide a tracer and a meter for any of it to leave the process, and that something is one layer:

export const TracingLive = (serviceName: string): Layer.Layer<never> =>
  NodeSdk.layer(() => ({
    resource: { serviceName },
    spanProcessor: [new BatchSpanProcessor(new OTLPTraceExporter())],
    // Effect's Metric registry (the router's llm.* counters + timer) rides
    // the same OTLP endpoint into Prometheus, every 5s.
    metricReader: new PeriodicExportingMetricReader({
      exporter: new OTLPMetricExporter(),
      exportIntervalMillis: 5000,
    }),
  }))

Each agent app composes it once, at the very bottom, next to its logger choice:

program.pipe(
  Effect.provide(smithAppLive(run)),
  // The kernel's spans (engine.run/turn, providers.generate) reach the local
  // LGTM stack when it's up (`bun run obs:up`); a missing collector fails
  // silently — the agent stays up even when observability is unavailable.
  Effect.provide(TracingLive('smith')), 
  Effect.provide(
    interactive
      ? FileLoggerLive(join(run.cwd, '.efferent', 'logs', 'smith.log'))
      : Logger.replace(Logger.defaultLogger, Logger.prettyLogger({ stderr: true })),
  ),
)

smith, math, canvas, social — four agent apps, each one line of TracingLive('<name>'), each showing up in Grafana under its own service name. The exporter reads the standard OTEL_EXPORTER_OTLP_ENDPOINT env itself and defaults to http://localhost:4318; where spans go is the collector’s address, not anything the instrumented code knows.

And “off” exists at two different depths. A program that does not provide TracingLive — unit tests and scripted scenario runs in CI — exports no spans or metrics. Its Effects and metric updates still execute; what is absent is the OpenTelemetry SDK, batching, and network path. In production the layer is composed unconditionally, and when no collector is listening batch exports fail silently (no diagnostic logger is registered — deliberately). That is a liveness choice, not a claim of zero overhead: spans are still built, metrics still recorded, and batches still attempted because operational consistency is worth the small local cost.

What it costs#

You wire the layer, and you wire it correctly. The convenience of “describe everywhere, decide at the edge” is paid for at that edge: the composition root has to provide the right layers, and Layer type errors are the steepest part of Effect’s learning curve. The flip side is that once it compiles, it’s correct — the tracer can’t be half-initialized, can’t leak, can’t be present in one code path and absent in another.

Pure recording means deliberate recording. You don’t get metrics by accident — you get exactly the ones you wrote, at exactly the chokepoints you chose. efferent’s whole metric surface is seven counters and one timer: three at the router seam, three in the loop, one on the judge gate. Eight decisions someone made about what’s worth counting, and the whole inventory fits in one sentence. Auto-instrumentation gives you a hundred metrics you didn’t choose; for an agent, where the interesting quantities are domain-specific — tokens by model, runs by outcome, self-corrections by kind — choosing beats guessing. For a generic CRUD service, you might miss the convenience.

Silent failure cuts both ways. The no-collector-no-cost posture means a typo’d endpoint also fails silently — there’s no red light for “your spans are going nowhere.” That’s a real trade, made on purpose, and it’s only safe because the local stack is one docker compose away from confirming data is flowing.

You describe; the edge decides#

Step back and the three built-ins are one idea wearing three hats. A span, a metric, a log record are all values — descriptions of something worth observing — and none of them names where it goes. The naming happens once per app, at the composition root: OTLP into the local Grafana stack, a logfmt file because the TUI owns the screen, stderr because stdout is the protocol, or nothing at all in a test. Swap the layer and the same instrumented program lights up a different backend, or none, without a line changing in the code that does the work.

That’s the inversion. Bolted-on observability makes every instrumented function know about the tracer; it threads a dependency through your domain logic and bills you for it on every call. Effect makes the domain logic describe what’s interesting and stay ignorant of who’s listening. The spans don’t know where they’re going. That’s the feature.

The next post takes these built-ins into the agent loop and asks the operational question: given that opening a span is free, which spans, carrying which attributes, make an agent you can debug at 2 a.m. — and why the attributes outlive every rename, including the one where the whole runtime got rewritten.