> ## Documentation Index
> Fetch the complete documentation index at: https://docs.promptlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Runner and tracing

> Pass any callable as the evaluate(...) runner. Return a final value; use tool spans only for Trajectory.

The `runner` is any callable that takes an `input` and returns a final value. There is no framework allowlist — a plain function works, and nothing else is required to run an eval.

`evaluate(...)` opens an `Eval: <name>` span per case; any PromptLayer client used inside the runner nests under it automatically.

## What the runner returns

Return the agent's **final** answer as a single value. `evaluate(...)` records it as the case's `Output` (stored on the eval span as the `eval.output` attribute), so it has to be a resolved value, not a live one:

* **Prefer a primitive.** Strings, numbers, and booleans are stored as-is — that's why the framework examples wrap the result in `str(...)`. Any other object is JSON-serialized, so return something JSON-friendly.
* **No streams.** A generator or async iterator is rejected — consume the stream and return its final text.
* **No un-awaited coroutines.** On the sync path a bare awaitable is rejected; use `aevaluate(...)` (Python) or an async function (JavaScript) for async runners.

| Runner | Python           | JavaScript                                                     |
| ------ | ---------------- | -------------------------------------------------------------- |
| Sync   | `evaluate(...)`  | `evaluate(...)`                                                |
| Async  | `aevaluate(...)` | `evaluate(...)` with an async function (no `aevaluate` export) |

Default concurrency is `1`; raise it with `max_concurrency` / `maxConcurrency` (see [Concurrency](/sdks/evals/concurrency)). Copy-paste harnesses for common frameworks are in the [Quickstart](/sdks/evals/quickstart).

## Tool spans (only for Trajectory)

Tool spans matter for exactly one thing: the [Trajectory](/sdks/evals/scorers/overview#trajectory) scorer, which grades the sequence of tools your agent called. It reads spans named `Tool: <name>` from the imported Trace and ignores everything else. Scorers that read the `Output` — Contains, Compare, LLM assertion — need no tool spans at all.

You don't create those spans by hand. Emit them with either:

* **A framework helper** — OpenAI Agents, Claude, Vercel AI, and the rest in [Telemetry Integrations](/features/integrations) — which instruments your tools automatically.
* **`traceTool`** on a custom agent's tool handlers. It records the `Tool: <name>` span for you (on a tracing-enabled client); see [Tracing Tools](/running-requests/traces#tracing-tools) for the full contract.

Only the tools you assert on (the names in `accepted_scenarios`) need a span — a helper tool you don't score can stay untraced. How the modes treat what they observe:

| `mode`       | Rule                                                          | Untraced / extra tools                                                                     |
| ------------ | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `strict`     | Observed tools must **exactly equal** the expected list.      | An expected tool that isn't traced fails; every traced tool must be in your expected list. |
| `non_strict` | Expected list must be an ordered **subsequence** of observed. | Extra traced tools are tolerated; only an untraced *expected* tool fails.                  |

## Common errors

| Issue                                | Fix                                                                                                                                                      |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Empty Trace / Trajectory fails       | Emit `Tool:` spans for the tools you assert on — a framework helper or [`traceTool`](/running-requests/traces#tracing-tools) on a tracing-enabled client |
| Traced tool missing in `strict` mode | Add it to the expected list, or switch to `non_strict`                                                                                                   |
| Streamed return                      | Collect the final answer before returning it                                                                                                             |
| Awaitable on the sync Python path    | Use `aevaluate`                                                                                                                                          |