runner is any callable that takes an input and returns a final value. There is no framework allowlist — a plain function works, and nothing else is required to run an eval.
evaluate(...) opens an Eval: <name> span per case; any PromptLayer client used inside the runner nests under it automatically.
What the runner returns
Return the agent’s final answer as a single value.evaluate(...) records it as the case’s Output (stored on the eval span as the eval.output attribute), so it has to be a resolved value, not a live one:
- Prefer a primitive. Strings, numbers, and booleans are stored as-is — that’s why the framework examples wrap the result in
str(...). Any other object is JSON-serialized, so return something JSON-friendly. - No streams. A generator or async iterator is rejected — consume the stream and return its final text.
- No un-awaited coroutines. On the sync path a bare awaitable is rejected; use
aevaluate(...)(Python) or an async function (JavaScript) for async runners.
Default concurrency is
1; raise it with max_concurrency / maxConcurrency (see Concurrency). Copy-paste harnesses for common frameworks are in the Quickstart.
Tool spans (only for Trajectory)
Tool spans matter for exactly one thing: the Trajectory scorer, which grades the sequence of tools your agent called. It reads spans namedTool: <name> from the imported Trace and ignores everything else. Scorers that read the Output — Contains, Compare, LLM assertion — need no tool spans at all.
You don’t create those spans by hand. Emit them with either:
- A framework helper — OpenAI Agents, Claude, Vercel AI, and the rest in Telemetry Integrations — which instruments your tools automatically.
traceToolon a custom agent’s tool handlers. It records theTool: <name>span for you (on a tracing-enabled client); see Tracing Tools for the full contract.
accepted_scenarios) need a span — a helper tool you don’t score can stay untraced. How the modes treat what they observe: