> ## Documentation Index
> Fetch the complete documentation index at: https://docs.promptlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scorers

> Add scorecard checks to evaluate(...) with typed helpers.

Pass scorers into `evaluate(...)` to grade each case. On the experiment sheet they show up as scorecard evaluators (Contains, Trajectory, and the rest).

Thresholds, weights, and the overall score work the same as [Scorecards](/features/tables/scorecards) in Tables. The underlying check types are documented in [Column Types](/features/tables/column-types).

## Scorers at a glance

| Scorer                          | Python                             | JavaScript                       | Checks                                                      |
| ------------------------------- | ---------------------------------- | -------------------------------- | ----------------------------------------------------------- |
| [Compare](#compare)             | `compare_scorer`                   | `compareScorer`                  | Two columns match (default `Output` vs `Expected`)          |
| [Contains](#contains)           | `contains_scorer`                  | `containsScorer`                 | A column contains a literal or another column's value       |
| [Regex](#regex)                 | `regex_scorer`                     | `regexScorer`                    | A column matches a regular expression                       |
| [Count](#count)                 | `count_scorer`                     | `countScorer`                    | Character / word / sentence / paragraph count within bounds |
| [Assert Valid](#assert-valid)   | `assert_valid_scorer`              | `assertValidScorer`              | A column is a valid object, number, or SQL                  |
| [LLM Assertion](#llm-assertion) | `llm_assertion_scorer`             | `llmAssertionScorer`             | An LLM judge grades a column                                |
| [Trajectory](#trajectory)       | `trajectory_scorer`                | `trajectoryScorer`               | Tool-call sequence matches accepted scenarios               |
| [Custom Code](#custom-code)     | named fn / `code_execution_column` | named fn / `codeExecutionColumn` | Your own scoring logic                                      |

Every typed helper also accepts [shared scoring options](#shared-scoring-options) — weight, required, and thresholds.

## Shared scoring options

Every scorer produces a score in `[0, 1]`. These three options — accepted by every typed helper — control how much a scorer counts and where its pass/fail line sits. They behave exactly like [Scorecards](/features/tables/scorecards) in Tables.

* **Weight** — how much this scorer contributes to the overall score. The scorecard is a weighted mean, so `weight=2` pulls twice as hard as the default `weight=1`. Use it to make the checks you care about most dominate the result.
* **Required** — if `true`, this scorer failing fails the whole scorecard, no matter how well the weighted average comes out. Use it for must-pass checks (e.g. the agent called the right tools).
* **Thresholds** — the per-scorer cutoffs that decide pass vs. warn vs. fail, each in `[0, 1]` (default pass `0.8`, warn `0.6`). Raise the pass threshold for a stricter bar on that check.

| Option     | Python              | JavaScript         | Meaning                                             |
| ---------- | ------------------- | ------------------ | --------------------------------------------------- |
| Weight     | `weight`            | `weight`           | How much this step counts (default `1`)             |
| Required   | `required`          | `required`         | If `true`, failing this step can fail the scorecard |
| Thresholds | `thresholds`        | `thresholds`       | Per-step pass / warn cutoffs in `[0, 1]`            |
| Pass alias | `pass_threshold`    | `passThreshold`    | Sets `thresholds.pass`                              |
| Warn alias | `failure_threshold` | `failureThreshold` | Sets `thresholds.warn`                              |

This support agent example weights the trajectory check higher, marks it required, and sets a pass threshold:

<CodeGroup>
  ```python Python theme={null}
  from agents import Agent, Runner, function_tool
  from promptlayer import evaluate, contains_scorer, trajectory_scorer
  from promptlayer.integrations.openai_agents import instrument_openai_agents

  instrument_openai_agents()

  @function_tool
  def lookup_user(email: str) -> str:
      """Look up a user account by email."""
      return f"Found user for {email}."

  @function_tool
  def send_reset(email: str) -> str:
      """Send a password reset email."""
      return f"Reset link sent to {email}."

  agent = Agent(
      name="Support agent",
      instructions=(
          "Help with password resets. Call lookup_user, then send_reset, "
          "then apologize briefly and confirm the reset was sent."
      ),
      model="gpt-5.6",
      tools=[lookup_user, send_reset],
  )

  def run_agent(user_message: str) -> str:
      return str(Runner.run_sync(agent, user_message).final_output)

  evaluate(
      "support-bot",
      dataset=[{"input": "Please reset my password for user@example.com"}],
      runner=run_agent,
      scorers=[
          contains_scorer(
              "Has apology",
              source="Output",
              value="sorry",
              weight=1.0,
          ),
          trajectory_scorer(
              accepted_scenarios=[["lookup_user", "send_reset"]],
              weight=2.5,
              required=True,
              pass_threshold=0.9,
          ),
      ],
  )
  ```

  ```javascript JavaScript theme={null}
  import { Agent, run, tool } from "@openai/agents";
  import { instrumentOpenAIAgents } from "promptlayer/openai-agents";
  import { evaluate, containsScorer, trajectoryScorer } from "promptlayer";
  import { z } from "zod";

  await instrumentOpenAIAgents();

  const lookupUser = tool({
    name: "lookup_user",
    description: "Look up a user account by email.",
    parameters: z.object({ email: z.string() }),
    execute: async ({ email }) => `Found user for ${email}.`,
  });

  const sendReset = tool({
    name: "send_reset",
    description: "Send a password reset email.",
    parameters: z.object({ email: z.string() }),
    execute: async ({ email }) => `Reset link sent to ${email}.`,
  });

  const agent = new Agent({
    name: "Support agent",
    instructions:
      "Help with password resets. Call lookup_user, then send_reset, then apologize briefly and confirm the reset was sent.",
    model: "gpt-5.6",
    tools: [lookupUser, sendReset],
  });

  async function runAgent(userMessage) {
    return String((await run(agent, userMessage)).finalOutput);
  }

  await evaluate("support-bot", {
    dataset: [{ input: "Please reset my password for user@example.com" }],
    runner: runAgent,
    scorers: [
      containsScorer({
        title: "Has apology",
        source: "Output",
        value: "sorry",
        weight: 1.0,
      }),
      trajectoryScorer({
        acceptedScenarios: [["lookup_user", "send_reset"]],
        weight: 2.5,
        required: true,
        passThreshold: 0.9,
      }),
    ],
  });
  ```
</CodeGroup>

## Scorer types

Each helper below maps to a scorecard check type in [Column Types](/features/tables/column-types). All of them accept the [shared scoring options](#shared-scoring-options) above.

### Compare

Compare two columns. By default that is `Output` vs `Expected` as strings. Pass exactly two column titles in `sources`. Behavior matches the Compare type in [Column Types](/features/tables/column-types).

Set how they compare with `comparison_type` / `comparisonType`:

| Mode               | What it does                                       | Example                                 |
| ------------------ | -------------------------------------------------- | --------------------------------------- |
| `STRING` (default) | Exact string match                                 | `"STRING"`                              |
| `JSON`             | Compare JSON values; optional path                 | `{"type": "JSON", "json_path": "$.id"}` |
| `NUMBER`           | Compare numbers with an operator (`EQ` by default) | `{"type": "NUMBER", "operator": "GTE"}` |

Number operators: `EQ`, `NE`, `GT`, `GTE`, `LT`, `LTE`.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, compare_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "Reply with exactly one word: hello",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "compare-demo",
      dataset=[{"input": "Greet the user.", "expected": "hello"}],
      runner=run_llm,
      scorers=[
          compare_scorer(
              "Exact match",
              sources=["Output", "Expected"],
              comparison_type="STRING",
          )
      ],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, compareScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        { role: "system", content: "Reply with exactly one word: hello" },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("compare-demo", {
    dataset: [{ input: "Greet the user.", expected: "hello" }],
    runner: runLlm,
    scorers: [
      compareScorer({
        title: "Exact match",
        sources: ["Output", "Expected"],
        comparisonType: "STRING",
      }),
    ],
  });
  ```
</CodeGroup>

### Contains

Check whether a column contains a literal or another column's value. Pass exactly one of `value` or `value_source` / `valueSource`.

| Key                            | Required | Default           |
| ------------------------------ | -------- | ----------------- |
| `source`                       | yes      | `"Output"`        |
| `value`                        | one of   | literal substring |
| `value_source` / `valueSource` | one of   | column title      |

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, contains_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "You are a support agent. Apologize and help in one short sentence.",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "contains-demo",
      dataset=[{"input": "I cannot log in to my account."}],
      runner=run_llm,
      scorers=[contains_scorer(source="Output", value="sorry")],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, containsScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content: "You are a support agent. Apologize and help in one short sentence.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("contains-demo", {
    dataset: [{ input: "I cannot log in to my account." }],
    runner: runLlm,
    scorers: [containsScorer({ source: "Output", value: "sorry" })],
  });
  ```
</CodeGroup>

### Regex

Match a column against a regular expression.

| Key                              | Required | Default    |
| -------------------------------- | -------- | ---------- |
| `source`                         | yes      | `"Output"` |
| `regex_pattern` / `regexPattern` | yes      | —          |

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, regex_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": (
                      "You are an order lookup bot. Reply with only an order id "
                      "like order-123."
                  ),
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "regex-demo",
      dataset=[{"input": "What is the order id for my headphones?"}],
      runner=run_llm,
      scorers=[regex_scorer(source="Output", regex_pattern=r"order-\d+")],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, regexScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content:
            "You are an order lookup bot. Reply with only an order id like order-123.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("regex-demo", {
    dataset: [{ input: "What is the order id for my headphones?" }],
    runner: runLlm,
    scorers: [regexScorer({ source: "Output", regexPattern: "order-\\d+" })],
  });
  ```
</CodeGroup>

### Count

Score by character, word, sentence, or paragraph count bounds. Require at least one of `min_count` / `minCount` or `max_count` / `maxCount`. Backend `type` values: `chars`, `words`, `sentences`, `paragraphs`.

| Key                      | Required       | Default    |
| ------------------------ | -------------- | ---------- |
| `source`                 | yes            | `"Output"` |
| `type`                   | no             | `"chars"`  |
| `min_count` / `minCount` | one of min/max | —          |
| `max_count` / `maxCount` | one of min/max | —          |

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, count_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "Answer in one short sentence of at most five words.",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "count-demo",
      dataset=[{"input": "How do I reset my password?"}],
      runner=run_llm,
      scorers=[count_scorer(source="Output", type="words", min_count=1, max_count=5)],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, countScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content: "Answer in one short sentence of at most five words.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("count-demo", {
    dataset: [{ input: "How do I reset my password?" }],
    runner: runLlm,
    scorers: [
      countScorer({ source: "Output", type: "words", minCount: 1, maxCount: 5 }),
    ],
  });
  ```
</CodeGroup>

### Assert Valid

Validate a column as object, number, or SQL. There is no `json` / `boolean` / `string` assert type.

| Key      | Required | Default    | Allowed                       |
| -------- | -------- | ---------- | ----------------------------- |
| `source` | yes      | `"Output"` | column title                  |
| `type`   | yes      | `"object"` | `object` \| `number` \| `sql` |

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, assert_valid_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": (
                      "Reply with a JSON object only, no markdown. "
                      'Example: {"status":"ok","ticket_id":101}'
                  ),
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "assert-valid-demo",
      dataset=[{"input": "Create a support ticket for a billing issue."}],
      runner=run_llm,
      scorers=[assert_valid_scorer(source="Output", type="object")],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, assertValidScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content:
            'Reply with a JSON object only, no markdown. Example: {"status":"ok","ticket_id":101}',
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("assert-valid-demo", {
    dataset: [{ input: "Create a support ticket for a billing issue." }],
    runner: runLlm,
    scorers: [assertValidScorer({ source: "Output", type: "object" })],
  });
  ```
</CodeGroup>

### LLM Assertion

Score a column with an LLM judge prompt. Pass exactly one of `prompt` or `prompt_source` / `promptSource`.

| Key                                      | Required | Default                        |
| ---------------------------------------- | -------- | ------------------------------ |
| `source`                                 | yes      | `"Output"`                     |
| `prompt`                                 | one of   | literal prompt text            |
| `prompt_source` / `promptSource`         | one of   | column title                   |
| `variable_mappings` / `variableMappings` | no       | Prompt variable → column title |

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, llm_assertion_scorer

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "Answer the user's question in one short factual sentence.",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "llm-assert-demo",
      dataset=[{"input": "Why is the sky blue?"}],
      runner=run_llm,
      scorers=[
          llm_assertion_scorer(
              source="Output",
              prompt="Is this factually reasonable? Reply yes or no.",
          )
      ],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, llmAssertionScorer } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content: "Answer the user's question in one short factual sentence.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("llm-assert-demo", {
    dataset: [{ input: "Why is the sky blue?" }],
    runner: runLlm,
    scorers: [
      llmAssertionScorer({
        source: "Output",
        prompt: "Is this factually reasonable? Reply yes or no.",
      }),
    ],
  });
  ```
</CodeGroup>

### Trajectory

Scores `Tool: <name>` spans from the imported `Trace` against accepted scenarios. Any `runner` works if those spans exist — emit them with a framework helper or [`traceTool`](/running-requests/traces#tracing-tools) (see [Runner](/sdks/evals/agent-tracing)), or copy a harness from the [Quickstart](/sdks/evals/quickstart).

Pass **exactly one of** inline `accepted_scenarios` / `acceptedScenarios`, or `expected_source` / `expectedSource` (usually `"Expected Trace"`).

| Key                                        | Required | Default                           |
| ------------------------------------------ | -------- | --------------------------------- |
| `trace_source` / `traceSource`             | yes      | `"Trace"`                         |
| `accepted_scenarios` / `acceptedScenarios` | one of   | list of non-empty tool-name lists |
| `expected_source` / `expectedSource`       | one of   | column title                      |
| `mode`                                     | no       | `"strict"`                        |

| Mode               | Behavior                                        |
| ------------------ | ----------------------------------------------- |
| `strict` (default) | Observed tools must equal the scenario exactly  |
| `non_strict`       | Required tools appear in order as a subsequence |

Inline `accepted_scenarios`:

<CodeGroup>
  ```python Python theme={null}
  from agents import Agent, Runner, function_tool
  from promptlayer import evaluate, trajectory_scorer, contains_scorer
  from promptlayer.integrations.openai_agents import instrument_openai_agents

  instrument_openai_agents()

  @function_tool
  def get_weather(city: str) -> str:
      """Return demo weather for a city."""
      return f"{city} is 72F and sunny."

  agent = Agent(
      name="Weather agent",
      instructions="Use get_weather, then answer in one short sentence.",
      model="gpt-5.6",
      tools=[get_weather],
  )

  evaluate(
      "trajectory-demo",
      dataset=[{"input": "What is the weather in Tokyo?"}],
      runner=lambda message: str(Runner.run_sync(agent, message).final_output),
      scorers=[
          trajectory_scorer(accepted_scenarios=[["get_weather"]], mode="non_strict"),
          contains_scorer(source="Output", value="72"),
      ],
  )
  ```

  ```javascript JavaScript theme={null}
  import { Agent, run, tool } from "@openai/agents";
  import { instrumentOpenAIAgents } from "promptlayer/openai-agents";
  import { evaluate, trajectoryScorer, containsScorer } from "promptlayer";
  import { z } from "zod";

  await instrumentOpenAIAgents();

  const getWeather = tool({
    name: "get_weather",
    description: "Return demo weather for a city.",
    parameters: z.object({ city: z.string() }),
    execute: async ({ city }) => `${city} is 72F and sunny.`,
  });

  const agent = new Agent({
    name: "Weather agent",
    instructions: "Use get_weather, then answer in one short sentence.",
    model: "gpt-5.6",
    tools: [getWeather],
  });

  await evaluate("trajectory-demo", {
    dataset: [{ input: "What is the weather in Tokyo?" }],
    runner: async (userMessage) => String((await run(agent, userMessage)).finalOutput),
    scorers: [
      trajectoryScorer({ acceptedScenarios: [["get_weather"]], mode: "non_strict" }),
      containsScorer({ source: "Output", value: "72" }),
    ],
  });
  ```
</CodeGroup>

Or read scenarios from an Expected Trace column. Cell JSON:

```json theme={null}
{
  "accepted_scenarios": [
    { "required_tools": ["get_weather"] }
  ]
}
```

<CodeGroup>
  ```python Python theme={null}
  trajectory_scorer(
      expected_source="Expected Trace",
      mode="non_strict",
  )
  ```

  ```javascript JavaScript theme={null}
  trajectoryScorer({
    expectedSource: "Expected Trace",
    mode: "non_strict",
  })
  ```
</CodeGroup>

Put that JSON on each case as `expected_trace` / `expectedTrace` (or on a dashboard dataset sheet).

Local helpers: `score_trajectory` / `scoreTrajectory`, `diagnose_trajectory_failure` / `diagnoseTrajectoryFailure`, and JS-only `extractTrajectoryToolNames`.

### Custom Code

Prefer passing a **named function by reference** in `scorers`. The SDK serializes it to a `CODE_EXECUTION` scorecard step.

Parameter names map to columns:

| Param      | Column     |
| ---------- | ---------- |
| `input`    | `Input`    |
| `output`   | `Output`   |
| `expected` | `Expected` |
| `trace`    | `Trace`    |

Any other param name is looked up as a column title of the same name. Return a number (or `{"score": x}`, which is normalized to `x`).

* Use a **named** function — no lambdas / anonymous functions.
* Sync only — async scorers are rejected.
* Default title is the function name with `_` → spaces. Override with `scorer_from_function(fn, title=...)` / `scorerFromFunction(fn, { title })` when needed.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "You are a support agent. Answer in one short sentence.",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  def exact_match(output, expected):
      return 1 if expected.lower() in (output or "").lower() else 0

  evaluate(
      "fn-scorer-demo",
      dataset=[
          {
              "input": "How do I reset my password?",
              "expected": "password",
          }
      ],
      runner=run_llm,
      scorers=[exact_match],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content: "You are a support agent. Answer in one short sentence.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  function exactMatch(output, expected) {
    return String(output || "")
      .toLowerCase()
      .includes(String(expected).toLowerCase())
      ? 1
      : 0;
  }

  await evaluate("fn-scorer-demo", {
    dataset: [
      {
        input: "How do I reset my password?",
        expected: "password",
      },
    ],
    runner: runLlm,
    scorers: [exactMatch],
  });
  ```
</CodeGroup>

#### `code_execution_column` / `codeExecutionColumn`

Use this when you want a raw code string instead of a function. `code` is required. Default `language` is `PYTHON` (Python helper) or `JAVASCRIPT` (JS helper). Row values are available as `data`.

<CodeGroup>
  ```python Python theme={null}
  from openai import OpenAI
  from promptlayer import evaluate, code_execution_column

  client = OpenAI()

  def run_llm(user_message: str) -> str:
      completion = client.chat.completions.create(
          model="gpt-5.6",
          messages=[
              {
                  "role": "system",
                  "content": "You are a support agent. Answer in one short sentence.",
              },
              {"role": "user", "content": str(user_message)},
          ],
      )
      return completion.choices[0].message.content or ""

  evaluate(
      "code-scorer-demo",
      dataset=[
          {
              "input": "How do I reset my password?",
              "expected": "password",
          }
      ],
      runner=run_llm,
      scorers=[
          code_execution_column(
              "exact",
              code=(
                  'return 1 if str(data.get("Expected", "")).lower() '
                  'in str(data.get("Output", "")).lower() else 0'
              ),
          )
      ],
  )
  ```

  ```javascript JavaScript theme={null}
  import OpenAI from "openai";
  import { evaluate, codeExecutionColumn } from "promptlayer";

  const client = new OpenAI();

  async function runLlm(userMessage) {
    const completion = await client.chat.completions.create({
      model: "gpt-5.6",
      messages: [
        {
          role: "system",
          content: "You are a support agent. Answer in one short sentence.",
        },
        { role: "user", content: String(userMessage) },
      ],
    });
    return completion.choices[0].message.content ?? "";
  }

  await evaluate("code-scorer-demo", {
    dataset: [
      {
        input: "How do I reset my password?",
        expected: "password",
      },
    ],
    runner: runLlm,
    scorers: [
      codeExecutionColumn("exact", {
        code: 'return String(data["Output"] || "").toLowerCase().includes(String(data["Expected"] || "").toLowerCase()) ? 1 : 0;',
      }),
    ],
  });
  ```
</CodeGroup>