Scorecards

Teams ship prompts quickly, then hit a quality wall. Results drift. Edge cases slip in. Regressions reach users before anyone sees the pattern. Scorecards give a sheet a repeatable evaluation contract. You define the checks that matter, run them against rows, and review the exact failures with evidence.

Open the scorecard panel

Open a table and click the score badge in the toolbar. The scorecard panel shows the current aggregate score, verdict, last calculation time, row evaluator status, and failing criteria.

Scorecard panel showing a 25 percent failing quality score, failing evaluators, and row evaluator summaries

The top score is the sheet-level result. Each evaluator contributes to that result based on its weight and thresholds. A failed required evaluator can fail the full scorecard, depending on the aggregation settings.

Add evaluators

Click Add evaluator to choose how each row should be graded.

Add row evaluator menu with AI Judge, Code Check, Compare Values, Contains Text, Regex Match, and Valid Format options

Available evaluator types include:

Evaluator	Use when
AI Judge	You need rubric-based grading for subjective output quality.
Code Check	You need deterministic checks with custom logic.
Compare Values	You need expected column, numeric threshold, or JSON comparison checks.
Contains Text	You need required words, phrases, or facts.
Regex Match	You need exact pattern checks for structured strings.
Valid Format	You need JSON, XML, or schema-style validation.

You can also add a column summary. Column summaries run once across the whole column. Row evaluators run once per row, then roll up into the overall score.

Configure an evaluator

Each evaluator has a name, source columns, matching rules, thresholds, and weight. In this example, Contains Text checks whether the Output column contains the expected value from another column.

Edit evaluator form for a Contains Text evaluator with source column, match source, column to match, and weight

Use weights to make important checks count more. A weight of 2 counts twice as much as a standard evaluator when scores are averaged.

Review all rows

After a calculation finishes, open All rows to find failures fast. Filter by verdict, search rows, then open a row to see why it failed.

All rows view with search, verdict filters, and row-level scorecard results

Rows show score, verdict, and drift when a previous calculation is available. This makes regression review concrete. You can find the rows that changed, inspect the evidence, and decide whether to update the prompt, data, or evaluator.

Inspect row evidence

Open a row to see the evaluated inputs, row score, failed evaluators, and per-evaluator results.

Row scorecard breakdown showing evaluated inputs, a 50 percent row score, and one failed evaluator

Expand a failed evaluator to see the result and evidence. In the example below, the expected value is 30 days, and the comparison failed because values differ.

Expanded evaluator evidence showing result, score, failed verdict, and evidence for a row

This is the core workflow for prompt iteration. Domain experts can read the row, judge whether the failure is real, and update the prompt or rubric without digging through code.

Inspect one evaluator

Open an evaluator to see its aggregate result across rows. This view shows the average score, pass count, and each row’s result for that criterion.

Criterion detail view showing a 50 percent average score and per-row contains text results

Use this view when one check is noisy. If many rows fail the same evaluator, the evaluator may need a clearer rule, a better expected column, or a different threshold.

Recalculate after changes

Editing a scorecard or changing inputs can make results stale. The panel marks the scorecard as edited and keeps the previous calculation visible until you rerun.

Scorecard panel showing edited state after criteria changed and results need recalculation

Recalculate after you:

Add, remove, or edit an evaluator.
Change a source or expected column.
Update weights or thresholds.
Rerun cells that feed the scorecard.
Import or edit rows in the sheet.

API workflow

The API uses the same model as the UI.

Step	Endpoint
Read the current scorecard	`GET /scorecard`
Configure criteria and aggregation	`PATCH /scorecard`
Queue a calculation	`POST /scorecard/recalculate`
Poll calculation status	`GET /scorecard/calculations/{calculation_id}`
Review row summaries	`GET /scorecard/rows`
Inspect one row	`GET /scorecard/rows/{row_index}`
Cancel active work	`POST /scorecard/cancel`

For scoped work, pass row_indices and step_ids to recalculation or cancel endpoints. Row pagination uses cursor / next_cursor; row breakdown fields such as aggregate_score, aggregate_verdict, and step_results are returned at the top level.

Migrate from legacy `/score`

If your sheet uses legacy scoring, migrate in place and keep current integrations stable.

Run POST /scorecard/migrate-legacy-score with delete_legacy_score: false.
Check converted_count, skipped, and generated steps.
Run POST /scorecard/recalculate.
Move consumers from /score to /scorecard endpoints.
Set delete_legacy_score: true only after migration is complete.

Some legacy score configurations cannot be converted automatically, including custom scoring code, missing score columns, and unsupported column types. Check skipped before deleting legacy score configuration.

Python SDK example

from promptlayer import PromptLayer

client = PromptLayer(api_key="pl_your_key")

table_id = "00000000-0000-0000-0000-000000000000"
sheet_id = "11111111-1111-1111-1111-111111111111"

client.tables.sheets.scorecards.configure(
    table_id,
    sheet_id,
    {
        "name": "Quality Scorecard",
        "aggregation": {
            "method": "weighted_mean",
            "required_step_failure_behavior": "fail",
            "pass_threshold": 0.8,
            "warn_threshold": 0.6,
        },
        "steps": [
            {
                "title": "Output matches expected",
                "primitive_type": "COMPARE",
                "source_column_ids": ["output-column-id", "expected-column-id"],
                "weight": 1,
                "required": False,
                "thresholds": {"pass": 0.8, "warn": 0.6},
                "primitive_config": {
                    "sources": ["output-column-id", "expected-column-id"],
                },
            }
        ],
    },
)

run = client.tables.sheets.scorecards.recalculate(
    table_id,
    sheet_id,
    {"row_indices": [0, 1, 2], "step_ids": ["22222222-2222-2222-2222-222222222222"]}
)
calculation_id = run["calculation_id"]

rows = client.tables.sheets.scorecards.list_rows(
    table_id,
    sheet_id,
    {"calculation_id": calculation_id, "verdict": "fail", "limit": 50},
)

if rows["rows"]:
    row = client.tables.sheets.scorecards.get_row(
        table_id,
        sheet_id,
        rows["rows"][0]["row_index"],
        {"calculation_id": calculation_id},
    )
    print(row["aggregate_score"], row["aggregate_verdict"])

API references

Get scorecard

Read scorecard config and display run state.

Configure scorecard

Create or update criteria and aggregation.

Recalculate scorecard

Queue full or scoped scorecard runs.

List scorecard rows

Review row-level verdicts and drill into failures.

Get Started

Core Concepts

Providers

Guides

AI Tools

Reference

Deprecated

Open the scorecard panel

Add evaluators

Configure an evaluator

Review all rows

Inspect row evidence

Inspect one evaluator

Recalculate after changes

API workflow

Migrate from legacy `/score`

Python SDK example

API references

Get scorecard

Configure scorecard

Recalculate scorecard

List scorecard rows

​Open the scorecard panel

​Add evaluators

​Configure an evaluator

​Review all rows

​Inspect row evidence

​Inspect one evaluator

​Recalculate after changes

​API workflow

​Migrate from legacy /score

​Python SDK example

​API references

Get scorecard

Configure scorecard

Recalculate scorecard

List scorecard rows

Open the scorecard panel

Add evaluators

Configure an evaluator

Review all rows

Inspect row evidence

Inspect one evaluator

Recalculate after changes

API workflow

Migrate from legacy `/score`

Python SDK example

API references