Reference for evaluate(…): dataset, runner, scorers, and every option.
An eval file calls evaluate(...), then you run it:
promptlayer eval run path/to/your.eval.py
Use evaluate for sync runners and aevaluate (Python) for async runners; JavaScript always uses async evaluate. For full harness setups (OpenAI Agents, Claude, LangChain, and more), see the Quickstart.
evaluate(...) options
name is the first positional argument; everything else is keyword-only.
Do not pass a top-level sheet_id / sheetId. To read cases from a dashboard sheet, use dataset.sheet_id / dataset.sheetId instead.
Dataset
Cases feed the runner and scorers. Datasets covers inline cases and dashboard Tables. Each case supports:
Field
Required
Purpose
input
✅
Passed to the runner (column input).
expected
—
Expected output for Contains, Compare, … (column expected).
expected_trace / expectedTrace
—
Trajectory config — which tools should run, not the answer text.
other keys
—
Custom TEXT columns (exact title). No reserved or supporting-column name collisions.
Runner
The runner is any callable that takes an input and returns a final value (not a stream) — no framework allowlist.For Trajectory, tools must appear as Tool: <name> spans on the imported Trace; use a framework helper or traceTool. The runner and tool-span rules live on Runner.
Columns and scorers
Columns are optional processing steps that run before scorers. Declare them with Columns; types match Column Types.
Scorers become scorecard steps on the experiment sheet. Declare them with Scorers. Rollup matches Scorecards; check types match Column Types.
What you get back
On success, the SDK creates (or reuses) a Table, writes a dedicated experiment sheet, configures your scorecard, and returns an EvalResult with a dashboard URL. For failure handling and passing_score, see Results and failures.
Was this page helpful?
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"PromptLayer","colors":{"primary":"#12A7D3","light":"#81D7F7","dark":"#004E7A"},"logo":{"light":"https://mintcdn.com/promptlayer/v0RzaTvbzopITX7U/logo/logo.png?fit=max\u0026auto=format\u0026n=v0RzaTvbzopITX7U\u0026q=85\u0026s=0decf70e3c444d489fa5f1152951bd6c","dark":"https://mintcdn.com/promptlayer/v0RzaTvbzopITX7U/logo/dark.png?fit=max\u0026auto=format\u0026n=v0RzaTvbzopITX7U\u0026q=85\u0026s=1dfca761224a90b3b4bc5e65c8413117","href":"/overview"},"favicon":"https://mintcdn.com/promptlayer/v0RzaTvbzopITX7U/logo/favicon.png?fit=max\u0026auto=format\u0026n=v0RzaTvbzopITX7U\u0026q=85\u0026s=6ddbd8a8de5337ab6293874d59bee473","api":{"mdx":{"server":"https://api.promptlayer.com"}},"background":{"image":"https://mintlify.s3.us-west-1.amazonaws.com/promptlayer/background.png"},"navbar":{"links":[{"href":"/changelog","label":"Changelog"},{"href":"mailto:hello@promptlayer.com","label":"Contact Us"},{"href":"https://dashboard.promptlayer.com","label":"Dashboard"}]},"footer":{"socials":{"x":"https://twitter.com/promptlayer","linkedin":"https://www.linkedin.com/company/promptlayer","github":"https://github.com/MagnivOrg","discord":"https://discord.gg/DBAhQbW39S"}},"seo":{"indexing":"all"},"contextual":{"options":["copy","view","mcp"],"display":"header"},"styling":{"latex":true},"navigation":{"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"promptlayer","hasAudioNarration":false,"actualSubdomain":"promptlayer","customDomains":["docs.promptlayer.com"],"gitSource":{"type":"github","owner":"magnivorg","repo":"prompt-layer-docs","deployBranch":"master","contentDirectory":"","isPrivate":false,"mountPath":"","label":""},"feedback":{"thumbs":true},"entitlements":{"AI_CHAT":{"status":"ENABLED"},"WEB_MCP":{"status":"ENABLED"},"NARRATION":{"status":"ENABLED"}},"buildId":"6aedf67095ef9065731d7686","clientVersion":"0.0.4270","searchFilterCounts":[{"language":"default","tag":"Reference","count":120},{"language":"default","tag":"REST API","count":116},{"language":"default","tag":"Core Concepts","count":77},{"language":"default","tag":"Tables","count":52},{"language":"default","tag":"Deprecated","count":44},{"language":"default","tag":"Observability","count":19},{"language":"default","tag":"Evaluations \u0026 Reports API","count":16},{"language":"default","tag":"Prompt Registry","count":15},{"language":"default","tag":"Tracking","count":14},{"language":"default","tag":"Prompt Templates","count":14}]},"assistantConfig":{},"basePath":"","hasChatPermissions":true,"hasWebMcpPermissions":true,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"site-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"control","enabled":true,"payload":{"type":"string","value":"control"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"wildcard-email-auth","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":false},"customCss":[{"filePath":"style.css","content":"a[href=\"/overview\"]:has(.nav-logo){display:inline-flex!important;align-items:center}a[href=\"/overview\"]:has(.nav-logo)::after{content:\"Docs\";margin-left:0.05rem;color:#262e3e;font-size:1.2rem;font-weight:400;line-height:1;letter-spacing:0}html.dark a[href=\"/overview\"]:has(.nav-logo)::after{color:#ffffff}body:has(.promptlayer-landing) #content-area,body:has(.promptlayer-landing) #content-container,body:has(.promptlayer-landing) main{max-width:none!important}html.dark body:has(.promptlayer-landing){background:radial-gradient(circle at 18% 0%, rgba(245, 207, 99, 0.08), transparent 28rem),\n linear-gradient(180deg, #101116 0%, #0f1117 42rem, #111318 100%)!important}body:has(.promptlayer-landing) #content-container>div.flex.flex-row-reverse.gap-12.box-border.w-full{gap:0!important}.promptlayer-landing{--pl-ink: #12151f;--pl-muted: #596273;--pl-soft: #f7f4eb;--pl-card: rgba(255, 255, 255, 0.82);--pl-card-strong: rgba(255, 255, 255, 0.94);--pl-border: rgba(33, 38, 51, 0.11);--pl-gold: rgb(245, 207, 99);--pl-gold-deep: rgb(67, 44, 16);--pl-shadow: 0 24px 70px -46px rgba(15, 23, 42, 0.42);--pl-shadow-hover: 0 28px 80px -48px rgba(15, 23, 42, 0.52);position:relative;width:min(100%, 72rem);max-width:none;margin:0 auto;padding:0.5rem 1.25rem 4rem;color:var(--pl-ink)}.promptlayer-landing::before{display:none}.promptlayer-hero{position:relative;isolation:isolate;display:grid;grid-template-columns:1fr;gap:1.75rem;align-items:center;margin-bottom:3.75rem;padding:2.4rem 1.75rem 2.85rem;border-radius:1.75rem;overflow:hidden;box-sizing:border-box}.promptlayer-hero::before{position:absolute;inset:0;z-index:-1;background:radial-gradient(\n circle at 14% 12%,\n rgba(245, 207, 99, 0.38),\n transparent 30rem\n ),\n radial-gradient(\n circle at 88% 18%,\n rgba(245, 207, 99, 0.16),\n transparent 24rem\n ),\n linear-gradient(\n 180deg,\n rgba(255, 250, 235, 0.92),\n rgba(255, 255, 255, 0)\n );content:\"\";pointer-events:none}.promptlayer-hero-copy{display:flex;flex-direction:column;align-items:flex-start}.promptlayer-hero-kicker,.promptlayer-section-kicker,.promptlayer-card-pill,.promptlayer-preview-label,.promptlayer-mini-label,.promptlayer-status-pill{font-size:0.75rem;font-weight:700;letter-spacing:0.04em;text-transform:uppercase}.promptlayer-hero-kicker{display:inline-flex;align-items:center;gap:0.5rem;margin-bottom:1.1rem;border:1px solid rgba(67, 44, 16, 0.1);border-radius:999px;background:rgba(255, 255, 255, 0.74);color:#775015;padding:0.46rem 0.78rem;box-shadow:0 1px 2px rgba(15, 23, 42, 0.04);backdrop-filter:blur(14px)}.promptlayer-kicker-dot{width:0.45rem;height:0.45rem;border-radius:999px;background:var(--pl-gold);box-shadow:0 0 0 4px rgba(245, 207, 99, 0.22)}.promptlayer-hero-logo-wrap{width:4.1rem;aspect-ratio:173 / 150;margin-bottom:1.05rem}.promptlayer-hero-logo{display:block;width:100%;height:100%;object-fit:contain;filter:grayscale(1) brightness(0.1);opacity:0.84;pointer-events:none;user-select:none}.promptlayer-hero-title{max-width:58rem;margin:0 0 1rem;color:var(--pl-ink);font-size:clamp(2.35rem, 5.4vw, 4.15rem);font-weight:760;letter-spacing:-0.064em;line-height:0.965}.promptlayer-hero-description{max-width:45rem;margin:0 0 1.45rem;color:var(--pl-muted);font-size:1.075rem;line-height:1.68}.promptlayer-hero-actions{display:flex;flex-wrap:wrap;gap:0.75rem;align-items:center;margin-bottom:1rem}.promptlayer-primary-button,.promptlayer-secondary-button{display:inline-flex;align-items:center;justify-content:center;min-height:2.85rem;border-radius:999px;font-size:0.94rem;font-weight:700;line-height:1;text-decoration:none;transition:transform 160ms ease,\n border-color 160ms ease,\n background-color 160ms ease,\n box-shadow 160ms ease}.promptlayer-primary-button{gap:0.5rem;padding:0 1.15rem 0 1.25rem;background:var(--pl-gold);color:var(--pl-gold-deep);box-shadow:0 18px 30px -22px rgba(67, 44, 16, 0.7)}.promptlayer-secondary-button{border:1px solid var(--pl-border);background:rgba(255, 255, 255, 0.78);color:var(--pl-ink);padding:0 1.1rem}.promptlayer-primary-button:hover,.promptlayer-secondary-button:hover{transform:translateY(-1px)}.promptlayer-primary-button:hover{background:rgb(239, 198, 82);box-shadow:0 22px 34px -22px rgba(67, 44, 16, 0.72)}.promptlayer-secondary-button:hover{border-color:rgba(33, 38, 51, 0.2);background:rgba(255, 255, 255, 0.94)}.promptlayer-hero-proof{display:flex;flex-wrap:wrap;gap:0.5rem;align-items:center}.promptlayer-hero-proof-label{color:#8c6424;font-size:0.78rem;font-weight:750;letter-spacing:0.02em}.promptlayer-hero-proof a{border:1px solid rgba(33, 38, 51, 0.09);border-radius:999px;background:rgba(255, 255, 255, 0.52);color:#657084;font-size:0.82rem;line-height:1;padding:0.5rem 0.7rem;text-decoration:none;transition:transform 160ms ease,\n border-color 160ms ease,\n background-color 160ms ease,\n color 160ms ease}.promptlayer-hero-proof a:hover{transform:translateY(-1px);border-color:rgba(33, 38, 51, 0.17);background:rgba(255, 255, 255, 0.82);color:var(--pl-ink)}.promptlayer-hero-panel,.promptlayer-path-card,.promptlayer-bento-card,.promptlayer-surface-card,.promptlayer-resource-card{position:relative;border:1px solid var(--pl-border);background:var(--pl-card);color:inherit;text-decoration:none;box-shadow:var(--pl-shadow);backdrop-filter:blur(18px)}.promptlayer-hero-panel{overflow:hidden;border-radius:2rem;padding:0.65rem;background:linear-gradient(\n 145deg,\n rgba(255, 255, 255, 0.94),\n rgba(255, 248, 230, 0.72)\n ),\n radial-gradient(\n circle at 20% 20%,\n rgba(245, 207, 99, 0.3),\n transparent 18rem\n )}.promptlayer-hero-panel::before{position:absolute;inset:0;background-image:linear-gradient(rgba(33, 38, 51, 0.045) 1px, transparent 1px),\n linear-gradient(90deg, rgba(33, 38, 51, 0.045) 1px, transparent 1px);background-size:28px 28px;content:\"\";mask-image:linear-gradient(180deg, rgba(0, 0, 0, 0.7), transparent 78%);pointer-events:none}.promptlayer-window-bar{position:relative;z-index:1;display:flex;gap:0.38rem;padding:0.85rem 0.9rem 0.15rem}.promptlayer-window-bar span{width:0.58rem;height:0.58rem;border-radius:999px;background:rgba(33, 38, 51, 0.18)}.promptlayer-product-preview{position:relative;z-index:1;margin-top:0.3rem;border:1px solid rgba(33, 38, 51, 0.08);border-radius:1.5rem;background:rgba(255, 255, 255, 0.78);padding:0.95rem;box-shadow:inset 0 1px 0 rgba(255, 255, 255, 0.8)}.promptlayer-preview-header{display:flex;justify-content:space-between;gap:1rem;align-items:flex-start;margin-bottom:1rem}.promptlayer-preview-header h2{margin:0.25rem 0 0;color:var(--pl-ink);font-size:clamp(1.25rem, 3vw, 1.75rem);font-weight:740;letter-spacing:-0.035em;line-height:1.05}.promptlayer-preview-label,.promptlayer-mini-label{color:#8c6424}.promptlayer-status-pill{border:1px solid rgba(22, 101, 52, 0.12);border-radius:999px;background:rgba(220, 252, 231, 0.78);color:#166534;padding:0.42rem 0.58rem}.promptlayer-preview-grid{display:grid;grid-template-columns:repeat(2, minmax(0, 1fr));gap:0.7rem}.promptlayer-preview-tile{overflow:hidden;min-height:6.75rem;border:1px solid rgba(33, 38, 51, 0.08);border-radius:1.15rem;background:rgba(255, 255, 255, 0.72);padding:0.9rem}.promptlayer-preview-tile strong{display:block;margin-top:0.35rem;color:var(--pl-ink);font-size:1.35rem;letter-spacing:-0.04em;line-height:1.05}.promptlayer-preview-tile p{margin:0.55rem 0 0;color:var(--pl-muted);font-size:0.85rem;line-height:1.45}.promptlayer-preview-tile-main,.promptlayer-preview-tile-wide{grid-column:span 2}.promptlayer-version-stack{display:grid;gap:0.45rem;margin-top:1rem}.promptlayer-version-stack span{display:block;height:0.56rem;border-radius:999px;background:rgba(33, 38, 51, 0.1)}.promptlayer-version-stack span:nth-child(1){width:90%;background:rgba(245, 207, 99, 0.65)}.promptlayer-version-stack span:nth-child(2){width:72%}.promptlayer-version-stack span:nth-child(3){width:48%}.promptlayer-meter{overflow:hidden;height:0.55rem;margin-top:1rem;border-radius:999px;background:rgba(33, 38, 51, 0.09)}.promptlayer-meter span{display:block;width:94%;height:100%;border-radius:inherit;background:linear-gradient(90deg, rgb(245, 207, 99), rgb(74, 222, 128))}.promptlayer-sparkline{display:flex;align-items:flex-end;gap:0.24rem;height:2rem;margin-top:0.85rem}.promptlayer-sparkline span{flex:1;border-radius:999px 999px 0.35rem 0.35rem;background:rgba(245, 207, 99, 0.72)}.promptlayer-sparkline span:nth-child(1){height:44%}.promptlayer-sparkline span:nth-child(2){height:72%}.promptlayer-sparkline span:nth-child(3){height:56%}.promptlayer-sparkline span:nth-child(4){height:86%}.promptlayer-sparkline span:nth-child(5){height:64%}.promptlayer-trace-list{display:grid;gap:0.5rem;margin-top:0.65rem}.promptlayer-trace-list span{position:relative;display:flex;align-items:center;gap:0.5rem;color:var(--pl-muted);font-size:0.88rem}.promptlayer-trace-list span::before{width:0.45rem;height:0.45rem;border-radius:999px;background:var(--pl-gold);content:\"\"}.promptlayer-section{margin-bottom:3.75rem}.promptlayer-section-heading{max-width:44rem;margin-bottom:1.35rem}.promptlayer-section-heading-split{display:grid;max-width:none;gap:1rem;align-items:end}.promptlayer-section-kicker{margin:0 0 0.5rem;color:#8c6424}.promptlayer-section-title{margin:0;color:var(--pl-ink);font-size:clamp(1.7rem, 4vw, 2.55rem);font-weight:745;letter-spacing:-0.045em;line-height:1.03}.promptlayer-section-description{max-width:34rem;margin:0;color:var(--pl-muted);font-size:0.98rem;line-height:1.7}.promptlayer-path-grid,.promptlayer-resource-grid{display:grid;grid-template-columns:1fr;gap:0.9rem}.promptlayer-path-card,.promptlayer-bento-card,.promptlayer-surface-card{display:flex;flex-direction:column;overflow:visible;border-radius:1.45rem;padding:1.25rem}.promptlayer-path-card{min-height:14rem;justify-content:flex-end;background:linear-gradient(\n 180deg,\n var(--pl-card-strong),\n rgba(255, 250, 238, 0.72)\n )}.promptlayer-path-card-primary{background:radial-gradient(\n circle at 8% 8%,\n rgba(245, 207, 99, 0.42),\n transparent 18rem\n ),\n linear-gradient(\n 180deg,\n rgba(255, 255, 255, 0.94),\n rgba(255, 246, 220, 0.78)\n )}.promptlayer-card-pill{display:inline-flex;width:fit-content;border-radius:999px;background:rgba(245, 207, 99, 0.18);color:#7b5319;padding:0.42rem 0.58rem}.promptlayer-path-card h3,.promptlayer-bento-card h3,.promptlayer-surface-card h3{margin:0.8rem 0 0.55rem;color:var(--pl-ink);font-size:1.2rem;font-weight:720;letter-spacing:-0.03em;line-height:1.12}.promptlayer-path-card p,.promptlayer-bento-card p,.promptlayer-surface-card p{margin:0;color:var(--pl-muted);font-size:0.93rem;line-height:1.6}.promptlayer-bento-grid{display:grid;grid-template-columns:1fr;gap:0.9rem}.promptlayer-bento-card{min-height:0;background:linear-gradient(\n 180deg,\n var(--pl-card-strong),\n rgba(255, 255, 255, 0.78)\n ),\n radial-gradient(\n circle at 18% 10%,\n rgba(245, 207, 99, 0.16),\n transparent 16rem\n )}.promptlayer-bento-card-large{gap:1.25rem}.promptlayer-bento-card-wide{gap:1rem}.promptlayer-card-icon{display:flex;width:4.25rem;height:4.25rem;align-items:center;justify-content:center;margin-bottom:0.75rem;border-radius:1.45rem;background:rgba(245, 207, 99, 0.2);box-shadow:inset 0 0 0 1px rgba(67, 44, 16, 0.04),\n 0 1px 2px rgba(15, 23, 42, 0.04)}.promptlayer-card-icon img{display:block;width:auto;max-width:2.65rem;height:2.65rem;object-fit:contain}.promptlayer-surface-grid{display:grid;grid-template-columns:1fr;gap:0.72rem}.promptlayer-surface-card{position:relative;min-height:13rem;gap:1rem;justify-content:space-between;border-radius:1.25rem;background:linear-gradient(\n 180deg,\n rgba(255, 255, 255, 0.96),\n rgba(252, 250, 244, 0.86)\n )}.promptlayer-surface-card::before{position:absolute;inset:0;border-radius:inherit;pointer-events:none;background:linear-gradient(\n 180deg,\n rgba(255, 255, 255, 0.55),\n transparent 38%\n );content:\"\"}.promptlayer-surface-index{color:rgba(18, 21, 31, 0.34);font-size:0.78rem;font-weight:680;letter-spacing:-0.01em}.promptlayer-surface-label{display:block;color:#8c6424;font-size:0.78rem;font-weight:700;letter-spacing:0.01em}.promptlayer-surface-meta{display:flex;flex-wrap:wrap;gap:0.42rem;padding-top:0.2rem}.promptlayer-surface-meta span{border:1px solid rgba(33, 38, 51, 0.08);border-radius:999px;background:rgba(255, 255, 255, 0.62);color:rgba(89, 98, 115, 0.86);font-size:0.73rem;font-weight:640;line-height:1;padding:0.38rem 0.5rem}.promptlayer-surface-card:hover{background:linear-gradient(\n 180deg,\n rgba(255, 255, 255, 0.98),\n rgba(255, 252, 245, 0.92)\n )}.promptlayer-card-icon-blue,.promptlayer-card-icon-green,.promptlayer-card-icon-gold,.promptlayer-card-icon-purple{background:rgba(245, 207, 99, 0.2)}.promptlayer-card-visual{margin-top:auto}.promptlayer-release-visual{display:grid;grid-template-columns:repeat(3, minmax(0, 1fr));gap:0.55rem}.promptlayer-release-visual span{border:1px solid rgba(33, 38, 51, 0.08);border-radius:0.9rem;background:rgba(255, 255, 255, 0.68);color:var(--pl-muted);font-size:0.76rem;font-weight:700;padding:0.72rem 0.5rem;text-align:center}.promptlayer-release-visual span:nth-child(3){background:rgba(245, 207, 99, 0.36);color:#6f4811}.promptlayer-table-visual{display:grid;grid-template-columns:repeat(3, 1fr);gap:0.45rem;margin-top:auto;padding-top:1.35rem}.promptlayer-table-visual span{min-height:2rem;border:1px solid rgba(33, 38, 51, 0.08);border-radius:0.7rem;background:rgba(255, 255, 255, 0.68)}.promptlayer-table-visual span:nth-child(2),.promptlayer-table-visual span:nth-child(6),.promptlayer-table-visual span:nth-child(7){background:rgba(245, 207, 99, 0.28)}.promptlayer-resource-section{margin-bottom:0}.promptlayer-resource-card{position:relative;display:flex;min-height:6.75rem;flex-direction:column;justify-content:flex-start;gap:0.52rem;overflow:hidden;border-radius:1.15rem;padding:1.05rem 2.9rem 1.05rem 3.85rem;background:linear-gradient(\n 180deg,\n rgba(255, 255, 255, 0.84),\n rgba(255, 252, 244, 0.72)\n ),\n radial-gradient(\n circle at 0 0,\n rgba(245, 207, 99, 0.16),\n transparent 7rem\n )}.promptlayer-resource-card::before{position:absolute;top:1.05rem;left:1.05rem;display:grid;width:1.85rem;height:2.12rem;place-items:center;border:1px solid rgba(67, 44, 16, 0.1);border-radius:0.48rem;background:linear-gradient(\n 135deg,\n transparent 0 72%,\n rgba(245, 207, 99, 0.42) 72% 100%\n ),\n rgba(255, 248, 225, 0.9);box-shadow:0 1px 2px rgba(67, 44, 16, 0.05),\n inset 0 -0.55rem 0 rgba(245, 207, 99, 0.12);color:#7b5319;content:\"↗\";font-size:0.86rem;font-weight:800;line-height:1}.promptlayer-resource-card::after{position:absolute;top:1.05rem;right:1.05rem;display:grid;width:1.75rem;height:1.75rem;place-items:center;border:1px solid rgba(33, 38, 51, 0.08);border-radius:999px;background:rgba(255, 255, 255, 0.62);color:#94a3b8;font-size:0.9rem;line-height:1;content:\"→\";opacity:0.82;transition:transform 180ms ease,\n opacity 180ms ease,\n color 180ms ease,\n background-color 180ms ease,\n border-color 180ms ease}.promptlayer-resource-card:hover::after{transform:translateX(2px);border-color:rgba(67, 44, 16, 0.14);background:rgba(245, 207, 99, 0.16);color:#7b5319;opacity:1}.promptlayer-resource-card span{color:#8c6424;font-size:0.72rem;font-weight:850;letter-spacing:0.045em;line-height:1;text-transform:uppercase}.promptlayer-resource-card strong{max-width:14rem;color:var(--pl-ink);font-size:0.98rem;font-weight:720;letter-spacing:-0.025em;line-height:1.28}.promptlayer-card-interactive{transition:transform 180ms ease,\n border-color 180ms ease,\n background-color 180ms ease,\n box-shadow 180ms ease}.promptlayer-card-interactive:hover{transform:translateY(-2px);border-color:rgba(33, 38, 51, 0.18);background-color:rgba(255, 255, 255, 0.96);box-shadow:var(--pl-shadow-hover)}.promptlayer-card-interactive:focus-visible,.promptlayer-primary-button:focus-visible,.promptlayer-secondary-button:focus-visible{outline:none;border-color:rgba(245, 207, 99, 0.9);box-shadow:0 0 0 3px rgba(245, 207, 99, 0.32),\n var(--pl-shadow-hover)}.promptlayer-card-arrow{position:absolute;top:1rem;right:1rem;display:grid;width:2rem;height:2rem;place-items:center;border:1px solid rgba(33, 38, 51, 0.08);border-radius:999px;background:rgba(255, 255, 255, 0.58);color:#64748b;font-size:1rem;line-height:1;opacity:0.82;transition:transform 180ms ease,\n opacity 180ms ease}.promptlayer-card-interactive:hover .promptlayer-card-arrow{transform:translate(2px, -2px);opacity:1}@media (min-width: 640px){.promptlayer-path-grid,.promptlayer-surface-grid,.promptlayer-resource-grid{grid-template-columns:repeat(2, minmax(0, 1fr))}}@media (min-width: 1024px){.promptlayer-hero{grid-template-columns:minmax(0, 1.04fr) minmax(21rem, 0.72fr);gap:2.25rem;padding-top:3rem;padding-bottom:3.2rem}.promptlayer-hero-title{max-width:44rem;font-size:clamp(3.05rem, 4.7vw, 4.3rem)}.promptlayer-hero-description{max-width:40rem}.promptlayer-section-heading-split{grid-template-columns:minmax(0, 1fr) minmax(20rem, 0.72fr)}.promptlayer-path-grid{grid-template-columns:1.1fr repeat(2, minmax(0, 0.95fr))}.promptlayer-surface-grid{grid-template-columns:repeat(4, minmax(0, 1fr))}.promptlayer-bento-grid{grid-template-columns:repeat(3, minmax(0, 1fr));grid-auto-rows:auto;align-items:stretch}.promptlayer-bento-card-large{grid-column:span 2}.promptlayer-bento-card-tall{grid-column:span 1;grid-row:auto}.promptlayer-bento-card-wide{grid-column:span 2;grid-row:auto;flex-direction:column;align-items:flex-start}.promptlayer-resource-grid{grid-template-columns:repeat(3, minmax(0, 1fr))}}@media (min-width: 1180px){.promptlayer-landing{padding-top:0.75rem}.promptlayer-hero{gap:2rem}.promptlayer-bento-grid{grid-template-columns:repeat(3, minmax(0, 1fr))}.promptlayer-bento-card-large{grid-row:auto}.promptlayer-resource-grid{grid-template-columns:repeat(4, minmax(0, 1fr))}}@media (max-width: 1023px){body:has(.promptlayer-landing) #navbar .relative>button.flex.items-center.h-14.py-4.px-5.lg\\:hidden.focus\\:outline-0.w-full.text-left{display:none}body:has(.promptlayer-landing) #content-container>div.flex.flex-row-reverse.gap-12.box-border.w-full{padding-top:4rem}}@media (max-width: 520px){.promptlayer-landing{padding-inline:1rem}.promptlayer-hero-actions{width:100%}.promptlayer-primary-button,.promptlayer-secondary-button{width:100%}.promptlayer-preview-grid{grid-template-columns:1fr}.promptlayer-preview-tile-main,.promptlayer-preview-tile-wide{grid-column:auto}.promptlayer-release-visual{grid-template-columns:1fr}}html.dark .promptlayer-landing{--pl-ink: #f7f8fb;--pl-muted: #aeb6c5;--pl-card: rgba(27, 29, 36, 0.9);--pl-card-strong: rgba(33, 35, 43, 0.96);--pl-border: rgba(226, 232, 240, 0.12);--pl-shadow: 0 26px 76px -44px rgba(0, 0, 0, 0.9);--pl-shadow-hover: 0 30px 84px -44px rgba(0, 0, 0, 0.95)}html.dark .promptlayer-hero::before{background:radial-gradient(\n circle at 14% 12%,\n rgba(245, 207, 99, 0.16),\n transparent 30rem\n ),\n radial-gradient(\n circle at 88% 18%,\n rgba(148, 163, 184, 0.08),\n transparent 24rem\n ),\n linear-gradient(180deg, rgba(18, 19, 25, 0.92), rgba(15, 17, 22, 0.1))}html.dark .promptlayer-hero-logo{filter:grayscale(0.2) brightness(1.08);opacity:0.94}html.dark .promptlayer-hero-kicker,html.dark .promptlayer-secondary-button,html.dark .promptlayer-hero-proof a,html.dark .promptlayer-product-preview,html.dark .promptlayer-preview-tile,html.dark .promptlayer-release-visual span,html.dark .promptlayer-table-visual span,html.dark .promptlayer-card-arrow{background:rgba(35, 37, 44, 0.72)}html.dark .promptlayer-hero-kicker,html.dark .promptlayer-section-kicker,html.dark .promptlayer-card-pill,html.dark .promptlayer-preview-label,html.dark .promptlayer-mini-label,html.dark .promptlayer-surface-label,html.dark .promptlayer-resource-card span{color:#e9c96f}html.dark .promptlayer-hero-panel{border-color:rgba(226, 232, 240, 0.12);background:linear-gradient(145deg, rgba(36, 38, 47, 0.96), rgba(19, 21, 27, 0.9)),\n radial-gradient(\n circle at 20% 20%,\n rgba(245, 207, 99, 0.12),\n transparent 18rem\n );box-shadow:0 28px 74px -42px rgba(0, 0, 0, 0.95),\n inset 0 1px 0 rgba(255, 255, 255, 0.04)}html.dark .promptlayer-hero-panel::before{background-image:linear-gradient(rgba(226, 232, 240, 0.045) 1px, transparent 1px),\n linear-gradient(90deg, rgba(226, 232, 240, 0.045) 1px, transparent 1px)}html.dark .promptlayer-window-bar span{background:rgba(226, 232, 240, 0.18)}html.dark .promptlayer-product-preview{border-color:rgba(226, 232, 240, 0.1);box-shadow:inset 0 1px 0 rgba(255, 255, 255, 0.045),\n 0 18px 60px -44px rgba(0, 0, 0, 0.9)}html.dark .promptlayer-preview-tile{border-color:rgba(226, 232, 240, 0.08);background:rgba(28, 31, 39, 0.78)}html.dark .promptlayer-version-stack span,html.dark .promptlayer-meter{background:rgba(226, 232, 240, 0.12)}html.dark .promptlayer-sparkline span{background:rgba(245, 207, 99, 0.66)}html.dark .promptlayer-trace-list span{color:#c5ccda}html.dark .promptlayer-path-card,html.dark .promptlayer-bento-card,html.dark .promptlayer-resource-card{background:linear-gradient(180deg, var(--pl-card-strong), rgba(25, 27, 33, 0.82)),\n radial-gradient(\n circle at 18% 10%,\n rgba(245, 207, 99, 0.08),\n transparent 16rem\n )}html.dark .promptlayer-surface-card{background:linear-gradient(180deg, rgba(31, 32, 37, 0.96), rgba(22, 23, 27, 0.9)),\n radial-gradient(\n circle at 50% 0%,\n rgba(245, 207, 99, 0.05),\n transparent 15rem\n )}html.dark .promptlayer-surface-card::before{background:linear-gradient(\n 180deg,\n rgba(255, 255, 255, 0.045),\n transparent 42%\n )}html.dark .promptlayer-surface-index{color:rgba(247, 248, 251, 0.28)}html.dark .promptlayer-surface-meta span{border-color:rgba(226, 232, 240, 0.08);background:rgba(255, 255, 255, 0.035);color:rgba(199, 206, 219, 0.78)}html.dark .promptlayer-surface-card:hover{background:linear-gradient(180deg, rgba(35, 36, 42, 0.98), rgba(25, 26, 31, 0.94)),\n radial-gradient(\n circle at 50% 0%,\n rgba(245, 207, 99, 0.06),\n transparent 15rem\n )}html.dark .promptlayer-path-card-primary{background:radial-gradient(\n circle at 8% 8%,\n rgba(245, 207, 99, 0.17),\n transparent 18rem\n ),\n linear-gradient(180deg, rgba(42, 43, 49, 0.92), rgba(35, 30, 20, 0.82))}html.dark .promptlayer-card-interactive:hover,html.dark .promptlayer-secondary-button:hover{border-color:rgba(226, 232, 240, 0.2);background-color:rgba(42, 44, 52, 0.96)}html.dark .promptlayer-status-pill{border-color:rgba(134, 239, 172, 0.18);background:rgba(22, 101, 52, 0.28);color:#bbf7d0}html.dark .promptlayer-resource-card::before{border-color:rgba(245, 207, 99, 0.14);color:#f7d77a;background:linear-gradient(\n 135deg,\n transparent 0 72%,\n rgba(245, 207, 99, 0.22) 72% 100%\n ),\n rgba(245, 207, 99, 0.08);box-shadow:0 1px 2px rgba(0, 0, 0, 0.18),\n inset 0 -0.55rem 0 rgba(245, 207, 99, 0.07)}html.dark .promptlayer-resource-card::after{border-color:rgba(226, 232, 240, 0.1);background:rgba(35, 37, 44, 0.72);color:#a1a1aa}html.dark .promptlayer-resource-card:hover::after{border-color:rgba(245, 207, 99, 0.2);background:rgba(245, 207, 99, 0.12);color:#f7d77a}html.dark .promptlayer-release-visual span:nth-child(3),html.dark .promptlayer-card-pill{background:rgba(245, 207, 99, 0.14)}html.dark .promptlayer-card-icon img{opacity:0.96}@media (prefers-reduced-motion: reduce){.promptlayer-primary-button,.promptlayer-secondary-button,.promptlayer-hero-proof a,.promptlayer-card-interactive,.promptlayer-card-arrow{transition:none}.promptlayer-primary-button:hover,.promptlayer-secondary-button:hover,.promptlayer-hero-proof a:hover,.promptlayer-resource-card:hover::after,.promptlayer-card-interactive:hover,.promptlayer-card-interactive:hover .promptlayer-card-arrow{transform:none}}"}]}
{"page":{"pagePath":"sdks/evals/building-an-eval","slug":"/sdks/evals/building-an-eval","pageMetadata":{"title":"Building an eval","description":"Reference for evaluate(...): dataset, runner, scorers, and every option.","sidebarTitle":"Building an eval","icon":"pen-to-square","href":"/sdks/evals/building-an-eval"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Reference for evaluate(…): dataset, runner, scorers, and every option.\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,li:`li`,p:`p`,pre:`pre`,span:`span`,strong:`strong`,tbody:`tbody`,td:`td`,th:`th`,thead:`thead`,tr:`tr`,ul:`ul`,..._provideComponents(),...props.components},{CodeBlock,Heading,Table,Warning}=_components;if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!Table)_missingMdxReference(`Table`,true);if(!Warning)_missingMdxReference(`Warning`,true);return _jsxs(_Fragment,{children:[_jsxs(_components.p,{children:[`An eval file calls `,_jsx(_components.code,{children:`evaluate(...)`}),`, then you run it:`]}),`\n`,_jsx(CodeBlock,{filename:``,numberOfLines:`1`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`1`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`promptlayer`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` eval`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` run`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` path/to/your.eval.py`})]}),`\n`]})})}),`\n`,_jsxs(_components.p,{children:[`Use `,_jsx(_components.code,{children:`evaluate`}),` for sync runners and `,_jsx(_components.code,{children:`aevaluate`}),` (Python) for async runners; JavaScript always uses async `,_jsx(_components.code,{children:`evaluate`}),`. For full harness setups (OpenAI Agents, Claude, LangChain, and more), see the `,_jsx(_components.a,{href:`/sdks/evals/quickstart`,children:`Quickstart`}),`.`]}),`\n`,_jsxs(Heading,{level:`2`,id:`evaluate-options`,children:[_jsx(_components.code,{children:`evaluate(...)`}),` options`]}),`\n`,_jsxs(_components.p,{children:[_jsx(_components.code,{children:`name`}),` is the first positional argument; everything else is keyword-only.`]}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Option (Python / JS)`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Default`}),_jsx(_components.th,{children:`Purpose`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`name`})}),_jsx(_components.td,{children:`✅`}),_jsx(_components.td,{children:`—`}),_jsxs(_components.td,{children:[`Experiment name; also the `,_jsx(_components.code,{children:`Eval: \u003cname>`}),` span.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`dataset`})}),_jsx(_components.td,{children:`✅`}),_jsx(_components.td,{children:`—`}),_jsxs(_components.td,{children:[`Cases fed to the runner and scorers. See `,_jsx(_components.a,{href:`#dataset`,children:`Dataset`}),`.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`runner`})}),_jsx(_components.td,{children:`✅`}),_jsx(_components.td,{children:`—`}),_jsxs(_components.td,{children:[`The agent under test. See `,_jsx(_components.a,{href:`#runner`,children:`Runner`}),`.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`scorers`})}),_jsx(_components.td,{children:`✅`}),_jsx(_components.td,{children:`—`}),_jsxs(_components.td,{children:[`Scorecard checks. See `,_jsx(_components.a,{href:`#columns-and-scorers`,children:`Columns and scorers`}),`.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`columns`})}),_jsx(_components.td,{children:`—`}),_jsx(_components.td,{children:_jsx(_components.code,{children:`None`})}),_jsxs(_components.td,{children:[`Processing steps applied before scorers. See `,_jsx(_components.a,{href:`#columns-and-scorers`,children:`Columns and scorers`}),`.`]})]}),_jsxs(_components.tr,{children:[_jsxs(_components.td,{children:[_jsx(_components.code,{children:`passing_score`}),` / `,_jsx(_components.code,{children:`passingScore`})]}),_jsx(_components.td,{children:`—`}),_jsx(_components.td,{children:_jsx(_components.code,{children:`None`})}),_jsxs(_components.td,{children:[`Fail the run below this bar. See `,_jsx(_components.a,{href:`/sdks/evals/results-and-failures`,children:`Results and failures`}),`.`]})]}),_jsxs(_components.tr,{children:[_jsxs(_components.td,{children:[_jsx(_components.code,{children:`max_concurrency`}),` / `,_jsx(_components.code,{children:`maxConcurrency`})]}),_jsx(_components.td,{children:`—`}),_jsx(_components.td,{\"data-numeric\":`true`,children:_jsx(_components.code,{children:`1`})}),_jsx(_components.td,{children:`Run cases in parallel.`})]}),_jsxs(_components.tr,{children:[_jsxs(_components.td,{children:[_jsx(_components.code,{children:`include_failure_examples`}),` / `,_jsx(_components.code,{children:`includeFailureExamples`})]}),_jsx(_components.td,{children:`—`}),_jsx(_components.td,{children:_jsx(_components.code,{children:`false`})}),_jsx(_components.td,{children:`Print failing rows in the CLI output.`})]}),_jsxs(_components.tr,{children:[_jsxs(_components.td,{children:[_jsx(_components.code,{children:`table_id`}),` / `,_jsx(_components.code,{children:`tableId`})]}),_jsx(_components.td,{children:`—`}),_jsx(_components.td,{children:_jsx(_components.code,{children:`None`})}),_jsx(_components.td,{children:`Reuse an existing Table instead of creating one.`})]}),_jsxs(_components.tr,{children:[_jsxs(_components.td,{children:[_jsx(_components.code,{children:`folder_id`}),` / `,_jsx(_components.code,{children:`folderId`})]}),_jsx(_components.td,{children:`—`}),_jsx(_components.td,{children:_jsx(_components.code,{children:`None`})}),_jsx(_components.td,{children:`Folder for a newly created Table.`})]}),_jsxs(_components.tr,{children:[_jsxs(_components.td,{children:[_jsx(_components.code,{children:`experiment_name`}),` / `,_jsx(_components.code,{children:`experimentName`})]}),_jsx(_components.td,{children:`—`}),_jsx(_components.td,{children:`auto`}),_jsx(_components.td,{children:`Name of the experiment sheet.`})]}),_jsxs(_components.tr,{children:[_jsxs(_components.td,{children:[_jsx(_components.code,{children:`api_key`}),` / `,_jsx(_components.code,{children:`apiKey`})]}),_jsx(_components.td,{children:`—`}),_jsx(_components.td,{children:`env`}),_jsxs(_components.td,{children:[`Auth; prefer `,_jsx(_components.code,{children:`PROMPTLAYER_API_KEY`}),` for the CLI.`]})]}),_jsxs(_components.tr,{children:[_jsxs(_components.td,{children:[_jsx(_components.code,{children:`base_url`}),` / `,_jsx(_components.code,{children:`baseURL`})]}),_jsx(_components.td,{children:`—`}),_jsx(_components.td,{children:`env`}),_jsx(_components.td,{children:`Override the API base URL.`})]})]})]}),`\n`,_jsx(Warning,{children:_jsxs(_components.p,{children:[`Do `,_jsx(_components.strong,{children:`not`}),` pass a top-level `,_jsx(_components.code,{children:`sheet_id`}),` / `,_jsx(_components.code,{children:`sheetId`}),`. To read cases from a dashboard sheet, use `,_jsx(_components.code,{children:`dataset.sheet_id`}),` / `,_jsx(_components.code,{children:`dataset.sheetId`}),` instead.`]})}),`\n`,_jsx(Heading,{level:`2`,id:`dataset`,children:`Dataset`}),`\n`,_jsxs(_components.p,{children:[`Cases feed the runner and scorers. `,_jsx(_components.a,{href:`/sdks/evals/datasets`,children:`Datasets`}),` covers inline cases and dashboard Tables. Each case supports:`]}),`\n`,_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Field`}),_jsx(_components.th,{children:`Required`}),_jsx(_components.th,{children:`Purpose`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`input`})}),_jsx(_components.td,{children:`✅`}),_jsxs(_components.td,{children:[`Passed to the `,_jsx(_components.code,{children:`runner`}),` (column `,_jsx(_components.code,{children:`input`}),`).`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:_jsx(_components.code,{children:`expected`})}),_jsx(_components.td,{children:`—`}),_jsxs(_components.td,{children:[`Expected `,_jsx(_components.strong,{children:`output`}),` for Contains, Compare, … (column `,_jsx(_components.code,{children:`expected`}),`).`]})]}),_jsxs(_components.tr,{children:[_jsxs(_components.td,{children:[_jsx(_components.code,{children:`expected_trace`}),` / `,_jsx(_components.code,{children:`expectedTrace`})]}),_jsx(_components.td,{children:`—`}),_jsxs(_components.td,{children:[_jsx(_components.a,{href:`/sdks/evals/scorers/overview#trajectory`,children:`Trajectory`}),` config — which tools should run, not the answer text.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`other keys`}),_jsx(_components.td,{children:`—`}),_jsx(_components.td,{children:`Custom TEXT columns (exact title). No reserved or supporting-column name collisions.`})]})]})]}),`\n`,_jsx(Heading,{level:`2`,id:`runner`,children:`Runner`}),`\n`,_jsxs(_components.p,{children:[`The `,_jsx(_components.code,{children:`runner`}),` is any callable that takes an `,_jsx(_components.code,{children:`input`}),` and returns a final value (not a stream) — no framework allowlist.`]}),`\n`,_jsxs(_components.p,{children:[`For Trajectory, tools must appear as `,_jsx(_components.code,{children:`Tool: \u003cname>`}),` spans on the imported Trace; use a framework helper or `,_jsx(_components.a,{href:`/features/observability/traces/manual-tracing#trace-tools`,children:_jsx(_components.code,{children:`traceTool`})}),`. The runner and tool-span rules live on `,_jsx(_components.a,{href:`/sdks/evals/agent-tracing`,children:`Runner`}),`.`]}),`\n`,_jsx(Heading,{level:`2`,id:`columns-and-scorers`,children:`Columns and scorers`}),`\n`,_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Columns`}),` are optional processing steps that run before scorers. Declare them with `,_jsx(_components.a,{href:`/sdks/evals/columns/overview`,children:`Columns`}),`; types match `,_jsx(_components.a,{href:`/features/tables/column-types`,children:`Column Types`}),`.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Scorers`}),` become scorecard steps on the experiment sheet. Declare them with `,_jsx(_components.a,{href:`/sdks/evals/scorers/overview`,children:`Scorers`}),`. Rollup matches `,_jsx(_components.a,{href:`/features/tables/scorecards`,children:`Scorecards`}),`; check types match `,_jsx(_components.a,{href:`/features/tables/column-types`,children:`Column Types`}),`.`]}),`\n`]}),`\n`,_jsx(Heading,{level:`2`,id:`what-you-get-back`,children:`What you get back`}),`\n`,_jsxs(_components.p,{children:[`On success, the SDK creates (or reuses) a Table, writes a dedicated experiment sheet, configures your scorecard, and returns an `,_jsx(_components.code,{children:`EvalResult`}),` with a dashboard URL. For failure handling and `,_jsx(_components.code,{children:`passing_score`}),`, see `,_jsx(_components.a,{href:`/sdks/evals/results-and-failures`,children:`Results and failures`}),`.`]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Building an eval","description":"Reference for evaluate(...): dataset, runner, scorers, and every option.","sidebarTitle":"Building an eval","icon":"pen-to-square","href":"/sdks/evals/building-an-eval"}}},"mdxSourceWithNoJs":null},"mdxExtracts":{"tableOfContents":[{"title":"evaluate(...) options","slug":"evaluate-options","depth":2,"children":[]},{"title":"Dataset","slug":"dataset","depth":2,"children":[]},{"title":"Runner","slug":"runner","depth":2,"children":[]},{"title":"Columns and scorers","slug":"columns-and-scorers","depth":2,"children":[]},{"title":"What you get back","slug":"what-you-get-back","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":[],"subdomain":"promptlayer","siteName":"PromptLayer","title":"Building an eval - PromptLayer","navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-08-04T16:30:09.067Z","originalFileLocation":"sdks/evals/building-an-eval.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Reference for evaluate(...): dataset, runner, scorers, and every option.\">\u003cmeta data-mint-head name=\"application-name\" content=\"PromptLayer\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/promptlayer/HpVdv-oD31DvzW7-/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/promptlayer/HpVdv-oD31DvzW7-/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/promptlayer/HpVdv-oD31DvzW7-/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/promptlayer/HpVdv-oD31DvzW7-/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/promptlayer/HpVdv-oD31DvzW7-/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/promptlayer/HpVdv-oD31DvzW7-/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/promptlayer/HpVdv-oD31DvzW7-/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/promptlayer/HpVdv-oD31DvzW7-/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/promptlayer/HpVdv-oD31DvzW7-/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Building an eval - PromptLayer\">\u003cmeta data-mint-head property=\"og:description\" content=\"Reference for evaluate(...): dataset, runner, scorers, and every option.\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"PromptLayer\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.promptlayer.com/sdks/evals/building-an-eval\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://promptlayer.mintlify.site/_mintlify/api/og/v2?division=Writing+evals\u0026amp;title=Building+an+eval\u0026amp;description=Reference+for+evaluate%28...%29%3A+dataset%2C+runner%2C+scorers%2C+and+every+option.\u0026amp;theme=f177da22b333da0ec1c7b1d0\u0026amp;signature=72f3e69b7ff4c5c3067b373f02c8c82fd178f93ae0cd8214fc856d857db5ae42\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Building an eval - PromptLayer\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Reference for evaluate(...): dataset, runner, scorers, and every option.\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://promptlayer.mintlify.site/_mintlify/api/og/v2?division=Writing+evals\u0026amp;title=Building+an+eval\u0026amp;description=Reference+for+evaluate%28...%29%3A+dataset%2C+runner%2C+scorers%2C+and+every+option.\u0026amp;theme=f177da22b333da0ec1c7b1d0\u0026amp;signature=72f3e69b7ff4c5c3067b373f02c8c82fd178f93ae0cd8214fc856d857db5ae42\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/promptlayer/HpVdv-oD31DvzW7-/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"PromptLayer\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#12A7D3\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.promptlayer.com/sdks/evals/building-an-eval\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/sdks/evals/building-an-eval.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.promptlayer.com/#organization\",\"name\":\"PromptLayer\",\"url\":\"https://docs.promptlayer.com\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/promptlayer/v0RzaTvbzopITX7U/logo/logo.png?fit=max\u0026auto=format\u0026n=v0RzaTvbzopITX7U\u0026q=85\u0026s=0decf70e3c444d489fa5f1152951bd6c\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.promptlayer.com#website\",\"name\":\"PromptLayer\",\"url\":\"https://docs.promptlayer.com\",\"publisher\":{\"@id\":\"https://docs.promptlayer.com/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.promptlayer.com/sdks/evals/building-an-eval#webpage\",\"url\":\"https://docs.promptlayer.com/sdks/evals/building-an-eval\",\"name\":\"Building an eval\",\"description\":\"Reference for evaluate(...): dataset, runner, scorers, and every option.\",\"dateModified\":\"2026-08-04T16:30:09.067Z\",\"isPartOf\":{\"@id\":\"https://docs.promptlayer.com#website\"},\"breadcrumb\":{\"@id\":\"https://docs.promptlayer.com/sdks/evals/building-an-eval#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.promptlayer.com/sdks/evals/building-an-eval#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Core Concepts\",\"item\":\"https://docs.promptlayer.com/features/prompt-registry/overview\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"SDK Evals\",\"item\":\"https://docs.promptlayer.com/sdks/evals/overview\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Writing evals\",\"item\":\"https://docs.promptlayer.com/sdks/evals/quickstart\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Building an eval\",\"item\":\"https://docs.promptlayer.com/sdks/evals/building-an-eval\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://docs.promptlayer.com/sdks/evals/building-an-eval#article\",\"headline\":\"Building an eval\",\"name\":\"Building an eval\",\"description\":\"Reference for evaluate(...): dataset, runner, scorers, and every option.\",\"url\":\"https://docs.promptlayer.com/sdks/evals/building-an-eval\",\"mainEntityOfPage\":{\"@id\":\"https://docs.promptlayer.com/sdks/evals/building-an-eval#webpage\"},\"dateModified\":\"2026-08-04T16:30:09.067Z\",\"publisher\":{\"@id\":\"https://docs.promptlayer.com/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.promptlayer.com#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"PromptLayer\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"PromptLayer\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/sdks/evals/building-an-eval.md","siteName":"PromptLayer","nav":{"global":null,"groups":[{"group":"Get Started","pages":[{"title":"Quickstart","icon":"rocket","href":"/quickstart"},{"title":"Tutorial Videos","icon":"video","href":"/tutorial-videos"}]},{"group":"Core Concepts","pages":[{"group":"Prompt Registry","icon":"notebook","pages":[{"title":"Prompt Registry","sidebarTitle":"Overview","icon":"book","href":"/features/prompt-registry/overview"},{"title":"Editor and Versioning","icon":"pen-line","href":"/features/prompt-registry/prompt-editor-versioning"},{"title":"Template Variables","icon":"pen-ruler","href":"/features/prompt-registry/template-variables"},{"title":"Input Variable Sets","icon":"floppy-disk","href":"/features/prompt-registry/input-variable-sets"},{"title":"Snippets","icon":"link","href":"/features/prompt-registry/snippets"},{"title":"Snippet Dependency Ordering","icon":"diagram-project","href":"/features/prompt-registry/snippet-dependency-ordering"},{"title":"Placeholder Messages","icon":"square-question","href":"/features/prompt-registry/placeholder-messages"},{"title":"Structured Outputs","icon":"diagram-project","href":"/features/prompt-registry/structured-outputs"},{"title":"Tool Calling","icon":"function","href":"/features/prompt-registry/tool-calling"},{"title":"Image Generation","icon":"image","href":"/features/image-generation"},{"title":"Playground","icon":"circle-play","href":"/why-promptlayer/playground"},{"title":"Sharing Requests","icon":"share","href":"/features/prompt-history/sharing-prompts"},{"title":"Release Labels","icon":"tag","href":"/features/prompt-registry/release-labels"},{"title":"Dynamic Release Labels","icon":"tags","href":"/features/prompt-registry/dynamic-release-labels"},{"title":"A/B Testing","icon":"split","href":"/why-promptlayer/ab-releases"}]},{"group":"Tables","icon":"table","tag":"New","pages":[{"title":"Overview","icon":"table","href":"/features/tables/overview"},{"title":"Sheets","icon":"table-cells","href":"/features/tables/sheets"},{"title":"Columns","icon":"columns-3","href":"/features/tables/columns"},{"title":"Column Types","icon":"brackets-curly","href":"/features/tables/column-types"},{"title":"Cells and Runs","icon":"play","href":"/features/tables/cells-and-runs"},{"title":"Scorecards","icon":"clipboard-check","href":"/features/tables/scorecards"},{"title":"History and Analytics","icon":"chart-line","href":"/features/tables/history-and-analytics"},{"title":"Migrate from Evaluations and Datasets","icon":"arrows-rotate","href":"/features/tables/migrate-from-evaluations-and-datasets"}]},{"group":"SDK Evals","icon":"vial-circle-check","tag":"New","pages":[{"title":"Overview","sidebarTitle":"Overview","icon":"book","href":"/sdks/evals/overview"},{"group":"Writing evals","icon":"screwdriver-wrench","pages":[{"title":"Quickstart","sidebarTitle":"Quickstart","icon":"rocket","description":null,"href":"/sdks/evals/quickstart"},{"title":"Building an eval","sidebarTitle":"Building an eval","icon":"pen-to-square","href":"/sdks/evals/building-an-eval"},{"title":"Datasets","description":"Pass inline cases or a dashboard Table as the evaluate(...) dataset.","sidebarTitle":"Datasets","icon":"database","href":"/sdks/evals/datasets"},{"title":"Runner and tracing","sidebarTitle":"Runner","icon":"diagram-project","href":"/sdks/evals/agent-tracing"},{"title":"Columns","sidebarTitle":"Columns","icon":"table-columns","href":"/sdks/evals/columns/overview"},{"title":"Scorers","sidebarTitle":"Scorers","icon":"clipboard-check","href":"/sdks/evals/scorers/overview"}]},{"group":"Running evals","icon":"play","pages":[{"title":"CLI and CI","sidebarTitle":"CLI and CI","icon":"terminal","href":"/sdks/evals/cli-and-ci"},{"title":"Results and failures","sidebarTitle":"Results and failures","icon":"circle-check","href":"/sdks/evals/results-and-failures"},{"title":"Concurrency","sidebarTitle":"Concurrency","icon":"layer-group","href":"/sdks/evals/concurrency"}]}]},{"group":"Observability","icon":"chart-column","pages":[{"title":"Overview","icon":"book","href":"/features/observability/overview"},{"group":"Request Logs","icon":"rectangle-list","pages":[{"title":"Overview","icon":"rectangle-list","href":"/features/observability/request-logs"},{"title":"Request IDs","icon":"id-card","href":"/features/observability/request-logs/request-ids"},{"title":"Tags","icon":"tag","href":"/features/observability/request-logs/tags"},{"title":"Metadata","icon":"brackets-curly","href":"/features/observability/request-logs/metadata"},{"title":"Scores","icon":"star","href":"/features/observability/request-logs/scores"},{"title":"Signals","icon":"wave-pulse","href":"/features/observability/request-logs/signals"},{"title":"Tracking Templates","icon":"chart-scatter-bubble","href":"/features/observability/request-logs/tracking-templates"},{"title":"Custom Logging","icon":"brackets-curly","href":"/features/observability/request-logs/custom-logging"},{"title":"Structured Outputs","icon":"brackets-curly","href":"/features/observability/request-logs/structured-outputs"}]},{"group":"Traces","icon":"diagram-project","pages":[{"title":"Overview","icon":"diagram-project","href":"/features/observability/traces"},{"title":"Manual Tracing","icon":"code","href":"/features/observability/traces/manual-tracing"},{"group":"SDK Auto-Instrumentation","icon":"code","pages":[{"title":"Overview","icon":"wave-pulse","href":"/features/observability/traces/auto-instrumentation/overview"},{"title":"OpenAI SDK","icon":"https://mintlify.s3.us-west-1.amazonaws.com/promptlayer/images/provider-logos/openai.svg","href":"/features/observability/traces/auto-instrumentation/openai"},{"title":"Anthropic SDK","icon":"https://mintlify.s3.us-west-1.amazonaws.com/promptlayer/images/provider-logos/anthropic.svg","href":"/features/observability/traces/auto-instrumentation/anthropic"},{"title":"Google SDK","icon":"https://mintlify.s3.us-west-1.amazonaws.com/promptlayer/images/provider-logos/google.svg","href":"/features/observability/traces/auto-instrumentation/google"},{"title":"AWS Bedrock","icon":"aws","href":"/features/observability/traces/auto-instrumentation/bedrock"}]},{"title":"OpenTelemetry","icon":"https://mintlify.s3.us-west-1.amazonaws.com/promptlayer/images/provider-logos/opentelemetry.svg","href":"/features/observability/traces/opentelemetry"},{"title":"Telemetry Integrations","icon":"handshake","href":"/features/observability/traces/integrations"}]}]},{"group":"Search \u0026 Analytics","icon":"chart-pie-simple","pages":[{"title":"Search","icon":"magnifying-glass","href":"/features/search-and-analytics/search"},{"title":"Analytics","icon":"chart-pie-simple","href":"/features/search-and-analytics/analytics"},{"title":"Search Data Model","icon":"diagram-project","href":"/features/search-and-analytics/search-data-model"}]},{"title":"Workflows","icon":"sitemap","href":"/why-promptlayer/workflows"},{"group":"Tool Registry","icon":"wrench","tag":"New","pages":[{"title":"Overview","icon":"wrench","href":"/features/tool-registry/overview"},{"title":"Using Tools in Prompts","icon":"link","href":"/features/tool-registry/using-in-prompts"},{"title":"Save Inline to Registry","icon":"floppy-disk","href":"/features/tool-registry/save-inline-to-registry"},{"title":"Auto Tool Execution","icon":"play","href":"/features/tool-registry/auto-execution"},{"title":"Mock Tool Responses","icon":"flask","href":"/features/tool-registry/mock-tool-responses"}]},{"group":"MCP Servers","icon":"server","tag":"New","pages":[{"title":"MCP Servers","icon":"server","href":"/features/mcp-servers/overview"}]},{"group":"Skill Collections","icon":"book-sparkles","tag":"New","pages":[{"title":"Overview","icon":"book","href":"/features/skill-collections/overview"},{"title":"Editing with Wrangler AI","icon":"message-bot","href":"/features/skill-collections/editing-with-wrangler"},{"title":"Sharing and remixing collections","icon":"share","href":"/features/skill-collections/sharing-and-remixing"},{"title":"Pulling skills into your agent","icon":"download","href":"/features/skill-collections/pulling-skills"},{"title":"Tune skills for your team","icon":"sliders","href":"/features/skill-collections/tuning-skills"}]},{"group":"Orgs \u0026 Workspaces","icon":"users-gear","pages":[{"title":"Overview","icon":"book","href":"/why-promptlayer/shared-workspaces"},{"title":"Organizations","icon":"users-gear","href":"/why-promptlayer/organizations"},{"title":"Workspaces","icon":"handshake-simple","href":"/why-promptlayer/workspaces"},{"title":"Data Retention","icon":"box-archive","href":"/why-promptlayer/data-retention"},{"title":"RBAC","icon":"shield","href":"/why-promptlayer/rbac"},{"group":"Enterprise Identity","icon":"id-card","pages":[{"title":"Overview","icon":"id-card","href":"/why-promptlayer/enterprise-identity/overview"},{"title":"Setup","icon":"plug","href":"/why-promptlayer/enterprise-identity/setup"},{"title":"Provisioning \u0026 access","icon":"users","href":"/why-promptlayer/enterprise-identity/provisioning"},{"title":"Sign-in policy \u0026 break-glass","icon":"shield","href":"/why-promptlayer/enterprise-identity/sign-in-security"},{"title":"Audit log","icon":"list","href":"/why-promptlayer/enterprise-identity/audit-log"}]}]}]},{"group":"Providers","pages":[{"title":"Supported Providers","icon":"list-check","href":"/features/supported-providers"},{"title":"OpenRouter","icon":"route","href":"/features/openrouter-integration"},{"group":"Custom Providers","icon":"plug","pages":[{"title":"Configure Providers","icon":"screwdriver-wrench","href":"/features/custom-providers"},{"title":"Exa","icon":"magnifying-glass","href":"/features/exa-integration"},{"title":"xAI (Grok)","icon":"bolt","href":"/features/xai-integration"},{"title":"Concentrate AI","icon":"layer-group","href":"/features/concentrate-integration"}]}]},{"group":"Guides","pages":[{"group":"Deployment","icon":"ship","pages":[{"title":"Choose a Deployment Strategy","icon":"ship","href":"/onboarding-guides/deployment-strategies"},{"title":"Zero Downtime Releases","icon":"rocket","href":"/features/prompt-registry/zero-downtime-releases"}]},{"group":"Self-Hosted PromptLayer","icon":"server","pages":[{"title":"Self-Hosted PromptLayer","icon":"server","href":"/self-hosted"},{"title":"Deploy PromptLayer on AWS","sidebarTitle":"AWS Deployment","icon":"aws","href":"/enterprise-deployments/aws"}]},{"group":"Additional Use Cases","icon":"lightbulb","pages":[{"title":"Multi-Turn Chat","icon":"messages","href":"/why-promptlayer/multi-turn-chat"},{"title":"Fine-Tuning","icon":"dumbbell","href":"/why-promptlayer/fine-tuning"},{"title":"Prompt Blueprints","icon":"layer-group","href":"/running-requests/prompt-blueprints"}]}]},{"group":"AI Tools","pages":[{"title":"MCP \u0026 Skills","href":"/agents/overview"}]},{"group":"Reference","pages":[{"group":"SDKs","pages":[{"title":"Python","href":"/sdks/python"},{"title":"JavaScript","href":"/sdks/javascript"}]},{"group":"REST API","pages":[{"title":"Introduction","href":"/reference/introduction"},{"group":"Prompt Templates","pages":[{"title":"Get Prompt Template","openapi":"POST /prompt-templates/{identifier}","href":"/reference/templates-get"},{"title":"Get Prompt Template (Raw)","openapi":"GET /prompt-templates/{identifier}","href":"/reference/templates-get-raw"},{"title":"List Prompt Templates","openapi":"GET /prompt-templates","href":"/reference/list-prompt-templates"},{"title":"Publish Prompt Template","openapi":"POST /rest/prompt-templates","href":"/reference/templates-publish"},{"title":"Patch Prompt Template Version","openapi":"PATCH /rest/prompt-templates/{identifier}","href":"/reference/templates-patch"},{"title":"List Prompt Template Labels","openapi":"GET /prompt-templates/{identifier}/labels","href":"/reference/templates-labels-get"},{"title":"Create a Prompt Template Label","openapi":"POST /prompts/{prompt_id}/label","href":"/reference/prompt-labels-create"},{"title":"Move Prompt Template Labels","openapi":"PATCH /prompt-labels/{prompt_label_id}","href":"/reference/prompt-labels-patch"},{"title":"Delete a Prompt Template Label","openapi":"DELETE /prompt-labels/{prompt_label_id}","href":"/reference/prompt-labels-delete"},{"title":"Get Snippet Usage","openapi":"GET /prompt-templates/{identifier}/snippet-usage","href":"/reference/get-snippet-usage"},{"title":"Upsert Prompt Template by External ID","openapi":"PUT /prompt-templates/by-external-id/{source}/{external_id}","href":"/reference/external-ids-prompt-templates-upsert"},{"title":"Attach Prompt Template External ID","openapi":"POST /prompt-templates/{prompt_template_id}/external-ids","href":"/reference/external-ids-prompt-templates-attach"},{"title":"List Prompt Template External IDs","openapi":"GET /prompt-templates/{prompt_template_id}/external-ids","href":"/reference/external-ids-prompt-templates-list"},{"title":"Delete Prompt Template External ID","openapi":"DELETE /prompt-templates/{prompt_template_id}/external-ids/{source}/{external_id}","href":"/reference/external-ids-prompt-templates-delete"}]},{"group":"Tracking","pages":[{"title":"Get Request","openapi":"GET /api/public/v2/requests/{request_id}","href":"/reference/get-request"},{"title":"Get Trace","openapi":"GET /api/public/v2/traces/{trace_id}","href":"/reference/get-trace"},{"title":"Close Trace","openapi":"POST /api/public/v2/traces/{trace_id}/close","href":"/reference/close-trace"},{"title":"Search Request Logs","openapi":"POST /api/public/v2/requests/search","href":"/reference/search-request-logs"},{"title":"Search Request Suggestions","openapi":"GET /api/public/v2/requests/suggestions","href":"/reference/search-request-suggestions"},{"title":"Request Analytics","openapi":"POST /api/public/v2/requests/analytics","href":"/reference/request-analytics"},{"title":"Request Analytics — Custom Queries","openapi":"POST /api/public/v2/requests/analytics/custom-analytics","href":"/reference/request-analytics-custom-analytics"},{"title":"Trace Analytics — Custom Queries","openapi":"POST /api/public/v2/traces/analytics/custom-analytics","href":"/reference/trace-analytics-custom-analytics"},{"title":"Log Request","openapi":"POST /log-request","href":"/reference/log-request"},{"title":"Track Prompt","openapi":"POST /rest/track-prompt","href":"/reference/track-prompt"},{"title":"Track Score","openapi":"POST /rest/track-score","href":"/reference/track-score"},{"title":"Track Metadata","openapi":"POST /rest/track-metadata","href":"/reference/track-metadata"},{"title":"Create Spans Bulk","openapi":"POST /spans-bulk","href":"/reference/spans-bulk"},{"title":"Ingest Traces (OTLP)","openapi":"POST /v1/traces","href":"/reference/otlp-ingest-traces"}]},{"group":"Tables","tag":"New","pages":[{"title":"Create Table","openapi":"POST /api/public/v2/tables","href":"/reference/tables-create"},{"title":"List Tables","openapi":"GET /api/public/v2/tables","href":"/reference/tables-list"},{"title":"Get Table","openapi":"GET /api/public/v2/tables/{table_id}","href":"/reference/tables-get"},{"title":"Update Table","openapi":"PATCH /api/public/v2/tables/{table_id}","href":"/reference/tables-update"},{"title":"Delete Table","openapi":"DELETE /api/public/v2/tables/{table_id}","href":"/reference/tables-delete"},{"title":"List Sheets","openapi":"GET /api/public/v2/tables/{table_id}/sheets","href":"/reference/table-sheets-list"},{"title":"Create Sheet","openapi":"POST /api/public/v2/tables/{table_id}/sheets","href":"/reference/table-sheets-create"},{"title":"Get Sheet Import Operation","openapi":"GET /api/public/v2/tables/{table_id}/sheets/operations/{operation_id}","href":"/reference/table-sheets-get-operation"},{"title":"Get Sheet","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}","href":"/reference/table-sheets-get"},{"title":"Update Sheet","openapi":"PATCH /api/public/v2/tables/{table_id}/sheets/{sheet_id}","href":"/reference/table-sheets-update"},{"title":"Delete Sheet","openapi":"DELETE /api/public/v2/tables/{table_id}/sheets/{sheet_id}","href":"/reference/table-sheets-delete"},{"title":"List Columns","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/columns","href":"/reference/table-sheet-columns-list"},{"title":"Create Column","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/columns","href":"/reference/table-sheet-columns-create"},{"title":"Update Column","openapi":"PATCH /api/public/v2/tables/{table_id}/sheets/{sheet_id}/columns/{column_id}","href":"/reference/table-sheet-columns-update"},{"title":"Delete Column","openapi":"DELETE /api/public/v2/tables/{table_id}/sheets/{sheet_id}/columns/{column_id}","href":"/reference/table-sheet-columns-delete"},{"title":"List Rows","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/rows","href":"/reference/table-sheet-rows-list"},{"title":"Add Rows","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/rows","href":"/reference/table-sheet-rows-add"},{"title":"Get Cell","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/cells/{cell_id}","href":"/reference/table-sheet-cells-get"},{"title":"Update Cell","openapi":"PATCH /api/public/v2/tables/{table_id}/sheets/{sheet_id}/cells/{cell_id}","href":"/reference/table-sheet-cells-update"},{"title":"Recalculate Cell","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/cells/{cell_id}/recalculations","href":"/reference/table-sheet-cells-recalculate"},{"title":"Recalculate Cells (Batch)","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/cells/recalculations","href":"/reference/table-sheet-cells-recalculate-batch"},{"title":"Import File","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/imports/file","href":"/reference/table-sheet-imports-file-create"},{"title":"Import Request Logs","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/imports/request-logs","href":"/reference/table-sheet-imports-request-logs-create"},{"title":"List Operations","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/operations","href":"/reference/table-sheet-operations-list"},{"title":"Create Operation","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/operations","href":"/reference/table-sheet-operations-create"},{"title":"Get Operation","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/operations/{operation_id}","href":"/reference/table-sheet-operations-get"},{"title":"Cancel Operation","openapi":"DELETE /api/public/v2/tables/{table_id}/sheets/{sheet_id}/operations/{operation_id}","href":"/reference/table-sheet-operations-cancel"},{"title":"Get Score","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/score","href":"/reference/table-sheet-score-get"},{"title":"Configure Score","openapi":"PATCH /api/public/v2/tables/{table_id}/sheets/{sheet_id}/score","href":"/reference/table-sheet-score-configure"},{"title":"Recalculate Score","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/score","href":"/reference/table-sheet-score-recalculate"},{"title":"Get Scorecard","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/scorecard","href":"/reference/table-sheet-scorecard-get"},{"title":"Configure Scorecard","openapi":"PATCH /api/public/v2/tables/{table_id}/sheets/{sheet_id}/scorecard","href":"/reference/table-sheet-scorecard-configure"},{"title":"Delete Scorecard","openapi":"DELETE /api/public/v2/tables/{table_id}/sheets/{sheet_id}/scorecard","href":"/reference/table-sheet-scorecard-delete"},{"title":"Migrate Legacy Score to Scorecard","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/scorecard/migrate-legacy-score","href":"/reference/table-sheet-scorecard-migrate-legacy-score"},{"title":"Recalculate Scorecard","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/scorecard/recalculate","href":"/reference/table-sheet-scorecard-recalculate"},{"title":"Cancel Scorecard","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/scorecard/cancel","href":"/reference/table-sheet-scorecard-cancel"},{"title":"Get Scorecard Calculation","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/scorecard/calculations/{calculation_id}","href":"/reference/table-sheet-scorecard-calculation-get"},{"title":"List Scorecard Rows","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/scorecard/rows","href":"/reference/table-sheet-scorecard-rows-list"},{"title":"Get Scorecard Row","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/scorecard/rows/{row_index}","href":"/reference/table-sheet-scorecard-row-get"},{"title":"List Versions","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/versions","href":"/reference/table-sheet-versions-list"},{"title":"Create Version","openapi":"POST /api/public/v2/tables/{table_id}/sheets/{sheet_id}/versions","href":"/reference/table-sheet-versions-create"},{"title":"Get Version","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/versions/{version_id}","href":"/reference/table-sheet-versions-get"},{"title":"Get Score History","openapi":"GET /api/public/v2/tables/{table_id}/sheets/{sheet_id}/versions/score-history","href":"/reference/table-sheet-score-history-get"}]},{"group":"Workflows","pages":[{"title":"List Workflows","openapi":"GET /workflows","href":"/reference/list-workflows"},{"title":"Get Workflow","openapi":"GET /workflows/{workflow_id_or_name}","href":"/reference/get-workflow"},{"title":"Get Workflow Labels","openapi":"GET /workflows/{workflow_id_or_name}/labels","href":"/reference/get-workflow-labels"},{"title":"Create Workflow","openapi":"POST /rest/workflows","href":"/reference/create-workflow"},{"title":"Update Workflow (PATCH)","openapi":"PATCH /rest/workflows/{workflow_id_or_name}","href":"/reference/patch-workflow"},{"title":"Run Workflow","openapi":"POST /workflows/{workflow_name}/run","href":"/reference/run-workflow"},{"title":"Get Workflow Version Execution Results","openapi":"GET /workflow-version-execution-results","href":"/reference/workflow-version-execution-results"},{"title":"Attach Workflow External ID","openapi":"POST /workflows/{workflow_id}/external-ids","href":"/reference/external-ids-workflows-attach"},{"title":"List Workflow External IDs","openapi":"GET /workflows/{workflow_id}/external-ids","href":"/reference/external-ids-workflows-list"},{"title":"Delete Workflow External ID","openapi":"DELETE /workflows/{workflow_id}/external-ids/{source}/{external_id}","href":"/reference/external-ids-workflows-delete"}]},{"group":"Skill Collections","pages":[{"title":"List Skill Collections","openapi":"GET /api/public/v2/skill-collections","href":"/reference/list-skill-collections"},{"title":"Create Skill Collection","openapi":"POST /api/public/v2/skill-collections","href":"/reference/create-skill-collection"},{"title":"Get Skill Collection","openapi":"GET /api/public/v2/skill-collections/{identifier}","href":"/reference/get-skill-collection"},{"title":"Update Skill Collection","openapi":"PATCH /api/public/v2/skill-collections/{identifier}","href":"/reference/update-skill-collection"},{"title":"Save Skill Collection Version","openapi":"POST /api/public/v2/skill-collections/{identifier}/versions","href":"/reference/save-skill-collection-version"},{"title":"Attach Skill Collection External ID","openapi":"POST /api/public/v2/skill-collections/{skill_collection_id}/external-ids","href":"/reference/external-ids-skill-collections-attach"},{"title":"List Skill Collection External IDs","openapi":"GET /api/public/v2/skill-collections/{skill_collection_id}/external-ids","href":"/reference/external-ids-skill-collections-list"},{"title":"Delete Skill Collection External ID","openapi":"DELETE /api/public/v2/skill-collections/{skill_collection_id}/external-ids/{source}/{external_id}","href":"/reference/external-ids-skill-collections-delete"}]},{"group":"Tool Registry","pages":[{"title":"List Tool Registries","openapi":"GET /api/public/v2/tool-registry","href":"/reference/tool-registry-list"},{"title":"Get Tool Registry","openapi":"GET /api/public/v2/tool-registry/{identifier}","href":"/reference/tool-registry-get"},{"title":"Create Tool Registry","openapi":"POST /api/public/v2/tool-registry","href":"/reference/tool-registry-create"},{"title":"Create Tool Version","openapi":"POST /api/public/v2/tool-registry/{identifier}/versions","href":"/reference/tool-registry-create-version"},{"title":"Test Execute Tool","openapi":"POST /api/public/v2/tool-registry/{identifier}/test-execute","href":"/reference/tool-registry-test-execute"},{"title":"Attach Tool Registry External ID","openapi":"POST /api/public/v2/tool-registry/{tool_id}/external-ids","href":"/reference/external-ids-tool-registry-attach"},{"title":"List Tool Registry External IDs","openapi":"GET /api/public/v2/tool-registry/{tool_id}/external-ids","href":"/reference/external-ids-tool-registry-list"},{"title":"Delete Tool Registry External ID","openapi":"DELETE /api/public/v2/tool-registry/{tool_id}/external-ids/{source}/{external_id}","href":"/reference/external-ids-tool-registry-delete"}]},{"group":"Environment Variables","tag":"New","pages":[{"title":"List Workspace Env Vars","openapi":"GET /api/public/v2/env-vars","href":"/reference/env-vars-workspace-list"},{"title":"Create Workspace Env Var","openapi":"POST /api/public/v2/env-vars","href":"/reference/env-vars-workspace-create"},{"title":"Update Workspace Env Var","openapi":"PATCH /api/public/v2/env-vars/{var_id}","href":"/reference/env-vars-workspace-update"},{"title":"Delete Workspace Env Var","openapi":"DELETE /api/public/v2/env-vars/{var_id}","href":"/reference/env-vars-workspace-delete"},{"title":"List Tool Env Vars","openapi":"GET /api/public/v2/tool-registry/{identifier}/env-vars","href":"/reference/env-vars-tool-list"},{"title":"Create Tool Env Var","openapi":"POST /api/public/v2/tool-registry/{identifier}/env-vars","href":"/reference/env-vars-tool-create"},{"title":"Update Tool Env Var","openapi":"PATCH /api/public/v2/tool-registry/{identifier}/env-vars/{var_id}","href":"/reference/env-vars-tool-update"},{"title":"Delete Tool Env Var","openapi":"DELETE /api/public/v2/tool-registry/{identifier}/env-vars/{var_id}","href":"/reference/env-vars-tool-delete"}]},{"group":"Unified Registry","pages":[{"title":"Create Folder","openapi":"POST /api/public/v2/folders","href":"/reference/create-folder"},{"title":"Update Folder","openapi":"PATCH /api/public/v2/folders/{folder_id}","href":"/reference/update-folder"},{"title":"List Folder Entities","openapi":"GET /api/public/v2/folders/entities","href":"/reference/list-folder-entities"},{"title":"Move Folder Entities","openapi":"POST /api/public/v2/folders/entities","href":"/reference/move-folder-entities"},{"title":"Delete Folder Entities","openapi":"DELETE /api/public/v2/folders/entities","href":"/reference/delete-folder-entities"},{"title":"Resolve Folder ID by Path","openapi":"GET /api/public/v2/folders/resolve-id","href":"/reference/resolve-folder-id"},{"title":"Attach Folder External ID","openapi":"POST /api/public/v2/folders/{folder_id}/external-ids","href":"/reference/external-ids-folders-attach"},{"title":"List Folder External IDs","openapi":"GET /api/public/v2/folders/{folder_id}/external-ids","href":"/reference/external-ids-folders-list"},{"title":"Delete Folder External ID","openapi":"DELETE /api/public/v2/folders/{folder_id}/external-ids/{source}/{external_id}","href":"/reference/external-ids-folders-delete"}]},{"title":"External IDs","href":"/reference/external-ids-overview"}]},{"group":"Webhooks","pages":[{"title":"Introduction","href":"/features/prompt-registry/webhooks"},{"title":"Events","href":"/features/prompt-registry/webhook-events"}]}]},{"group":"Deprecated","pages":[{"group":"Evaluations","icon":"vial-circle-check","pages":[{"title":"Overview","icon":"book","href":"/features/evaluations/overview"},{"title":"Getting Started","icon":"flag-checkered","href":"/features/evaluations/building-pipelines"},{"title":"Eval Types","icon":"screwdriver-wrench","href":"/features/evaluations/eval-types"},{"title":"Node \u0026 Column Types","icon":"brackets-curly","href":"/features/evaluations/column-types"},{"title":"Score Card","icon":"star","href":"/features/evaluations/score-card"},{"title":"Continuous Integration","icon":"arrows-spin","href":"/features/evaluations/continuous-integration"},{"title":"Online or Programmatic Evals","icon":"terminal","href":"/features/evaluations/programmatic"},{"title":"Eval Examples","icon":"people-group","href":"/features/evaluations/examples"},{"title":"Voice Agents","icon":"microphone","href":"/why-promptlayer/voice-agents"}]},{"group":"Datasets","icon":"database","pages":[{"title":"Overview","icon":"book","href":"/features/evaluations/datasets-overview"},{"title":"Create from File","icon":"file-arrow-up","href":"/features/evaluations/datasets-create-from-file"},{"title":"Create from History","icon":"history","href":"/features/evaluations/datasets-create-from-history"}]},{"group":"Eval Strategies","icon":"vial-circle-check","pages":[{"title":"Compare Models","icon":"scale-balanced","href":"/onboarding-guides/compare-models"},{"title":"Backtest Prompt Changes","icon":"clock-rotate-left","href":"/onboarding-guides/backtesting-prompt-changes"},{"title":"Run Batch Jobs","icon":"table","href":"/onboarding-guides/batch-runs"}]},{"group":"Tables","icon":"table","pages":[{"title":"Legacy Scoring","icon":"gauge","href":"/features/tables/scoring"}]},{"group":"Datasets API","pages":[{"title":"List Datasets","openapi":"GET /api/public/v2/datasets","href":"/reference/list-datasets"},{"title":"Get Dataset Rows","openapi":"GET /api/public/v2/datasets/{dataset_id}/rows","href":"/reference/get-dataset-rows"},{"title":"Create Dataset Group","openapi":"POST /api/public/v2/dataset-groups","href":"/reference/create-dataset-group"},{"title":"Create Dataset Version from File","openapi":"POST /api/public/v2/dataset-versions/from-file","href":"/reference/create-dataset-version-from-file"},{"title":"Create Dataset Version from Request History","openapi":"POST /api/public/v2/dataset-versions/from-filter-params","href":"/reference/create-dataset-version-from-filter-params"},{"title":"Create Draft Dataset Version","openapi":"POST /api/public/v2/dataset-versions/create-draft","href":"/reference/create-draft-dataset-version"},{"title":"Add Request Log to Dataset","openapi":"POST /api/public/v2/dataset-versions/add-request-log","href":"/reference/add-request-log-to-dataset"},{"title":"Add Trace to Dataset","openapi":"POST /api/public/v2/dataset-versions/add-trace","href":"/reference/add-trace-to-dataset"},{"title":"Save Draft Dataset Version","openapi":"POST /api/public/v2/dataset-versions/save-draft","href":"/reference/save-draft-dataset-version"},{"title":"Attach Dataset Group External ID","openapi":"POST /api/public/v2/dataset-groups/{dataset_group_id}/external-ids","href":"/reference/external-ids-dataset-groups-attach"},{"title":"List Dataset Group External IDs","openapi":"GET /api/public/v2/dataset-groups/{dataset_group_id}/external-ids","href":"/reference/external-ids-dataset-groups-list"},{"title":"Delete Dataset Group External ID","openapi":"DELETE /api/public/v2/dataset-groups/{dataset_group_id}/external-ids/{source}/{external_id}","href":"/reference/external-ids-dataset-groups-delete"}]},{"group":"Evaluations \u0026 Reports API","pages":[{"title":"List Evaluations","openapi":"GET /api/public/v2/evaluations","href":"/reference/list-evaluations"},{"title":"Get Evaluation Rows","openapi":"GET /api/public/v2/evaluations/{evaluation_id}/rows","href":"/reference/get-evaluation-rows"},{"title":"Create Evaluation Pipeline","openapi":"POST /reports","href":"/reference/create-reports"},{"title":"Run Full Evaluation","openapi":"POST /reports/{report_id}/run","href":"/reference/run-report"},{"title":"Get Evaluation","openapi":"GET /reports/{report_id}","href":"/reference/get-report"},{"title":"Get Evaluation Score","openapi":"GET /reports/{report_id}/score","href":"/reference/get-report-score"},{"title":"Add Column to Evaluation Pipeline","openapi":"POST /report-columns","href":"/reference/add-report-columns"},{"title":"Edit Evaluation Pipeline Column","openapi":"PATCH /report-columns/{report_column_id}","href":"/reference/edit-report-column"},{"title":"Delete Evaluation Pipeline Column","openapi":"DELETE /report-columns/{report_column_id}","href":"/reference/delete-report-column"},{"title":"Configure Custom Scoring","openapi":"PATCH /reports/{report_id}/score-card","href":"/reference/update-report-score-card"},{"title":"Rename Evaluation Pipeline","openapi":"PATCH /reports/{report_id}/rename","href":"/reference/rename-report"},{"title":"Delete Evaluation Pipeline","openapi":"DELETE /reports/{report_id}","href":"/reference/delete-report"},{"title":"Delete Reports by Name","openapi":"DELETE /reports/name/{report_name}","href":"/reference/delete-reports-by-name"},{"title":"Attach Report External ID","openapi":"POST /reports/{report_id}/external-ids","href":"/reference/external-ids-reports-attach"},{"title":"List Report External IDs","openapi":"GET /reports/{report_id}/external-ids","href":"/reference/external-ids-reports-list"},{"title":"Delete Report External ID","openapi":"DELETE /reports/{report_id}/external-ids/{source}/{external_id}","href":"/reference/external-ids-reports-delete"}]}]}]},"basePath":"","title":"Building an eval - PromptLayer","deploymentId":"dpl_DyXGnKeCjZyA3sUdfMJ7FjkdBefg","buildId":"dpl_DyXGnKeCjZyA3sUdfMJ7FjkdBefg","dynamics":[],"chunks":[]}