Create Evaluation Pipeline

curl --request POST \
  --url https://api.promptlayer.com/reports \
  --header 'Content-Type: application/json' \
  --header 'X-API-KEY: <api-key>' \
  --data '
{
  "dataset_group_id": 123,
  "name": "Pipeline with Built-in Scoring",
  "columns": [
    {
      "column_type": "LLM_ASSERTION",
      "name": "Accuracy Check",
      "configuration": {
        "source": "response",
        "prompt": "Is this response accurate?"
      },
      "is_part_of_score": true
    },
    {
      "column_type": "LLM_ASSERTION",
      "name": "Safety Check",
      "configuration": {
        "source": "response",
        "prompt": "Is this response safe?"
      },
      "is_part_of_score": true
    }
  ]
}
'

{
  "success": true,
  "report_id": 456,
  "report_columns": [
    {
      "id": 789,
      "name": "Accuracy Check",
      "column_type": "LLM_ASSERTION",
      "position": 1,
      "configuration": {
        "source": "response"
      }
    }
  ],
  "external_ids": []
}

{
  "message": "Dataset must have at least one non-draft version"
}

{
  "success": false,
  "message": "<string>",
  "error": "<string>"
}

{
  "success": false,
  "message": "<string>",
  "error": "<string>"
}

{
  "success": false,
  "message": "<string>",
  "error": "<string>"
}

{
  "message": "<string>",
  "success": false
}

{
  "detail": [
    {
      "loc": [
        "<string>"
      ],
      "msg": "<string>",
      "type": "<string>"
    }
  ]
}

POST

reports

curl --request POST \
  --url https://api.promptlayer.com/reports \
  --header 'Content-Type: application/json' \
  --header 'X-API-KEY: <api-key>' \
  --data '
{
  "dataset_group_id": 123,
  "name": "Pipeline with Built-in Scoring",
  "columns": [
    {
      "column_type": "LLM_ASSERTION",
      "name": "Accuracy Check",
      "configuration": {
        "source": "response",
        "prompt": "Is this response accurate?"
      },
      "is_part_of_score": true
    },
    {
      "column_type": "LLM_ASSERTION",
      "name": "Safety Check",
      "configuration": {
        "source": "response",
        "prompt": "Is this response safe?"
      },
      "is_part_of_score": true
    }
  ]
}
'

{
  "success": true,
  "report_id": 456,
  "report_columns": [
    {
      "id": 789,
      "name": "Accuracy Check",
      "column_type": "LLM_ASSERTION",
      "position": 1,
      "configuration": {
        "source": "response"
      }
    }
  ],
  "external_ids": []
}

{
  "message": "Dataset must have at least one non-draft version"
}

{
  "success": false,
  "message": "<string>",
  "error": "<string>"
}

{
  "success": false,
  "message": "<string>",
  "error": "<string>"
}

{
  "success": false,
  "message": "<string>",
  "error": "<string>"
}

{
  "message": "<string>",
  "success": false
}

{
  "detail": [
    {
      "loc": [
        "<string>"
      ],
      "msg": "<string>",
      "type": "<string>"
    }
  ]
}

Legacy Dataset, Evaluation, and Report endpoints are deprecated for new workflows. Use the Tables API for new dataset import, evaluation, scoring, recalculation, and reporting workflows.

Create an Evaluation Pipeline associated with a dataset group. Use this endpoint to create a pipeline blueprint with optional columns, custom scoring configuration, folder placement, and external IDs.

Behavior Notes

Evaluation columns use the same node definitions as Workflows. See Node & Column Types.
Set is_part_of_score on columns for built-in scoring, or provide score_configuration for custom scoring logic.
Custom scoring concepts are covered in Score Card.

Authorizations

X-API-KEY

string

header

required

Body

application/json

Evaluation pipeline creation payload.

dataset_group_id

integer

required

ID of the dataset group to use.

name

string | null

Name for the pipeline. Auto-generated if omitted.

folder_id

integer | null

Folder ID for organization.

dataset_version_number

integer | null

Specific dataset version. Uses latest published version if omitted.

columns

EvaluationColumnDefinition · object[] | null

Evaluation columns to add to the pipeline.

Show child attributes

score_configuration

ScoreConfiguration · object | null

Optional custom scoring logic.

Show child attributes

Example:

{
  "code": "score = sum(1 for row in data if row.get(\"Accuracy Check\") is True) / len(data) * 100 if data else 0\nreturn {\"score\": score}",
  "code_language": "PYTHON"
}

external_ids

ExternalId · object[]

External ID mappings to attach to the pipeline.

Show child attributes

Response

Evaluation pipeline created.

success

enum<boolean>

required

Available options:

true

report_id

integer

required

ID of the created evaluation pipeline.

external_ids

ExternalId · object[]

required

External ID mappings attached to the pipeline.

Show child attributes

report_columns

ReportColumnSummary · object[]

Columns created on the pipeline.

Show child attributes

Get Evaluation Rows Run Full Evaluation

⌘I

Get Started

Core Concepts

Providers

Guides

AI Tools

Reference

Deprecated

Create Evaluation Pipeline

Behavior Notes

Authorizations

Body

Response

​Behavior Notes

​Related

Authorizations

Body

Response

Behavior Notes

Related