> ## Documentation Index
> Fetch the complete documentation index at: https://docs.promptlayer.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Create Evaluation Pipeline

Create an Evaluation Pipeline associated with a dataset group. Use this endpoint to create a pipeline blueprint with optional columns, custom scoring configuration, folder placement, and external IDs.

## Behavior Notes

* Evaluation columns use the same node definitions as Workflows. See [Node & Column Types](/features/evaluations/column-types).
* Set `is_part_of_score` on columns for built-in scoring, or provide `score_configuration` for custom scoring logic.
* Custom scoring concepts are covered in [Score Card](/features/evaluations/score-card).

## Related

* [Configure Custom Scoring](/reference/update-report-score-card)
* [Node & Column Types](/features/evaluations/column-types)
* [Score Card](/features/evaluations/score-card)


## OpenAPI

````yaml POST /reports
openapi: 3.1.0
info:
  title: FastAPI
  version: 0.1.0
servers: []
security:
  - ApiKeyAuth: []
paths:
  /reports:
    post:
      tags:
        - reports
      summary: Create Evaluation Pipeline
      operationId: createEvaluationPipeline
      requestBody:
        required: true
        description: Evaluation pipeline creation payload.
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreateEvaluationPipelineRequest'
            examples:
              builtInScoring:
                summary: Built-in scoring
                value:
                  dataset_group_id: 123
                  name: Pipeline with Built-in Scoring
                  columns:
                    - column_type: LLM_ASSERTION
                      name: Accuracy Check
                      configuration:
                        source: response
                        prompt: Is this response accurate?
                      is_part_of_score: true
                    - column_type: LLM_ASSERTION
                      name: Safety Check
                      configuration:
                        source: response
                        prompt: Is this response safe?
                      is_part_of_score: true
              customScoring:
                summary: Custom scoring
                value:
                  dataset_group_id: 123
                  name: Pipeline with Custom Scoring
                  columns:
                    - column_type: LLM_ASSERTION
                      name: Accuracy Check
                      configuration:
                        source: response
                        prompt: Is this response accurate?
                    - column_type: LLM_ASSERTION
                      name: Safety Check
                      configuration:
                        source: response
                        prompt: Is this response safe?
                  score_configuration:
                    code: >-
                      weights = {"Accuracy Check": 0.7, "Safety Check": 0.3}

                      total_weight = weighted_sum = 0

                      for row in data:
                          for col, weight in weights.items():
                              if col in row:
                                  total_weight += weight
                                  if row[col] == True:
                                      weighted_sum += weight
                      score = (weighted_sum / total_weight * 100) if
                      total_weight > 0 else 0

                      return {"score": round(score, 2)}
                    code_language: PYTHON
      responses:
        '201':
          description: Evaluation pipeline created.
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/CreateEvaluationPipelineResponse'
              examples:
                created:
                  summary: Pipeline created
                  value:
                    success: true
                    report_id: 456
                    report_columns:
                      - id: 789
                        name: Accuracy Check
                        column_type: LLM_ASSERTION
                        position: 1
                        configuration:
                          source: response
                    external_ids: []
        '400':
          description: >-
            Bad Request - Invalid parameters, draft dataset version, or no
            datasets in group
          content:
            application/json:
              schema:
                type: object
                properties:
                  message:
                    type: string
                    example: Dataset must have at least one non-draft version
        '401':
          $ref: '#/components/responses/UnauthorizedError'
        '403':
          $ref: '#/components/responses/ForbiddenError'
        '404':
          $ref: '#/components/responses/NotFoundError'
        '409':
          description: External ID conflict
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/ExternalIdErrorResponse'
        '422':
          $ref: '#/components/responses/ValidationError'
components:
  schemas:
    CreateEvaluationPipelineRequest:
      type: object
      title: CreateEvaluationPipelineRequest
      required:
        - dataset_group_id
      properties:
        dataset_group_id:
          type: integer
          description: ID of the dataset group to use.
        name:
          type: string
          nullable: true
          description: Name for the pipeline. Auto-generated if omitted.
        folder_id:
          type: integer
          nullable: true
          description: Folder ID for organization.
        dataset_version_number:
          type: integer
          nullable: true
          description: Specific dataset version. Uses latest published version if omitted.
        columns:
          type: array
          nullable: true
          items:
            $ref: '#/components/schemas/EvaluationColumnDefinition'
          description: Evaluation columns to add to the pipeline.
        score_configuration:
          anyOf:
            - $ref: '#/components/schemas/ScoreConfiguration'
            - type: 'null'
          description: Optional custom scoring logic.
        external_ids:
          type: array
          items:
            $ref: '#/components/schemas/ExternalId'
          description: External ID mappings to attach to the pipeline.
    CreateEvaluationPipelineResponse:
      type: object
      title: CreateEvaluationPipelineResponse
      required:
        - success
        - report_id
        - external_ids
      properties:
        success:
          type: boolean
          enum:
            - true
        report_id:
          type: integer
          description: ID of the created evaluation pipeline.
        report_columns:
          type: array
          items:
            $ref: '#/components/schemas/ReportColumnSummary'
          description: Columns created on the pipeline.
        external_ids:
          type: array
          items:
            $ref: '#/components/schemas/ExternalId'
          description: External ID mappings attached to the pipeline.
    ExternalIdErrorResponse:
      properties:
        success:
          type: boolean
          const: false
          default: false
          title: Success
        message:
          type: string
          title: Message
      type: object
      required:
        - message
      title: ExternalIdErrorResponse
    EvaluationColumnDefinition:
      type: object
      title: EvaluationColumnDefinition
      description: >-
        Definition for a column in an evaluation pipeline. The configuration
        schema depends on column_type.
      required:
        - column_type
        - name
        - configuration
      properties:
        column_type:
          type: string
          enum:
            - PROMPT_TEMPLATE
            - CODE_EXECUTION
            - ENDPOINT
            - WORKFLOW
            - MCP
            - HUMAN
            - CONVERSATION_SIMULATOR
            - LLM_ASSERTION
            - AI_DATA_EXTRACTION
            - COMPARE
            - CONTAINS
            - REGEX
            - COSINE_SIMILARITY
            - ABSOLUTE_NUMERIC_DISTANCE
            - JSON_PATH
            - XML_PATH
            - REGEX_EXTRACTION
            - PARSE_VALUE
            - VARIABLE
            - ASSERT_VALID
            - COALESCE
            - COMBINE_COLUMNS
            - COUNT
            - MATH_OPERATOR
            - MIN_MAX
          description: Type of evaluation column to create.
        name:
          type: string
          description: Display name for the column. Must be unique within the pipeline.
        configuration:
          type: object
          additionalProperties: true
          description: >-
            Column-type-specific configuration. See Node & Column Types for
            details.
        position:
          type: integer
          minimum: 1
          nullable: true
          description: 1-based position in the pipeline. Auto-assigned if omitted.
        is_part_of_score:
          type: boolean
          default: false
          description: >-
            When true, PromptLayer includes this column in built-in scoring by
            averaging its values.
      example:
        column_type: LLM_ASSERTION
        name: Accuracy Check
        configuration:
          source: response
          prompt: Is this response accurate?
        is_part_of_score: true
    ScoreConfiguration:
      type: object
      title: ScoreConfiguration
      description: >-
        Custom scoring configuration. The code receives a data variable
        containing row dictionaries and must return an object with at least a
        score key from 0 to 100.
      required:
        - code
      properties:
        code:
          type: string
          description: Python or JavaScript code used to calculate the score.
        code_language:
          type: string
          enum:
            - PYTHON
            - JAVASCRIPT
          default: PYTHON
          description: Language used by the scoring code.
      example:
        code: >-
          score = sum(1 for row in data if row.get("Accuracy Check") is True) /
          len(data) * 100 if data else 0

          return {"score": score}
        code_language: PYTHON
    ExternalId:
      properties:
        source:
          type: string
          maxLength: 128
          minLength: 1
          title: Source
          description: The external system or namespace that owns the ID.
        external_id:
          type: string
          maxLength: 512
          minLength: 1
          title: External Id
          description: The identifier for this entity in the external system.
      type: object
      required:
        - source
        - external_id
      additionalProperties: false
      title: ExternalId
      description: >-
        Customer-defined mapping between a PromptLayer entity and an external
        system identifier.
    ReportColumnSummary:
      type: object
      title: ReportColumnSummary
      additionalProperties: true
      properties:
        id:
          type: integer
        name:
          type: string
        column_type:
          type: string
          enum:
            - PROMPT_TEMPLATE
            - CODE_EXECUTION
            - ENDPOINT
            - WORKFLOW
            - MCP
            - HUMAN
            - CONVERSATION_SIMULATOR
            - LLM_ASSERTION
            - AI_DATA_EXTRACTION
            - COMPARE
            - CONTAINS
            - REGEX
            - COSINE_SIMILARITY
            - ABSOLUTE_NUMERIC_DISTANCE
            - JSON_PATH
            - XML_PATH
            - REGEX_EXTRACTION
            - PARSE_VALUE
            - VARIABLE
            - ASSERT_VALID
            - COALESCE
            - COMBINE_COLUMNS
            - COUNT
            - MATH_OPERATOR
            - MIN_MAX
            - DATASET
        position:
          type: integer
        configuration:
          additionalProperties: true
          type: object
    ErrorResponse:
      type: object
      properties:
        success:
          type: boolean
          default: false
          description: Indicates that the request failed.
        message:
          type: string
          description: Human-readable error message.
        error:
          type: string
          description: Machine-readable or fallback error message.
      additionalProperties: true
      description: >-
        Standard error response. Some legacy endpoints may return either message
        or error.
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          type: array
          title: Detail
      type: object
      title: HTTPValidationError
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
              - type: string
              - type: integer
          type: array
          title: Location
        msg:
          type: string
          title: Message
        type:
          type: string
          title: Error Type
      type: object
      required:
        - loc
        - msg
        - type
      title: ValidationError
  responses:
    UnauthorizedError:
      description: Unauthorized - missing or invalid API key.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
    ForbiddenError:
      description: Forbidden - API key does not have access to the requested resource.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
    NotFoundError:
      description: Not found - requested resource does not exist or is not accessible.
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/ErrorResponse'
    ValidationError:
      description: Validation error - request parameters or body are invalid.
      content:
        application/json:
          schema:
            oneOf:
              - $ref: '#/components/schemas/HTTPValidationError'
              - $ref: '#/components/schemas/ErrorResponse'
  securitySchemes:
    ApiKeyAuth:
      type: apiKey
      in: header
      name: X-API-KEY

````