Skip to main content
This page documents all available column types for evaluation pipelines. Each column type has specific configuration options that determine its behavior.

​
Execution Types

These columns execute prompts, code, or external services.
Runs a prompt template from the registry against each row.
FieldTypeRequiredDescription
template.namestringYesName of the prompt template
template.version_numberintegerNoSpecific version number. Uses latest if omitted
template.labelstringNoRelease label to use, e.g. “production”
prompt_template_variable_mappingsobjectYesMaps template input variables to dataset/column names
engineobjectNoOverride the template’s default model settings
engine.providerstringNoProvider name, e.g. “openai”, “anthropic”
engine.modelstringNoModel name, e.g. “gpt-4”, “claude-3-opus”
engine.parametersobjectNoModel parameters like temperature, max_tokens
{
  "column_type": "PROMPT_TEMPLATE",
  "name": "Generate Response",
  "configuration": {
    "template": {
      "name": "my-prompt",
      "label": "production"
    },
    "prompt_template_variable_mappings": {
      "question": "user_question",
      "context": "retrieved_context"
    }
  }
}
Executes custom Python or JavaScript code. The code receives a data dictionary containing all column values for the current row.
FieldTypeRequiredDescription
codestringYesThe code to execute
languagestringYes”PYTHON” or “JAVASCRIPT”
{
  "column_type": "CODE_EXECUTION",
  "name": "Custom Logic",
  "configuration": {
    "code": "result = len(data['response'].split())\nreturn result",
    "language": "PYTHON"
  }
}
Calls an external HTTP endpoint. The request body contains all column values for the current row.
FieldTypeRequiredDescription
urlstringYesThe HTTP endpoint URL
headersobjectNoHTTP headers to include
{
  "column_type": "ENDPOINT",
  "name": "External Validator",
  "configuration": {
    "url": "https://api.example.com/validate",
    "headers": {
      "Authorization": "Bearer token123"
    }
  }
}
Runs a PromptLayer workflow.
FieldTypeRequiredDescription
workflow_idintegerYesID of the workflow to run
workflow_version_numberintegerNoSpecific version. Uses latest if omitted
workflow_labelstringNoRelease label to use
input_mappingsobjectYesMaps workflow inputs to column names
{
  "column_type": "WORKFLOW",
  "name": "Run Analysis Workflow",
  "configuration": {
    "workflow_id": 123,
    "input_mappings": {
      "input_text": "response"
    }
  }
}
Executes an MCP (Model Context Protocol) action.
FieldTypeRequiredDescription
mcp_server_idintegerYesID of the MCP server
tool_namestringYesName of the tool to call
input_mappingsobjectYesMaps tool inputs to column names
{
  "column_type": "MCP",
  "name": "MCP Tool Call",
  "configuration": {
    "mcp_server_id": 456,
    "tool_name": "search",
    "input_mappings": {
      "query": "search_query"
    }
  }
}
Adds a column for manual human evaluation.
FieldTypeRequiredDescription
data_typestringYes”number” or “string”
ui_elementobjectYesUI configuration for the input
{
  "column_type": "HUMAN",
  "name": "Human Rating",
  "configuration": {
    "data_type": "number",
    "ui_element": {
      "type": "slider",
      "min": 1,
      "max": 5
    }
  }
}

​
Evaluation Types

These columns evaluate or compare data and typically return boolean or numeric scores.
Uses an LLM to evaluate content against a natural language prompt. Returns a boolean indicating pass/fail.
FieldTypeRequiredDescription
sourcestringYesColumn name containing the content to evaluate
promptstringConditionalThe assertion prompt. Required if prompt_source not set
prompt_sourcestringConditionalColumn name containing the prompt. Required if prompt not set
{
  "column_type": "LLM_ASSERTION",
  "name": "Quality Check",
  "configuration": {
    "source": "response",
    "prompt": "Is this response helpful, accurate, and free of harmful content?"
  },
  "is_part_of_score": true
}
Compares two values for equality. Supports string comparison and JSON comparison with optional JSONPath.
FieldTypeRequiredDescription
sourcesarrayYesArray of exactly 2 column names to compare
comparison_type.typestringYes”STRING” or “JSON”
comparison_type.json_pathstringNoJSONPath to extract before comparing. Only for JSON type
{
  "column_type": "COMPARE",
  "name": "Accuracy",
  "configuration": {
    "sources": ["predicted_value", "ground_truth"],
    "comparison_type": {"type": "STRING"}
  },
  "is_part_of_score": true
}
With JSON path:
{
  "column_type": "COMPARE",
  "name": "JSON Field Match",
  "configuration": {
    "sources": ["api_response", "expected_response"],
    "comparison_type": {
      "type": "JSON",
      "json_path": "$.result.status"
    }
  }
}
Checks if a value contains a substring (case-insensitive).
FieldTypeRequiredDescription
sourcestringYesColumn name to search in
valuestringConditionalStatic substring to find. Required if value_source not set
value_sourcestringConditionalColumn name containing the substring. Required if value not set
{
  "column_type": "CONTAINS",
  "name": "Has Keyword",
  "configuration": {
    "source": "response",
    "value": "thank you"
  }
}
Tests if content matches a regular expression pattern. Returns boolean.
FieldTypeRequiredDescription
sourcestringYesColumn name to test
regex_patternstringYesRegular expression pattern
{
  "column_type": "REGEX",
  "name": "Valid Email Format",
  "configuration": {
    "source": "email_field",
    "regex_pattern": "^[\\w.-]+@[\\w.-]+\\.\\w+$"
  }
}
Calculates semantic similarity between two texts using embeddings. Returns a float between 0 and 1.
FieldTypeRequiredDescription
sourcesarrayYesArray of exactly 2 column names to compare
{
  "column_type": "COSINE_SIMILARITY",
  "name": "Semantic Similarity",
  "configuration": {
    "sources": ["generated_response", "reference_response"]
  },
  "is_part_of_score": true
}
Calculates the absolute difference between two numeric values.
FieldTypeRequiredDescription
sourcesarrayYesArray of exactly 2 column names containing numbers
{
  "column_type": "ABSOLUTE_NUMERIC_DISTANCE",
  "name": "Score Difference",
  "configuration": {
    "sources": ["predicted_score", "actual_score"]
  }
}
Uses an LLM to extract specific information from content based on a natural language query.
FieldTypeRequiredDescription
sourcestringYesColumn name containing the content
querystringYesNatural language description of what to extract
{
  "column_type": "AI_DATA_EXTRACTION",
  "name": "Extract Sentiment",
  "configuration": {
    "source": "response",
    "query": "What is the overall sentiment? Return only: positive, negative, or neutral"
  }
}

​
Extraction Types

These columns extract or parse data from other columns.
Extracts data from JSON using JSONPath expressions.
FieldTypeRequiredDescription
sourcestringYesColumn name containing JSON data
json_pathstringYesJSONPath expression (e.g., “.field",".field", ".items[0].name”)
return_first_matchbooleanNoReturn only first match (default: true) or all matches
{
  "column_type": "JSON_PATH",
  "name": "Extract Agent",
  "configuration": {
    "source": "llm_output",
    "json_path": "$.selected_agent",
    "return_first_match": true
  }
}
Extracts data from XML using XPath expressions.
FieldTypeRequiredDescription
sourcestringYesColumn name containing XML data
xml_pathstringYesXPath expression
typestringNo”find” for first match or “findall” for all matches. Default: “find”
return_textbooleanNoReturn text content only or full XML. Default: true
{
  "column_type": "XML_PATH",
  "name": "Extract Title",
  "configuration": {
    "source": "xml_response",
    "xml_path": ".//item/title",
    "type": "find",
    "return_text": true
  }
}
Extracts content matching a regular expression pattern. Returns an array of all matches.
FieldTypeRequiredDescription
sourcestringYesColumn name to extract from
regex_patternstringYesRegular expression pattern
{
  "column_type": "REGEX_EXTRACTION",
  "name": "Extract Numbers",
  "configuration": {
    "source": "text_content",
    "regex_pattern": "\\d+\\.?\\d*"
  }
}
Parses and converts a value to a specific type.
FieldTypeRequiredDescription
sourcestringYesColumn name to parse
typestringYesTarget type: “string”, “number”, “boolean”, or “object”
{
  "column_type": "PARSE_VALUE",
  "name": "Parse Score",
  "configuration": {
    "source": "score_string",
    "type": "number"
  }
}

​
Transformation Types

These columns transform, combine, or validate data.
Creates a static value that can be referenced by other columns.
FieldTypeRequiredDescription
value.typestringYes”string” or “json”
value.valueanyYesThe static value
String variable:
{
  "column_type": "VARIABLE",
  "name": "Environment",
  "configuration": {
    "value": {
      "type": "string",
      "value": "production"
    }
  }
}
JSON variable:
{
  "column_type": "VARIABLE",
  "name": "Config",
  "configuration": {
    "value": {
      "type": "json",
      "value": {"threshold": 0.8, "max_retries": 3}
    }
  }
}
Validates that data is in a valid format. Returns boolean.
FieldTypeRequiredDescription
sourcestringYesColumn name to validate
typestringYesExpected format: “object” for valid JSON, “number”, or “sql”
{
  "column_type": "ASSERT_VALID",
  "name": "Is Valid JSON",
  "configuration": {
    "source": "api_response",
    "type": "object"
  }
}
Returns the first non-null value from multiple sources.
FieldTypeRequiredDescription
sourcesarrayYesArray of column names, minimum 2
{
  "column_type": "COALESCE",
  "name": "Best Response",
  "configuration": {
    "sources": ["primary_response", "fallback_response", "default_response"]
  }
}
Combines multiple column values into a single dictionary object.
FieldTypeRequiredDescription
sourcesarrayYesArray of column names to combine
{
  "column_type": "COMBINE_COLUMNS",
  "name": "Combined Context",
  "configuration": {
    "sources": ["question", "context", "metadata"]
  }
}
Counts occurrences in text content.
FieldTypeRequiredDescription
sourcestringYesColumn name to count in
typestringYesWhat to count: “chars”, “words”, “sentences”, or “paragraphs”
{
  "column_type": "COUNT",
  "name": "Word Count",
  "configuration": {
    "source": "response",
    "type": "words"
  }
}
Performs numeric comparisons. Returns boolean.
FieldTypeRequiredDescription
sourcesarrayYesArray with first source column, and optionally second source column
operatorstringYesComparison operator: “lt” for less than, “le” for less or equal, “gt” for greater than, “ge” for greater or equal
valuenumberConditionalStatic value to compare against. Required if second source not provided
Compare to static value:
{
  "column_type": "MATH_OPERATOR",
  "name": "Above Threshold",
  "configuration": {
    "sources": ["score"],
    "operator": "ge",
    "value": 0.8
  }
}
Compare two columns:
{
  "column_type": "MATH_OPERATOR",
  "name": "A Greater Than B",
  "configuration": {
    "sources": ["score_a", "score_b"],
    "operator": "gt"
  }
}
Finds the minimum or maximum value from an array or JSON structure.
FieldTypeRequiredDescription
sourcestringYesColumn name containing the data
typestringYes”min” or “max”
json_pathstringNoJSONPath to extract values from, if source is JSON
{
  "column_type": "MIN_MAX",
  "name": "Highest Score",
  "configuration": {
    "source": "scores_array",
    "type": "max",
    "json_path": "$[*].value"
  }
}