In Workflows, these are called nodes. In evaluation pipelines, they’re called
columns. The configuration is identical.
How Column Sources Work
Columns can reference data from two places:- Dataset columns - Reference data directly from your dataset by using the dataset column name
- Other evaluation columns - Reference the output of a previous column by using that column’s
name
source or include a column name in sources, the system first looks for an evaluation column with that name, then falls back to looking for a dataset column.
Columns are executed in order based on their
position. A column can only
reference other columns that come before it in the pipeline.Example: Chaining Columns Together
A common pattern is to chain columns: run a prompt, extract a field from the JSON output, then compare it to a ground truth value from the dataset.Execution Types
These columns execute prompts, code, or external services.Prompt Template
Prompt Template
Runs a prompt template against each row. You can reference a template from the Prompt Registry or define one inline.Registry Reference (using Complete example with all input variables:If your prompt template has variables Inline Template (using
template)The
prompt_template_variable_mappings object maps prompt input variables (keys) to dataset or column names (values). The key is the variable name in your prompt template (e.g., {{question}}), and the value is where to get the data from.{{company}}, {{product}}, and {{query}}, map each one:inline_template)Define a prompt template directly in the configuration without saving it to the registry. This is useful for quick experimentation or one-off evaluations.Code Execution
Code Execution
Executes custom Python or JavaScript code. The code receives a
data dictionary containing all column values for the current row.Endpoint
Endpoint
Calls an external HTTP endpoint. The request body contains all column values for the current row.
Workflow
Workflow
Runs a PromptLayer workflow.
MCP
MCP
Executes an MCP (Model Context Protocol) action.
Human
Human
Adds a column for manual human evaluation.
Conversation Simulator
Conversation Simulator
Simulates multi-turn conversations to test chatbots and conversational agents. An AI-powered user persona engages in realistic dialogue with your prompt template, allowing you to evaluate how well your agent handles extended interactions.Basic example with static persona:Dynamic personas from dataset:For comprehensive testing, store different user personas in your dataset to test various scenarios:Where your dataset has a You can also use Where your dataset has a
The
user_persona defines how the simulated user behaves - their goals, communication style, and what questions they ask. Use user_persona_source to pull different personas from your dataset for varied test scenarios.The
conversation_completed_prompt provides explicit guidance for determining when a conversation should end. This is useful for defining specific end conditions like tool calls, confirmation messages, or goal achievement. The guidance can be holistic (general rules) or specific (look for a certain phrase or tool call).test_persona column with different personas:- Row 1: “You are a busy executive who needs quick answers. Be impatient if responses are too long.”
- Row 2: “You are a technical user who asks detailed follow-up questions about implementation.”
- Row 3: “You are price-sensitive and keep asking about discounts and alternatives.”
conversation_completed_prompt to define specific end conditions for your conversations:conversation_completed_prompt_source to pull completion guidance from your dataset:completion_condition column with different end conditions:- Row 1: “End when the user says ‘thank you’ or indicates satisfaction”
- Row 2: “Complete when the assistant provides a ticket number”
- Row 3: “End when the refund_process tool is called”
LLM_ASSERTION to evaluate the full conversation:Loop Types
These nodes enable iterating over collections or executing repeated operations within Agents.For Loop
For Loop
Iterates over a collection of items or runs a fixed number of times, executing a prompt template or sub-workflow on each iteration.Output structure:
Special loop variables for
variable_mappings:loop_index- Current iteration index (0-based)previous_outputs- Array of all outputs from previous iterations_iterator_item- Current item from the iterated collection
While Loop
While Loop
Executes repeatedly until an end condition is met or maximum iterations are reached.
Termination behavior:
- If
end_condition_json_pathis set: Loop ends when JSONPath extracts a truthy value - If not set: Loop ends when output is falsy (empty, null, false)
Evaluation Types
These columns evaluate or compare data and typically return boolean or numeric scores.LLM Assertion
LLM Assertion
Uses an LLM to evaluate content against a natural language prompt. Returns a boolean indicating pass/fail.Dynamic prompts from dataset:Use Where your dataset has an Where your dataset’s The output will be a dictionary with each assertion as a key and its boolean result as the value.
Basic example with static prompt:
prompt_source to pull assertion prompts from a dataset column. This lets you define different assertions per row.assertions column containing the prompt text for each row.Multiple assertions per row:You can run multiple assertions against the same content by providing a JSON array of prompts. Each assertion is evaluated independently, and the results are returned as a dictionary.llm_assertions column contains a JSON array:Compare
Compare
Compares two values for equality. Supports string comparison and JSON comparison with optional JSONPath.With JSON path:
Contains
Contains
Checks if a value contains a substring (case-insensitive).
Regex
Regex
Tests if content matches a regular expression pattern. Returns boolean.
Cosine Similarity
Cosine Similarity
Calculates semantic similarity between two texts using embeddings. Returns a float between 0 and 1.
Absolute Numeric Distance
Absolute Numeric Distance
Calculates the absolute difference between two numeric values.
AI Data Extraction
AI Data Extraction
Uses an LLM to extract specific information from content based on a natural language query.
Extraction Types
These columns extract or parse data from other columns.JSON Path
JSON Path
Extracts data from JSON using JSONPath expressions.
XML Path
XML Path
Extracts data from XML using XPath expressions.
Regex Extraction
Regex Extraction
Extracts content matching a regular expression pattern. Returns an array of all matches.
Parse Value
Parse Value
Parses and converts a value to a specific type.
Transformation Types
These columns transform, combine, or validate data.Variable
Variable
Creates a static value that can be referenced by other columns.JSON variable:
String variable:
Assert Valid
Assert Valid
Validates that data is in a valid format. Returns boolean.
Coalesce
Coalesce
Returns the first non-null value from multiple sources.
Combine Columns
Combine Columns
Combines multiple column values into a single dictionary object.
Count
Count
Counts occurrences in text content.
Math Operator
Math Operator
Performs numeric comparisons. Returns boolean.Compare two columns:
Compare to static value:
Min/Max
Min/Max
Finds the minimum or maximum value from an array or JSON structure.

