Skip to main content
Legacy Evaluations, Reports, and Datasets are deprecated for new workflows. Use Tables for new evaluation, dataset, report, backtesting, and batch workflows. See Migrate from Evaluations and Datasets.
This page documents all available node types for Workflows and column types for evaluation pipelines. Workflows and evaluations share the same node types—each has specific configuration options that determine its behavior.
In Workflows, these are called nodes. In evaluation pipelines, they’re called columns. The configuration is identical.

How Column Sources Work

Columns can reference data from two places:
  1. Dataset columns - Reference data directly from your dataset by using the dataset column name
  2. Other evaluation columns - Reference the output of a previous column by using that column’s name
When you specify a source or include a column name in sources, the system first looks for an evaluation column with that name, then falls back to looking for a dataset column.
Columns are executed in order based on their position. A column can only reference other columns that come before it in the pipeline.

Example: Chaining Columns Together

A common pattern is to chain columns: run a prompt, extract a field from the JSON output, then compare it to a ground truth value from the dataset.

Execution Types

These columns execute prompts, code, or external services.
Runs a prompt template against each row. You can reference a template from the Prompt Registry or define one inline.Registry Reference (using template)
The prompt_template_variable_mappings object maps prompt input variables (keys) to dataset or column names (values). The key is the variable name in your prompt template (e.g., {{question}}), and the value is where to get the data from.
Complete example with all input variables:If your prompt template has variables {{company}}, {{product}}, and {{query}}, map each one:
Inline Template (using inline_template)Define a prompt template directly in the configuration without saving it to the registry. This is useful for quick experimentation or one-off evaluations.
You must provide exactly one of template or inline_template. They are mutually exclusive.
Executes custom Python or JavaScript code. The code receives a data dictionary containing all column values for the current row.
Calls an external HTTP endpoint. The request body contains all column values for the current row.
Runs a PromptLayer workflow.
Executes an MCP (Model Context Protocol) action.
Adds a column for manual human evaluation.
Simulates multi-turn conversations to test chatbots and conversational agents. An AI-powered user persona engages in realistic dialogue with your prompt template, allowing you to evaluate how well your agent handles extended interactions.
The user_persona defines how the simulated user behaves - their goals, communication style, and what questions they ask. Use user_persona_source to pull different personas from your dataset for varied test scenarios.
The conversation_completed_prompt provides explicit guidance for determining when a conversation should end. This is useful for defining specific end conditions like tool calls, confirmation messages, or goal achievement. The guidance can be holistic (general rules) or specific (look for a certain phrase or tool call).
Basic example with static persona:
Dynamic personas from dataset:For comprehensive testing, store different user personas in your dataset to test various scenarios:
Where your dataset has a test_persona column with different personas:
  • Row 1: “You are a busy executive who needs quick answers. Be impatient if responses are too long.”
  • Row 2: “You are a technical user who asks detailed follow-up questions about implementation.”
  • Row 3: “You are price-sensitive and keep asking about discounts and alternatives.”
Custom completion conditions:Use conversation_completed_prompt to define specific end conditions for your conversations:
You can also use conversation_completed_prompt_source to pull completion guidance from your dataset:
Where your dataset has a completion_condition column with different end conditions:
  • Row 1: “End when the user says ‘thank you’ or indicates satisfaction”
  • Row 2: “Complete when the assistant provides a ticket number”
  • Row 3: “End when the refund_process tool is called”
Evaluating conversation quality:Chain with LLM_ASSERTION to evaluate the full conversation:

Loop Types

These nodes enable iterating over collections or executing repeated operations within Agents.
Iterates over a collection of items or runs a fixed number of times, executing a prompt template or sub-workflow on each iteration.Special loop variables for variable_mappings:
  • loop_index - Current iteration index (0-based)
  • previous_outputs - Array of all outputs from previous iterations
  • _iterator_item - Current item from the iterated collection
Exactly one of iterator_source or max_iterations must be provided.
Output structure:
Executes repeatedly until an end condition is met or maximum iterations are reached.
Termination behavior:
  • If end_condition_json_path is set: Loop ends when JSONPath extracts a truthy value
  • If not set: Loop ends when output is falsy (empty, null, false)

Evaluation Types

These columns evaluate or compare data and typically return boolean or numeric scores.
Uses an LLM to evaluate content against a natural language prompt. Returns a boolean indicating pass/fail.Basic example with static prompt:
Dynamic prompts from dataset:Use prompt_source to pull assertion prompts from a dataset column. This lets you define different assertions per row.
Where your dataset has an assertions column containing the prompt text for each row.Multiple assertions per row:You can run multiple assertions against the same content by providing a JSON array of prompts. Each assertion is evaluated independently, and the results are returned as a dictionary.
Where your dataset’s llm_assertions column contains a JSON array:
The output will be a dictionary with each assertion as a key and its boolean result as the value.
Compares two values for equality. Supports string comparison and JSON comparison with optional JSONPath.
With JSON path:
Checks if a value contains a substring (case-insensitive).
Tests if content matches a regular expression pattern. Returns boolean.
Calculates semantic similarity between two texts using embeddings. Returns a float between 0 and 1.
Calculates the absolute difference between two numeric values.
Uses an LLM to extract specific information from content based on a natural language query.

Extraction Types

These columns extract or parse data from other columns.
Extracts data from JSON using JSONPath expressions.
Extracts data from XML using XPath expressions.
Extracts content matching a regular expression pattern. Returns an array of all matches.
Parses and converts a value to a specific type.

Transformation Types

These columns transform, combine, or validate data.
Creates a static value that can be referenced by other columns.String variable:
JSON variable:
Validates that data is in a valid format. Returns boolean.
Returns the first non-null value from multiple sources.
Combines multiple column values into a single dictionary object.
Counts occurrences in text content.
Performs numeric comparisons. Returns boolean.Compare to static value:
Compare two columns:
Finds the minimum or maximum value from an array or JSON structure.