Columns

Pass columns into evaluate(...) to transform row data before scorers run. On the experiment sheet they show up as computed columns (Prompt Template, JSON Path, and the rest). They use the same types as Tables Column Types. You can also add them from the Columns UI.

Columns at a glance

Column	`ColumnType`	Does
Prompt Template	`PROMPT_TEMPLATE`	Run a Registry or inline prompt per row
JSON Path	`JSON_PATH`	Extract a value from a JSON column
Coalesce	`COALESCE`	First non-null value across sources
Composition	`COMPOSITION`	Mirror an upstream column value
Cosine Similarity	`COSINE_SIMILARITY`	Embedding similarity between two columns
AI Data Extraction	`AI_DATA_EXTRACTION`	Extract info from a column with an LLM query
For Loop	`FOR_LOOP`	Iterate a prompt/workflow over a collection or count
While Loop	`WHILE_LOOP`	Repeat a prompt/workflow until a condition or cap

Column types

Each helper below maps to a ColumnType in Tables Column Types.

Prompt Template

Run a Prompt Registry or inline prompt per row (ColumnType.PROMPT_TEMPLATE). Provide exactly one of template or inline_template.

Key	Required	Notes
`template`	one of	Registry ref: `name` or `id`; optional `version_number` or `label` (not both)
`inline_template`	one of	Inline prompt content
`prompt_template_variable_mappings`	yes	Variable → column title
`engine`	no	Optional model override
`verbose`	no	Default `false`
`return_template_only`	no	Default `false`
`hydrate_input_variables`	no	Default `true`
`chat_history_source`	no	Optional chat history column

from openai import OpenAI
from promptlayer import evaluate, column, ColumnType, compare_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": "You are a support agent. Answer in one short sentence.",
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "prompt-column-demo",
    dataset=[{"input": "cats", "expected": "OK"}],
    runner=run_llm,
    columns=[
        column(
            "Prompt output",
            ColumnType.PROMPT_TEMPLATE,
            {
                "template": {"name": "summarize", "version_number": 1},
                "prompt_template_variable_mappings": {"topic": "Input"},
            },
        )
    ],
    scorers=[compare_scorer(sources=["Prompt output", "Expected"])],
)

import OpenAI from "openai";
import { evaluate, column, ColumnType, compareScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content: "You are a support agent. Answer in one short sentence.",
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("prompt-column-demo", {
  dataset: [{ input: "cats", expected: "OK" }],
  runner: runLlm,
  columns: [
    column("Prompt output", ColumnType.PROMPT_TEMPLATE, {
      template: { name: "summarize", version_number: 1 },
      prompt_template_variable_mappings: { topic: "Input" },
    }),
  ],
  scorers: [compareScorer({ sources: ["Prompt output", "Expected"] })],
});

JSON Path

Extract a value from a JSON column (ColumnType.JSON_PATH).

Key	Required	Notes
`source`	yes	Column title
`json_path`	yes	JSONPath expression
`return_first_match`	no	Set explicitly; do not rely on defaults

from openai import OpenAI
from promptlayer import evaluate, column, ColumnType, contains_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": (
                    "Reply with JSON only, no markdown. "
                    'Shape: {"user":{"id":"<id>"}}'
                ),
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "json-path-demo",
    dataset=[{"input": "Create a user object with id 42."}],
    runner=run_llm,
    columns=[
        column(
            "User id",
            ColumnType.JSON_PATH,
            {"source": "Output", "json_path": "$.user.id", "return_first_match": True},
        )
    ],
    scorers=[contains_scorer(source="User id", value="42")],
)

import OpenAI from "openai";
import { evaluate, column, ColumnType, containsScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content:
          'Reply with JSON only, no markdown. Shape: {"user":{"id":"<id>"}}',
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("json-path-demo", {
  dataset: [{ input: "Create a user object with id 42." }],
  runner: runLlm,
  columns: [
    column("User id", ColumnType.JSON_PATH, {
      source: "Output",
      json_path: "$.user.id",
      return_first_match: true,
    }),
  ],
  scorers: [containsScorer({ source: "User id", value: "42" })],
});

Coalesce

Return the first non-null value from two or more sources (ColumnType.COALESCE).

Key	Required	Notes
`sources`	yes	At least two column titles

from openai import OpenAI
from promptlayer import evaluate, column, ColumnType, contains_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": "You are a support agent. Answer in one short sentence.",
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "coalesce-demo",
    dataset=[
        {
            "input": "How do I reset my password?",
            "expected": "primary",
        }
    ],
    runner=run_llm,
    columns=[
        column(
            "Primary or output",
            ColumnType.COALESCE,
            {"sources": ["Expected", "Output"]},
        )
    ],
    scorers=[contains_scorer(source="Primary or output", value="primary")],
)

import OpenAI from "openai";
import { evaluate, column, ColumnType, containsScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content: "You are a support agent. Answer in one short sentence.",
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("coalesce-demo", {
  dataset: [
    {
      input: "How do I reset my password?",
      expected: "primary",
    },
  ],
  runner: runLlm,
  columns: [
    column("Primary or output", ColumnType.COALESCE, {
      sources: ["Expected", "Output"],
    }),
  ],
  scorers: [containsScorer({ source: "Primary or output", value: "primary" })],
});

Composition

Mirror an upstream column value without re-running upstream logic (ColumnType.COMPOSITION).

Key	Required	Notes
`source`	one of	Same-sheet title, or `{table}.{sheet}.{column}`
`sources`	one of	Also accepted by the backend

from promptlayer import column, ColumnType

column("Mirrored output", ColumnType.COMPOSITION, {"source": "Output"})

import { column, ColumnType } from "promptlayer";

column("Mirrored output", ColumnType.COMPOSITION, { source: "Output" });

Cosine Similarity

Embedding cosine similarity between exactly two source columns (ColumnType.COSINE_SIMILARITY). The example below computes a Similarity column; score that column when you need a pass/fail bar (often custom code).

Key	Required	Notes
`sources`	yes	Exactly two column titles

from openai import OpenAI
from promptlayer import evaluate, column, ColumnType, count_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": "Reply with a friendly greeting in a few words.",
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "cosine-demo",
    dataset=[{"input": "Say hello to the customer.", "expected": "hello there"}],
    runner=run_llm,
    columns=[
        column(
            "Similarity",
            ColumnType.COSINE_SIMILARITY,
            {"sources": ["Output", "Expected"]},
        )
    ],
    scorers=[count_scorer(source="Output", type="words", min_count=1)],
)

import OpenAI from "openai";
import { evaluate, column, ColumnType, countScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content: "Reply with a friendly greeting in a few words.",
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("cosine-demo", {
  dataset: [{ input: "Say hello to the customer.", expected: "hello there" }],
  runner: runLlm,
  columns: [
    column("Similarity", ColumnType.COSINE_SIMILARITY, {
      sources: ["Output", "Expected"],
    }),
  ],
  scorers: [countScorer({ source: "Output", type: "words", minCount: 1 })],
});

AI Data Extraction

Extract information from a source column with an LLM query (ColumnType.AI_DATA_EXTRACTION).

Key	Required	Notes
`source`	yes	Column title
`query`	yes	Non-empty extraction query

from openai import OpenAI
from promptlayer import evaluate, column, ColumnType, contains_scorer

client = OpenAI()

def run_llm(user_message: str) -> str:
    completion = client.chat.completions.create(
        model="gpt-5.6",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a shipping assistant. Mention the order number "
                    "and when it ships in one short sentence."
                ),
            },
            {"role": "user", "content": str(user_message)},
        ],
    )
    return completion.choices[0].message.content or ""

evaluate(
    "extract-demo",
    dataset=[{"input": "Customer asks about order 99 shipping tomorrow."}],
    runner=run_llm,
    columns=[
        column(
            "Order id",
            ColumnType.AI_DATA_EXTRACTION,
            {"source": "Output", "query": "Extract the order number only"},
        )
    ],
    scorers=[contains_scorer(source="Order id", value="99")],
)

import OpenAI from "openai";
import { evaluate, column, ColumnType, containsScorer } from "promptlayer";

const client = new OpenAI();

async function runLlm(userMessage) {
  const completion = await client.chat.completions.create({
    model: "gpt-5.6",
    messages: [
      {
        role: "system",
        content:
          "You are a shipping assistant. Mention the order number and when it ships in one short sentence.",
      },
      { role: "user", content: String(userMessage) },
    ],
  });
  return completion.choices[0].message.content ?? "";
}

await evaluate("extract-demo", {
  dataset: [{ input: "Customer asks about order 99 shipping tomorrow." }],
  runner: runLlm,
  columns: [
    column("Order id", ColumnType.AI_DATA_EXTRACTION, {
      source: "Output",
      query: "Extract the order number only",
    }),
  ],
  scorers: [containsScorer({ source: "Order id", value: "99" })],
});

For Loop

Iterate a prompt or workflow over a collection or a fixed count (ColumnType.FOR_LOOP). Provide exactly one of iterator_source or max_iterations.

Key	Required	Notes
`loop_type`	yes	`prompt` \| `workflow`
`prompt_config`	if prompt	Nested prompt config (`version_number`, not `version`)
`workflow_config`	if workflow	Needs `workflow_id` or `workflow_version_id`
`iterator_source`	one of	Source collection column
`max_iterations`	one of	Positive int
`return_all_outputs`	no	Default `false`
`variable_mappings`	no	Default `{}`; avoid `_iterator_source` / `_iterator_item` as keys

from promptlayer import column, ColumnType

column(
    "Expand items",
    ColumnType.FOR_LOOP,
    {
        "loop_type": "prompt",
        "iterator_source": "Input",
        "prompt_config": {
            "template": {"name": "process-item", "version_number": 1},
            "prompt_template_variable_mappings": {"item": "_iterator_item"},
        },
        "variable_mappings": {},
        "return_all_outputs": False,
    },
)

import { column, ColumnType } from "promptlayer";

column("Expand items", ColumnType.FOR_LOOP, {
  loop_type: "prompt",
  iterator_source: "Input",
  prompt_config: {
    template: { name: "process-item", version_number: 1 },
    prompt_template_variable_mappings: { item: "_iterator_item" },
  },
  variable_mappings: {},
  return_all_outputs: false,
});

While Loop

Repeat a prompt or workflow until a condition or max iterations (ColumnType.WHILE_LOOP).

Key	Required	Notes
`loop_type`	yes	`prompt` \| `workflow`
`prompt_config`	if prompt	Nested prompt config (`version_number`, not `version`)
`workflow_config`	if workflow	Needs `workflow_id` or `workflow_version_id`
`end_condition_json_path`	no	Stop condition path
`max_iterations`	no	Safety cap
`return_all_outputs`	no	Default `false`
`variable_mappings`	no	Default `{}`

from promptlayer import column, ColumnType

column(
    "Refine until done",
    ColumnType.WHILE_LOOP,
    {
        "loop_type": "prompt",
        "max_iterations": 3,
        "end_condition_json_path": "$.done",
        "prompt_config": {
            "template": {"name": "refine", "version_number": 1},
            "prompt_template_variable_mappings": {"draft": "Input"},
        },
        "variable_mappings": {},
        "return_all_outputs": False,
    },
)

import { column, ColumnType } from "promptlayer";

column("Refine until done", ColumnType.WHILE_LOOP, {
  loop_type: "prompt",
  max_iterations: 3,
  end_condition_json_path: "$.done",
  prompt_config: {
    template: { name: "refine", version_number: 1 },
    prompt_template_variable_mappings: { draft: "Input" },
  },
  variable_mappings: {},
  return_all_outputs: false,
});

Get Started

Core Concepts

Providers

Guides

AI Tools

Reference

Deprecated

Columns at a glance

Column types

Prompt Template

JSON Path

Coalesce

Composition

Cosine Similarity

AI Data Extraction

For Loop

While Loop

​Columns at a glance

​Column types

​Prompt Template

​JSON Path

​Coalesce

​Composition

​Cosine Similarity

​AI Data Extraction

​For Loop

​While Loop

Columns at a glance

Column types

Prompt Template

JSON Path

Coalesce

Composition

Cosine Similarity

AI Data Extraction

For Loop

While Loop