test

Tests are first-class citizens in BAML, designed to make testing AI functions straightforward and robust. BAML tests can be written anywhere in your codebase and run with minimal setup.

Syntax

test TestName {
  functions [FunctionName]
  args {
    paramName value
  }
}

Basic Example

function ExtractEmail(text: string) -> string {
  client GPT4
  prompt #"
    Extract the email address from: {{ text }}
  "#
}

test TestEmailExtraction {
  functions [ExtractEmail]
  args {
    text "Contact John at [email protected] for details"
  }
}

Test Components

test

required

The test keyword begins the declaration.

TestName

identifier

required

Unique test name. Use descriptive PascalCase names.

test TestBasicExtraction { }
test TestEdgeCaseWithEmptyInput { }

functions

identifier[]

required

List of functions to test. Must have compatible signatures.

functions [ExtractEmail]
functions [ParsePerson, ValidatePerson]  // Multiple functions

args

required

Input arguments for the test. Must match function parameter types.

args {
  text "Sample input"
  count 42
}

type_builder

Optional block for defining or modifying dynamic types.

type_builder {
  class NewClass {
    field string
  }
}

@@check

Optional conditional check for test validity using Jinja expressions.

@@check(has_content, {{ this.field|length > 0 }})

@@assert

Optional assertion on test results using Jinja expressions.

@@assert({{ this.email|length > 0 }})

Input Types

Primitive Values

test TestPrimitives {
  functions [ProcessData]
  args {
    name "John Doe"
    age 30
    score 95.5
    isActive true
  }
}

Objects

class Person {
  name string
  age int
}

test TestObject {
  functions [AnalyzePerson]
  args {
    person {
      name "Alice"
      age 28
    }
  }
}

Nested Objects

class Message {
  user string
  content string
  metadata Metadata
}

class Metadata {
  timestamp int
  priority string
}

test TestNested {
  functions [ProcessMessage]
  args {
    message {
      user "john_doe"
      content "Hello world"
      metadata {
        timestamp 1234567890
        priority "high"
      }
    }
  }
}

Arrays

test TestArrays {
  functions [BatchProcess]
  args {
    items ["item1", "item2", "item3"]
    numbers [1, 2, 3, 4, 5]
    messages [
      {
        user "user1"
        content "Message 1"
      }
      {
        user "user2"
        content "Message 2"
      }
    ]
  }
}

Multi-line Strings

test TestLongText {
  functions [AnalyzeText]
  args {
    content #"
      This is a multi-line
      text input that preserves
      formatting and whitespace
    "#
  }
}

Media Inputs

Images

File Reference:

test TestImageFile {
  functions [AnalyzeImage]
  args {
    image {
      file "../images/test.png"
    }
  }
}

URL Reference:

test TestImageUrl {
  functions [AnalyzeImage]
  args {
    image {
      url "https://example.com/image.jpg"
    }
  }
}

Base64 Data:

test TestImageBase64 {
  functions [AnalyzeImage]
  args {
    image {
      base64 "iVBORw0KGgo..."
      media_type "image/png"
    }
  }
}

Audio

test TestAudioFile {
  functions [TranscribeAudio]
  args {
    audio {
      file "../audio/sample.mp3"
    }
  }
}

test TestAudioUrl {
  functions [TranscribeAudio]
  args {
    audio {
      url "https://example.com/audio.mp3"
    }
  }
}

test TestAudioBase64 {
  functions [TranscribeAudio]
  args {
    audio {
      base64 "//uQx..."
      media_type "audio/mp3"
    }
  }
}

PDFs

PDFs cannot be supplied via URL. Use file reference or base64 only.

test TestPdfFile {
  functions [AnalyzePdf]
  args {
    pdf {
      file "../documents/report.pdf"
    }
  }
}

test TestPdfBase64 {
  functions [AnalyzePdf]
  args {
    pdf {
      base64 "JVBERi0xLj..."
      media_type "application/pdf"
    }
  }
}

Videos

test TestVideoFile {
  functions [AnalyzeVideo]
  args {
    video {
      file "../videos/sample.mp4"
    }
  }
}

test TestVideoUrl {
  functions [AnalyzeVideo]
  args {
    video {
      url "https://example.com/video.mp4"
    }
  }
}

test TestVideoBase64 {
  functions [AnalyzeVideo]
  args {
    video {
      base64 "AAAAGGZ0eXBpc29t..."
      media_type "video/mp4"
    }
  }
}

Template Strings

Reuse string data across tests:

template_string EchoTestData(num: int) ##"
  The number is {{ num }}
"##

test TestWithTemplate {
  functions [Echo]
  args {
    input EchoTestData(42)
  }
  @@assert({{ this == "The number is 42" }})
}

Testing Multiple Functions

Test multiple functions with the same signature:

function ExtractInfo(text: string) -> Person { }
function ParseInfo(text: string) -> Person { }
function ValidateInfo(text: string) -> Person { }

test TestAllParsers {
  functions [
    ExtractInfo
    ParseInfo
    ValidateInfo
  ]
  args {
    text "John Doe, age 30, [email protected]"
  }
}

Assertions

Basic Assertions

test TestWithAssertion {
  functions [ExtractEmail]
  args {
    text "Contact: [email protected]"
  }
  @@assert({{ this|length > 0 }})
  @@assert({{ "@" in this }})
}

Multiple Assertions

test TestMultipleAssertions {
  functions [ParsePerson]
  args {
    text "Alice, 28 years old"
  }
  @@assert({{ this.name == "Alice" }})
  @@assert({{ this.age >= 18 }})
  @@assert({{ this.age < 100 }})
}

Conditional Checks

test TestWithCheck {
  functions [ProcessData]
  args {
    data "sample"
  }
  @@check(not_empty, {{ this.result|length > 0 }})
  @@assert({{ this.status == "success" }})
}

Dynamic Types

Modify types at test time:

class FlexibleData {
  id string
  name string
  
  @@dynamic
}

function ProcessData(data: FlexibleData) -> FlexibleData {
  client GPT4
  prompt #"
    Process: {{ data }}
    {{ ctx.output_format }}
  "#
}

test TestDynamicType {
  functions [ProcessData]
  type_builder {
    dynamic class FlexibleData {
      custom_field string
      extra_number int
    }
  }
  args {
    data {
      id "123"
      name "Test"
      custom_field "dynamic value"
      extra_number 42
    }
  }
}

Complete Examples

Email Extraction

function ExtractEmail(text: string) -> string {
  client GPT4
  prompt #"
    Extract the email from: {{ text }}
  "#
}

test TestValidEmail {
  functions [ExtractEmail]
  args {
    text "Contact John at [email protected]"
  }
  @@assert({{ "@" in this }})
  @@assert({{ "." in this }})
}

test TestNoEmail {
  functions [ExtractEmail]
  args {
    text "No email address here"
  }
}

Classification

enum Sentiment {
  Positive
  Negative
  Neutral
}

function ClassifySentiment(text: string) -> Sentiment {
  client GPT4
  prompt #"
    Classify sentiment: {{ text }}
    {{ ctx.output_format }}
  "#
}

test TestPositive {
  functions [ClassifySentiment]
  args {
    text "This is amazing! I love it!"
  }
  @@assert({{ this == "Positive" }})
}

test TestNegative {
  functions [ClassifySentiment]
  args {
    text "Terrible experience. Very disappointed."
  }
  @@assert({{ this == "Negative" }})
}

Structured Extraction

class Person {
  name string
  age int?
  email string?
}

function ParsePerson(text: string) -> Person {
  client GPT4
  prompt #"
    {{ ctx.output_format }}
    Extract person info from: {{ text }}
  "#
}

test TestCompletePerson {
  functions [ParsePerson]
  args {
    text "Alice Smith, 28, [email protected]"
  }
  @@assert({{ this.name == "Alice Smith" }})
  @@assert({{ this.age == 28 }})
  @@assert({{ this.email != null }})
}

test TestPartialPerson {
  functions [ParsePerson]
  args {
    text "Bob Johnson"
  }
  @@assert({{ this.name|length > 0 }})
}

Image Analysis

function DescribeImage(img: image) -> string {
  client "openai/gpt-4o"
  prompt #"
    Describe this image in one sentence:
    {{ img }}
  "#
}

test TestImageDescription {
  functions [DescribeImage]
  args {
    img {
      url "https://example.com/test-image.jpg"
    }
  }
  @@assert({{ this|length > 10 }})
}

VSCode Integration

Run from Playground: Tests can be executed directly in the BAML playground
Real-time Validation: Syntax errors highlighted as you type
Result Visualization: View test results inline
Quick Navigation: Jump to function definitions from tests

Running Tests

Tests run in the BAML playground or via CLI:

# Run all tests
baml test

# Run specific test
baml test TestEmailExtraction

# Run tests for specific function
baml test --function ExtractEmail

Best Practices

Naming: Use descriptive test names that explain what’s being tested
Coverage: Test both happy paths and edge cases
Assertions: Add assertions to validate expected behavior
Variety: Test with different input types and sizes
Media: Use small test files for media inputs
Documentation: Add comments explaining complex test cases
Organization: Group related tests near their functions
Isolation: Each test should be independent

Function → - Define functions to test
Class → - Define input/output types
Dynamic Types → - Runtime type modification
Testing Guide → - Testing best practices

BAML Language

Type System

CLI

Client API

LLM Providers

Syntax

Basic Example

Test Components

Input Types

Primitive Values

Objects

Nested Objects

Arrays

Multi-line Strings

Media Inputs

Images

Audio

PDFs

Videos

Template Strings

Testing Multiple Functions

Assertions

Basic Assertions

Multiple Assertions

Conditional Checks

Dynamic Types

Complete Examples

Email Extraction

Classification

Structured Extraction

Image Analysis

VSCode Integration

Running Tests

Best Practices

Build docs developers (and LLMs) love

BAML Language

Type System

CLI

Client API

LLM Providers

​Syntax

​Basic Example

​Test Components

​Input Types

​Primitive Values

​Objects

​Nested Objects

​Arrays

​Multi-line Strings

​Media Inputs

​Images

​Audio

​PDFs

​Videos

​Template Strings

​Testing Multiple Functions

​Assertions

​Basic Assertions

​Multiple Assertions

​Conditional Checks

​Dynamic Types

​Complete Examples

​Email Extraction

​Classification

​Structured Extraction

​Image Analysis

​VSCode Integration

​Running Tests

​Best Practices

​Related

Build docs developers (and LLMs) love

Syntax

Basic Example

Test Components

Input Types

Primitive Values

Objects

Nested Objects

Arrays

Multi-line Strings

Media Inputs

Images

Audio

PDFs

Videos

Template Strings

Testing Multiple Functions

Assertions

Basic Assertions

Multiple Assertions

Conditional Checks

Dynamic Types

Complete Examples

Email Extraction

Classification

Structured Extraction

Image Analysis

VSCode Integration

Running Tests

Best Practices

Related