Skip to main content

​
Logging Utilities

Utilities for configuring and controlling Verifiers logging output.

​
Overview

The verifiers.utils.logging_utils module provides:
  • Centralized logging configuration
  • JSON logging support
  • Context managers for temporary log level changes
  • Formatted output for training/evaluation

​
Logger Name

LOGGER_NAME = "verifiers"
All Verifiers logging uses the "verifiers" logger. Access it via: 
import logging
logger = logging.getLogger("verifiers")

​
Functions

​
setup_logging

def setup_logging(
    level: str | None = "INFO",
    log_format: str | None = None,
    date_format: str | None = None,
    log_file: str | None = None,
    log_file_level: str | None = None,
    json_logging: bool = False,
) -> None
Configure logging for the Verifiers package.
​
level
str | None
default:"INFO"
Console logging level ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"). If None, disables logging.
​
log_format
str | None
Custom log format string. Defaults to "%(asctime)s - %(name)s - %(levelname)s - %(message)s".
​
date_format
str | None
Date format string. Defaults to "%Y-%m-%d %H:%M:%S".
​
log_file
str | None
Optional path to log file. If specified, logs are written to this file.
​
log_file_level
str | None
Log level for file handler. If None, uses same level as console.
​
json_logging
bool
default:"False"
If True, output logs as JSON (structured logging).
Example: 
import verifiers as vf

# Basic setup
vf.setup_logging(level="INFO")

# Debug mode
vf.setup_logging(level="DEBUG")

# With file logging
vf.setup_logging(
    level="INFO",
    log_file="verifiers.log",
    log_file_level="DEBUG",  # More verbose in file
)

# JSON logging for production
vf.setup_logging(
    level="INFO",
    json_logging=True,
)

# Disable logging
vf.setup_logging(level=None)

​
log_level

@contextmanager
def log_level(level: str | int)
Context manager to temporarily change the Verifiers log level.
​
level
str | int
New log level (e.g., "DEBUG", "WARNING", or logging.DEBUG).
Example: 
import verifiers as vf

vf.setup_logging(level="INFO")

# Temporarily increase verbosity
with vf.log_level("DEBUG"):
    # Detailed logs here
    env.generate(...)

# Back to INFO level

​
quiet_verifiers

def quiet_verifiers() -> ContextManager
Context manager to temporarily silence Verifiers logging (sets level to WARNING). Useful for suppressing INFO and DEBUG messages during specific operations while still showing warnings and errors. Returns: Context manager that temporarily sets log level to WARNING. Example: 
import verifiers as vf

vf.setup_logging(level="INFO")

# Silence Verifiers logs
with vf.quiet_verifiers():
    # Only warnings and errors shown
    env.generate(...)

# Normal logging resumed

def print_prompt_completions_sample(
    prompts: list[Messages],
    completions: list[Messages],
    errors: list[Error | ErrorInfo | None],
    rewards: list[float],
    step: int,
    num_samples: int = 1,
) -> None
Print a formatted table of prompts, completions, and rewards. Displays samples in a Rich table with color-coded columns (yellow for prompts, green for completions, cyan for rewards). Errors are shown in red within the completion column.
​
prompts
list[Messages]
required
List of input prompts (message format).
​
completions
list[Messages]
required
List of model completions (message format).
​
errors
list[Error | ErrorInfo | None]
required
List of errors (or None for successful rollouts).
​
rewards
list[float]
required
List of reward scores. If fewer rewards than prompts, pads with 0.0.
​
step
int
required
Training step number (displayed in panel title).
​
num_samples
int
default:"1"
Number of samples to display from the lists.
Example: 
from verifiers.utils.logging_utils import print_prompt_completions_sample

prompts = [
    [{"role": "user", "content": "What is 2+2?"}],
    [{"role": "user", "content": "Solve x^2=16"}],
]
completions = [
    [{"role": "assistant", "content": "2+2=4"}],
    [{"role": "assistant", "content": "x=4 or x=-4"}],
]
errors = [None, None]
rewards = [1.0, 1.0]

print_prompt_completions_sample(
    prompts=prompts,
    completions=completions,
    errors=errors,
    rewards=rewards,
    step=42,
    num_samples=2,
)
# Displays formatted table with both samples

def print_time(time_s: float) -> str
Format seconds as human-readable time string. Examples:
  • 0.05 → "50ms"
  • 30.0 → "30s"
  • 90.0 → "1m 30s"
  • 3700.0 → "1h 2m"
  • 90000.0 → "1d 1h"
from verifiers.utils.logging_utils import print_time

print(print_time(0.123))   # "123ms"
print(print_time(65))      # "1m 5s"
print(print_time(3661))    # "1h 1m"

​
JSON Logging

​
JsonFormatter

class JsonFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str
JSON formatter for structured logging. Output Format: 
{
    "timestamp": "2024-03-15T10:30:00.000Z",
    "level": "INFO",
    "name": "verifiers.envs.single_turn_env",
    "message": "Starting rollout",
    "module": "single_turn_env",
    "function": "rollout",
    "line": 42,
    "exception": "Traceback..."  // Only if exception
}
Example: 
import verifiers as vf

vf.setup_logging(
    level="INFO",
    json_logging=True,
    log_file="app.log",
)

# All logs are JSON
logger = logging.getLogger("verifiers")
logger.info("Started evaluation")  # JSON output

​
Example Usage

​
Basic Setup

import verifiers as vf

# Configure logging at start of script
vf.setup_logging(level="INFO")

# Use Verifiers normally
env = vf.SingleTurnEnv(...)
results = await env.generate(...)

​
Debug Mode

import verifiers as vf

# Enable debug logging
vf.setup_logging(level="DEBUG")

# See detailed logs
env = vf.SingleTurnEnv(...)
# DEBUG: Loading dataset...
# DEBUG: Initialized rubric with 2 functions
# DEBUG: Starting rollout for example 0
# ...

​
File Logging

import verifiers as vf

vf.setup_logging(
    level="INFO",              # Console: INFO and above
    log_file="verifiers.log",  # Also log to file
    log_file_level="DEBUG",    # File: DEBUG and above
)

# Console shows INFO+, file shows DEBUG+

​
Quiet Mode

import verifiers as vf

vf.setup_logging(level="WARNING")  # Only warnings and errors

# Or use context manager
vf.setup_logging(level="INFO")

with vf.quiet_verifiers():
    # Suppressed during this block
    results = await env.generate(...)

​
Production Logging

import verifiers as vf

vf.setup_logging(
    level="INFO",
    json_logging=True,
    log_file="/var/log/verifiers.log",
)

# Structured JSON logs for parsing

​
Temporary Debug

import verifiers as vf

vf.setup_logging(level="INFO")

with vf.log_level("DEBUG"):
    # Temporarily verbose
    results = await env.generate(num_examples=1)
    # See detailed internal state

# Back to INFO

​
Custom Format

import verifiers as vf

vf.setup_logging(
    level="INFO",
    log_format="[%(levelname)s] %(name)s: %(message)s",
    date_format="%H:%M:%S",
)

# Output: [INFO] verifiers.envs.single_turn_env: Starting rollout

​
Logging Levels

​
DEBUG
Detailed information for diagnosing problems. Very verbose.
​
INFO
Confirmation that things are working as expected. Default level.
​
WARNING
Indication of potential issues (e.g., deprecated features).
​
ERROR
Serious problem that prevented a function from completing.
​
CRITICAL
Very serious error that may cause termination.

​
Best Practices

​
1. Set Up Early

import verifiers as vf

vf.setup_logging(level="INFO")  # First thing in script

# Rest of code...

​
2. Use Appropriate Levels

import logging

logger = logging.getLogger("verifiers")

logger.debug("Detailed state: %s", state)       # Development only
logger.info("Processing example %d", i)         # Progress info
logger.warning("Deprecated API used")           # Important notices
logger.error("Failed to load dataset: %s", e)  # Problems

​
3. Environment-Specific Config

import os
import verifiers as vf

# Development: verbose
if os.getenv("ENV") == "dev":
    vf.setup_logging(level="DEBUG")
else:
    # Production: structured
    vf.setup_logging(
        level="INFO",
        json_logging=True,
        log_file="/var/log/app.log",
    )

​
4. Quiet External Libraries

import logging
import verifiers as vf

vf.setup_logging(level="INFO")

# Silence noisy libraries
logging.getLogger("httpx").setLevel(logging.WARNING)
logging.getLogger("httpcore").setLevel(logging.WARNING)

​
See Also

Build docs developers (and LLMs) love

Get started for freeTalk to us