Skip to main content
Policy files give teams explicit control over which models are permitted, which runtimes are allowed, and how violations are surfaced — either as warnings or as blocking errors. All policy functionality is available through the policy subcommands and the --policy flag on check and recommend.

​
What Policy Files Control

A policy file is a YAML document that governs:
  • Model allowlists — which models or model families are permitted or blocked
  • Runtime restrictions — which inference runtimes (ollama, vllm, mlx, llama.cpp) are approved
  • Enforcement mode — whether violations log a warning (audit) or fail the command (enforce)
  • Exit code — the non-zero exit code returned in enforce mode (configurable via enforcement.exit_code)
  • Reporting formats — which export formats (json, csv, sarif) are produced by audit export
  • Provenance requirements — minimum metadata fields that must be present on any candidate model

​
Enforcement Modes

enforcement:
  mode: audit
  # exit_code is irrelevant in audit mode — command always exits 0
ModeBehaviourExit code
auditViolations are reported in output but the command exits 0Always 0
enforceBlocking violations cause a non-zero exitDefault 1, overridable via exit_code
In enforce mode, report artifacts are still written to disk before the process exits non-zero. This ensures compliance evidence is always available even when a pipeline job fails.

​
Creating a Policy File

1

Generate a policy template

policy init writes a fully commented policy.yaml template to your working directory:
llm-checker policy init
Open policy.yaml and edit the allowlists, restrictions, and enforcement settings to match your organization’s requirements.
2

Validate the policy file

policy validate checks the file against the v1 schema and returns a non-zero exit code if the file is invalid:
llm-checker policy validate
To validate a file at a specific path:
llm-checker policy validate --policy ./path/to/policy.yaml
A passing validation produces structured output listing each rule and its status. Fix any schema errors before using the file in check or recommend.
3

Apply the policy to check and recommend

Pass --policy <file> to check or recommend to enable policy evaluation:
llm-checker check --policy ./policy.yaml
llm-checker check --policy ./policy.yaml --use-case coding --runtime vllm
llm-checker recommend --policy ./policy.yaml --category coding
Policy outcomes are embedded in the command output alongside the usual model rankings. In enforce mode, any blocking violation halts the command with a non-zero exit.
4

Export compliance reports

audit export runs the policy evaluation and writes machine-readable reports:
# Single format
llm-checker audit export \
  --policy ./policy.yaml \
  --command check \
  --format json \
  --out ./reports/check-policy.json

# All configured formats at once
llm-checker audit export \
  --policy ./policy.yaml \
  --command check \
  --format all \
  --out-dir ./reports
--command check|recommend selects the candidate source. --format all honours reporting.formats in your policy file and falls back to json,csv,sarif.

​
Audit Export Formats

llm-checker audit export \
  --policy ./policy.yaml \
  --command check \
  --format json \
  --out ./reports/policy-report.json
FormatBest for
jsonPost-processing in pipeline jobs, dashboards, custom tooling
csvSIEM ingestion — Splunk, ELK, DataDog pipelines
sarifSecurity and code-scanning tool integrations (GitHub Advanced Security, etc.)

​
Provenance Fields in Reports

All check, recommend, and audit export outputs include normalized model provenance fields. These fields are present even when the underlying model metadata does not supply them — missing values are represented as "unknown" rather than being omitted, keeping downstream parsers deterministic.
FieldDescription
sourceWhere the model entry originated (dynamic catalog, local install, curated fallback)
registryThe model registry the entry was sourced from (e.g. ollama)
versionModel version or tag
licenseCanonicalized SPDX-style license identifier
digestContent digest / hash when available

​
License Canonicalization

License values are normalized before policy checks so that free-text license strings from model metadata match policy rules reliably:
Raw valueCanonical form
MIT Licensemit
Apache 2.0apache-2.0
This means a policy allowlist entry of apache-2.0 will match models tagged Apache 2.0, Apache-2.0, or similar variants.

Build docs developers (and LLMs) love

Get started for freeTalk to us