Enterprise Policy

LLM Checker’s policy system lets platform and security teams define rules about which models are allowed, what runtimes are approved, what licenses are acceptable, and how violations are handled. Policy rules run inside check and recommend and produce machine-readable compliance reports for CI/CD pipelines and SIEM tools.

What a policy controls

Model allow/deny lists — wildcard patterns for approved or blocked model identifiers
Size and parameter limits — hard caps on model disk size or parameter count
Quantization allowlists — restrict to specific quantization formats (e.g., Q4_K_M only)
Runtime requirements — require specific backends (CUDA, Metal) or local-only execution
License compliance — allowlist approved open-source licenses
Enforcement mode — audit (report-only) or enforce (block on violation)
Exceptions — time-limited per-model overrides with approver attribution
Audit export — JSON, CSV, and SARIF reports for downstream tooling

Policy file structure

A policy is a YAML file, typically named policy.yaml. Generate a fully annotated template with:

llm-checker policy init

The generated template:

version: 1
org: your-org
mode: enforce # audit | enforce

rules:
  models:
    allow:
      - "qwen2.5-coder:*"
      - "llama3.1:*"
    deny:
      - "*uncensored*"
    max_size_gb: 24
    max_params_b: 32
    allowed_quantizations: ["Q4_K_M", "Q5_K_M", "Q8_0"]

  runtime:
    required_backends: ["metal", "cuda"]
    min_ram_gb: 32
    local_only: true

  compliance:
    approved_licenses: ["mit", "apache-2.0", "llama"]

enforcement:
  on_violation: error # warn | error
  exit_code: 3
  allow_exceptions: true

exceptions:
  - model: "deepseek-r1:32b"
    reason: "Approved PoC"
    approver: "[email protected]"
    expires_at: "2026-06-30"

reporting:
  formats: ["json", "csv", "sarif"]

Enforcement modes

audit

Policy violations are recorded and reported but the command exits with code 0. Use audit mode to monitor compliance without blocking developer workflows.

enforce

Blocking violations cause the command to exit with a non-zero code. Use enforce mode in CI gates to prevent non-compliant models from being pulled or deployed.

Set the mode at the top level of your policy file:

mode: enforce  # or: audit

Exit code override

By default, a blocked violation exits with code 1. Override this with enforcement.exit_code to integrate with shell pipelines or CI systems that check specific exit codes:

enforcement:
  on_violation: error
  exit_code: 3

Valid range: 1–255.

Policy commands

`policy init`

Generate a starter policy.yaml in the current directory:

llm-checker policy init

To overwrite an existing file:

llm-checker policy init --force

`policy validate`

Validate a policy file against the v1 schema. Returns non-zero if the file has schema errors:

llm-checker policy validate
llm-checker policy validate --file ./path/to/policy.yaml
llm-checker policy validate --json

The validator checks:

version is an integer ≥ 1
org is a non-empty string
mode is audit or enforce
rules is a well-formed object
enforcement.exit_code is between 1 and 255
reporting.formats contains only json, csv, or sarif
All array fields contain non-empty strings

Both check and recommend accept a --policy flag that applies the policy rules to the candidate model set:

llm-checker check --policy ./policy.yaml
llm-checker check --policy ./policy.yaml --use-case coding --runtime vllm
llm-checker recommend --policy ./policy.yaml --category coding

In audit mode, violations are printed and the command exits 0. In enforce mode, any blocking violation causes a non-zero exit.

--policy takes precedence over --calibrated for routing resolution. See the Calibration Guide for flag precedence details.

Rules reference

`rules.models`

rules.models.allow

string[]

Wildcard patterns for allowed model identifiers. Supports * as a wildcard. If this list is non-empty, any model that does not match at least one pattern fails with MODEL_NOT_ALLOWED.Example: ["qwen2.5-coder:*", "llama3.1:*"]

rules.models.deny

string[]

Wildcard patterns for explicitly blocked model identifiers. A model matching any deny pattern fails with MODEL_DENIED regardless of the allow list.Example: ["*uncensored*"]

rules.models.max_size_gb

number

Maximum disk size in GB. Models exceeding this limit fail with MODEL_TOO_LARGE.

rules.models.max_params_b

number

Maximum parameter count in billions. Models exceeding this limit fail with MODEL_TOO_MANY_PARAMS.

rules.models.allowed_quantizations

string[]

Allowlist of quantization formats. Models with a quantization not in this list fail with QUANTIZATION_NOT_ALLOWED.Example: ["Q4_K_M", "Q5_K_M", "Q8_0"]

`rules.runtime`

rules.runtime.required_backends

string[]

List of required runtime backends. The system backend must match one of these values or the model fails with BACKEND_NOT_ALLOWED.Example: ["metal", "cuda"]

rules.runtime.min_ram_gb

number

Minimum required system RAM in GB. Fails with INSUFFICIENT_RAM if available memory is below this threshold.

rules.runtime.local_only

boolean

default:"false"

When true, models from non-local sources (cloud, remote, hosted) fail with MODEL_NOT_LOCAL.

`rules.compliance`

rules.compliance.approved_licenses

string[]

Allowlist of approved license identifiers. License values are canonicalized before comparison (for example MIT License → mit, Apache 2.0 → apache-2.0). Models without an approved license fail with LICENSE_NOT_APPROVED.Example: ["mit", "apache-2.0", "llama"]

`enforcement`

enforcement.on_violation

string

default:"error"

Behavior when a violation is detected in enforce mode. error blocks and exits non-zero. warn prints the violation but exits 0.

enforcement.exit_code

number

default:"1"

Exit code returned when a blocking violation occurs. Must be an integer between 1 and 255.

enforcement.allow_exceptions

boolean

default:"false"

When true, entries in the exceptions list can suppress violations for specific models. Suppressed violations are recorded in audit reports with status: suppressed.

`exceptions`

Each entry in the exceptions array grants a time-limited override for a specific model:

exceptions:
  - model: "deepseek-r1:32b"
    reason: "Approved PoC"
    approver: "[email protected]"
    expires_at: "2026-06-30"

exceptions[].model

string

Model identifier or wildcard pattern to match. Required.

exceptions[].reason

string

Human-readable justification for the exception. Appears in audit reports.

exceptions[].approver

string

Name or email of the person who approved the exception.

exceptions[].expires_at

string

ISO 8601 date or datetime string. The exception is automatically inactive after this timestamp.

`reporting`

reporting.formats

string[]

Default export formats used when audit export --format all is run. Valid values: json, csv, sarif. If omitted, all three formats are exported.

Audit export

The audit export command evaluates policy outcomes and writes compliance reports without modifying exit behavior:

# Single format
llm-checker audit export \
  --policy ./policy.yaml \
  --command check \
  --format json \
  --out ./reports/check-policy.json

# All formats at once (honors reporting.formats from policy)
llm-checker audit export \
  --policy ./policy.yaml \
  --command check \
  --format all \
  --out-dir ./reports

--command check|recommend selects the candidate source. In enforce mode with blocking violations, reports are written before non-zero exit.

Export formats

JSON

Structured compliance report with full model provenance, violation details, hardware summary, and enforcement metadata. Suitable for pipeline post-processing and dashboards.

llm-checker audit export --policy ./policy.yaml --command check --format json --out ./reports/policy-report.json

Each finding includes:

model_identifier, model_name, source, registry, version, license, digest
violation_code, rule_path, rule_id, severity, message
expected vs actual values
recommendation for remediation
status: active or suppressed (by exception)

CSV

Flat tabular format for SIEM ingestion (Splunk, ELK, DataDog). One row per violation finding, with the same fields as JSON.

llm-checker audit export --policy ./policy.yaml --command check --format csv --out ./reports/policy-report.csv

If no violations exist, a single compliant row is written so downstream parsers always receive a valid file.

SARIF

Static Analysis Results Interchange Format (SARIF 2.1.0) for security scanning tooling. Compatible with GitHub code scanning, VS Code SARIF viewer, and other SARIF consumers.

llm-checker audit export --policy ./policy.yaml --command check --format sarif --out ./reports/policy-report.sarif

Violation severity maps to SARIF levels: high → error, medium → warning, low → note. Suppressed violations are emitted with level note.

SIEM integration examples

# CI artifact (JSON) for post-processing in pipeline jobs
llm-checker audit export --policy ./policy.yaml --command check --format json --out ./reports/policy-report.json

# Flat CSV for SIEM ingestion (Splunk/ELK/DataDog pipelines)
llm-checker audit export --policy ./policy.yaml --command check --format csv --out ./reports/policy-report.csv

# SARIF for security/code-scanning tooling integrations
llm-checker audit export --policy ./policy.yaml --command check --format sarif --out ./reports/policy-report.sarif

GitHub Actions policy gate

Copy this workflow to add an LLM policy gate to your pull request checks:

name: Policy Gate
on: [pull_request]

jobs:
  policy-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: node bin/enhanced_cli.js check --policy ./policy.yaml --runtime ollama --no-verbose
      - if: always()
        run: node bin/enhanced_cli.js audit export --policy ./policy.yaml --command check --format all --runtime ollama --no-verbose --out-dir ./policy-reports
      - if: always()
        uses: actions/upload-artifact@v4
        with:
          name: policy-audit-reports
          path: ./policy-reports

The if: always() conditions ensure audit reports are written and uploaded even when the policy gate fails, giving reviewers access to the full violation details.

Start with mode: audit to observe what would be blocked before switching to mode: enforce. This lets you tune your allow lists and exceptions without interrupting existing workflows.

Provenance fields in reports

check, recommend, and audit export outputs include normalized provenance fields for every model:

Field	Description
`source`	Origin of the model (`local`, `ollama`, `unknown`)
`registry`	Registry the model was fetched from
`version`	Model version or tag
`license`	Canonicalized license identifier
`digest`	Content digest for reproducibility

If a field is unavailable from model metadata, outputs use "unknown" instead of omitting the field, keeping downstream parsers deterministic.

Get Started

Command Reference

Configuration

Guides

Reference

What a policy controls

Policy file structure

Enforcement modes

audit

enforce

Exit code override

Policy commands

`policy init`

`policy validate`

Rules reference

`rules.models`

`rules.runtime`

`rules.compliance`

`enforcement`

`exceptions`

`reporting`

Audit export

Export formats

SIEM integration examples

GitHub Actions policy gate

Provenance fields in reports

Build docs developers (and LLMs) love

Get Started

Command Reference

Configuration

Guides

Reference

​What a policy controls

​Policy file structure

​Enforcement modes

audit

enforce

​Exit code override

​Policy commands

​policy init

​policy validate

​Using --policy with check and recommend

​Rules reference

​rules.models

​rules.runtime

​rules.compliance

​enforcement

​exceptions

​reporting

​Audit export

​Export formats

​SIEM integration examples

​GitHub Actions policy gate

​Provenance fields in reports

Build docs developers (and LLMs) love

What a policy controls

Policy file structure

Enforcement modes

Exit code override

Policy commands

`policy init`

`policy validate`

Using `--policy` with `check` and `recommend`

Rules reference

`rules.models`

`rules.runtime`

`rules.compliance`

`enforcement`

`exceptions`

`reporting`

Audit export

Export formats

SIEM integration examples

GitHub Actions policy gate

Provenance fields in reports