OCR Modes

Overview

Invoice OCR offers three distinct OCR modes, each optimized for different use cases. Understanding when to use each mode helps you balance accuracy, performance, and debugging needs.

Raw Text

Plain text extraction for debugging

Structured

Legacy schema with basic reconciliation

Structured V4

Advanced India GST with multi-pass reconciliation

Raw Text Mode

When to Use

Debugging: Verify what the model actually sees before structuring
Quick inspection: Check if text is being extracted correctly from images/PDFs
Custom parsing: Build your own structured extraction pipeline

API Endpoint

POST /api/ocr

Request Format

Image
PDF
PDF URL

{
  "imageBase64": "data:image/png;base64,...",
  "mimeType": "image/png",
  "model": "openai/gpt-4o-mini" // optional
}

{
  "pdfBase64": "data:application/pdf;base64,...",
  "filename": "invoice.pdf",
  "model": "google/gemini-2.0-flash",
  "plugins": [
    {
      "id": "file-parser",
      "pdf": { "engine": "pdf-text" }
    }
  ]
}

{
  "pdfUrl": "https://example.com/invoice.pdf",
  "model": "openai/gpt-4o-mini"
}

Response

{
  "text": "TAX INVOICE\nInvoice No: INV-2024-001\nDate: 15-01-2024\n..."
}

Example Usage

const response = await fetch('/api/ocr', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    imageBase64: dataUrl,
    mimeType: 'image/jpeg'
  })
});

const { text } = await response.json();
console.log(text); // Raw extracted text

Raw OCR mode preserves line breaks and spacing but does not structure the data. Use this to verify extraction quality before attempting structured parsing.

Structured Mode

When to Use

Simple invoices: Basic line items without complex discounts
Legacy integration: Existing systems using the compact schema
After-tax discounts: Invoices where discounts apply after GST calculation

API Endpoint

POST /api/ocr-structured

Schema

The legacy structured mode returns a compact schema with three main sections:

type InvoiceDoc = {
  voucher: {
    invoice_number: string;
    invoice_date: string;
    invoice_due_date: string;
    invoice_discount: string; // voucher-level discount
    invoice_discount_mode: string; // "before_tax" | "after_tax"
    round_off: string;
    total_invoice_amount: string;
    additional_charges?: AdditionalCharge[];
    reconciliation?: {
      status: "matched" | "unmatched";
      // ... computed totals
    };
  };
  items: Array<{
    name: string;
    price: string; // tax-exclusive
    quantity: string;
    unit: string;
    hsn_sac_code: string;
    tax_rate: string; // total GST %
    discount_rate: string; // item-level discount %
  }>;
  party: {
    party_name: string;
    party_gstin_number: string;
    party_address: string;
    party_city: string;
    party_pincode: string;
    party_pan_number: string;
  };
};

Reconciliation Logic

The structured mode tries multiple assumptions to match the printed total:

Discount Mode Assumptions

Before Tax: Voucher discount reduces taxable amount before GST calculation
After Tax: Voucher discount applies to the final amount after GST

The reconciliation engine tries both modes and picks the one with the smallest difference vs. the printed total.

Charges Interpretation

Additional charges (Freight, Packing, etc.) can be:

Inclusive: amount includes GST → derive taxable and tax amounts
Exclusive: amount is pre-tax → add GST on top

The engine tests both interpretations to find the best match.

Response Example

{
  "voucher": {
    "invoice_number": "AST/1501/B2C25",
    "invoice_date": "14-10-2025",
    "invoice_discount": "0",
    "invoice_discount_mode": "before_tax",
    "round_off": "0.10",
    "total_invoice_amount": "1245",
    "reconciliation": {
      "status": "matched",
      "method": {
        "discount_mode": "before_tax",
        "charges_inclusive": false
      },
      "computed_grand_total": "1245.00",
      "difference": "0.00"
    }
  },
  "items": [
    {
      "name": "Quick Heal-IER 1-Int Essential-1 User",
      "price": "250",
      "quantity": "1",
      "unit": "NOS",
      "hsn_sac_code": "8523",
      "tax_rate": "18",
      "discount_rate": "0"
    }
  ],
  "party": {
    "party_name": "ASTER DISTRIBUTORS",
    "party_gstin_number": "32ABDFA4059P1ZA"
  }
}

Structured mode does not handle:

Multiple sequential discounts (d1%, d2%)
Flat per-unit discounts
Header-level discount allocation by GST bucket
HSN tax table anchoring

For these features, use Structured V4 mode.

Structured V4 Mode

When to Use

India GST compliance: Full HSN/SAC support with CGST/SGST/IGST splits
Complex discounts: Multiple cascading discounts, flat discounts, header discounts
Multi-page PDFs: Process invoices spanning multiple pages with tax tables
High accuracy: Multi-pass reconciliation with printed anchor matching

API Endpoint

POST /api/ocr-structured-v4

Key Features

Multi-Pass Reconciliation

Tests multiple price mode interpretations (WITH_TAX, WITHOUT_TAX) and picks the best match

HSN Tax Table Anchoring

Scales item buckets to match printed per-rate taxable values when HSN table exists

Smart Discount Allocation

Allocates header discounts intelligently across GST rate buckets to match printed GST total

CGST/SGST/IGST Split

Automatically determines intra vs. inter-state tax split based on GSTIN state codes

Schema

The V4 schema provides comprehensive invoice representation:

type V4Doc = {
  doc_level: {
    supplier_name: string;
    supplier_gstin: string;
    invoice_number: string;
    invoice_date: string; // YYYY-MM-DD
    place_of_supply_state_code: string;
    buyer_gstin: string;
    currency: string; // "INR"
  };
  items: Array<{
    name: string;
    hsn: string;
    qty: number;
    uom: string; // normalized UQC code
    rate_ex_tax: number;
    discount: {
      d1_pct: number | null; // first discount %
      d2_pct: number | null; // second discount %
      flat_per_unit: number | null; // flat ₹ per unit
      effective_pct: number; // combined discount %
      amount: number; // total discount ₹
    };
    gst: {
      rate: number; // normalized to GST slab
      cgst: number;
      sgst: number;
      igst: number;
      amount: number;
    };
    totals: {
      line_ex_tax: number;
      line_inc_tax: number;
    };
    raw?: {
      rate_printed: number | null;
      price_mode_detected: "WITH_TAX" | "WITHOUT_TAX";
      // ... other raw OCR hints
    };
    confidence: number;
  }>;
  header_discounts: Array<{
    label: string;
    type: "PERCENT" | "ABSOLUTE";
    value: number;
    order: number; // sequential application
  }>;
  charges: Array<{
    label: string;
    ex_tax: number;
    gst_rate_hint: number | null;
    gst_amount: number;
    inc_tax: number;
    taxable: boolean;
  }>;
  tcs: {
    rate: number;
    amount: number;
    base_used: string;
  };
  round_off: number;
  totals: {
    items_ex_tax: number;
    header_discounts_ex_tax: number;
    charges_ex_tax: number;
    taxable_ex_tax: number;
    gst_total: number;
    grand_total: number;
  };
  printed: {
    taxable_subtotal: number | null;
    gst_total: number | null;
    hsn_tax_table: Array<{
      hsn?: string;
      taxable_value?: number;
      cgst_rate?: number;
      sgst_rate?: number;
      igst_rate?: number;
      cgst_amount?: number;
      sgst_amount?: number;
      igst_amount?: number;
    }>;
    grand_total: number | null;
  };
  reconciliation: {
    error_absolute: number; // difference vs. printed total
    alternates_considered: string[];
    warnings: string[];
  };
  meta: {
    pages_processed: number;
    language: string;
    overall_confidence: number;
  };
};

Request Format

Basic
With Annotations
Custom PDF Engine

{
  "imageBase64": "data:image/jpeg;base64,...",
  "mimeType": "image/jpeg",
  "model": "google/gemini-2.0-flash"
}

{
  "pdfBase64": "data:application/pdf;base64,...",
  "filename": "invoice.pdf",
  "annotations": {
    // Pass-through from previous OpenRouter response
    // to avoid re-parsing costs
  }
}

{
  "pdfBase64": "data:application/pdf;base64,...",
  "plugins": [
    {
      "id": "file-parser",
      "pdf": { "engine": "mistral-ocr" }
    }
  ]
}

Available engines: pdf-text, mistral-ocr, native

Response Example

{
  "doc_level": {
    "supplier_name": "Tech Solutions Pvt Ltd",
    "supplier_gstin": "27AABCT1234M1Z5",
    "invoice_number": "INV-2024-001",
    "invoice_date": "2024-01-15",
    "place_of_supply_state_code": "27",
    "buyer_gstin": "27XYZAB5678N2Z1",
    "currency": "INR"
  },
  "items": [
    {
      "name": "Laptop - Dell Inspiron 15",
      "hsn": "8471",
      "qty": 2,
      "uom": "NOS",
      "rate_ex_tax": 45000.00,
      "discount": {
        "d1_pct": 10,
        "d2_pct": 5,
        "flat_per_unit": null,
        "effective_pct": 14.50,
        "amount": 13050.00
      },
      "gst": {
        "rate": 18,
        "cgst": 13851.00,
        "sgst": 13851.00,
        "igst": 0,
        "amount": 27702.00
      },
      "totals": {
        "line_ex_tax": 76950.00,
        "line_inc_tax": 104652.00
      },
      "raw": {
        "rate_printed": 45000,
        "price_mode_detected": "WITHOUT_TAX"
      },
      "confidence": 0.95
    }
  ],
  "header_discounts": [
    {
      "label": "Trade Discount",
      "type": "PERCENT",
      "value": 2.00,
      "order": 1
    }
  ],
  "charges": [
    {
      "label": "Freight",
      "ex_tax": 500.00,
      "gst_rate_hint": 18,
      "gst_amount": 90.00,
      "inc_tax": 590.00,
      "taxable": true
    }
  ],
  "totals": {
    "items_ex_tax": 76950.00,
    "header_discounts_ex_tax": 1539.00,
    "charges_ex_tax": 500.00,
    "taxable_ex_tax": 75911.00,
    "gst_total": 13664.00,
    "grand_total": 89575.00
  },
  "printed": {
    "taxable_subtotal": 75911.00,
    "gst_total": 13664.00,
    "hsn_tax_table": [
      {
        "hsn": "8471",
        "taxable_value": 75911.00,
        "cgst_rate": 9,
        "sgst_rate": 9,
        "cgst_amount": 6832.00,
        "sgst_amount": 6832.00
      }
    ],
    "grand_total": 89575.00
  },
  "reconciliation": {
    "error_absolute": 0.00,
    "alternates_considered": [
      "as_is:err=0.00,implied_round=0.00,score=0.00",
      "from_printed_with_tax:err=125.50,implied_round=125.50,score=251.00"
    ],
    "warnings": []
  },
  "meta": {
    "pages_processed": 1,
    "language": "en",
    "overall_confidence": 0.92
  }
}

The reconciliation.alternates_considered array shows all hypothesis tests performed. The engine picks the interpretation with the lowest error and most reasonable round-off (≤ ₹1.02).

Mode Comparison

Feature	Raw Text	Structured	Structured V4
Text Extraction	✅	✅	✅
JSON Schema	❌	✅ (Compact)	✅ (Normalized)
Reconciliation	❌	✅ (Basic)	✅ (Multi-pass)
India GST Support	❌	⚠️ (Partial)	✅ (Full)
HSN/SAC Codes	❌	✅	✅ + Table anchoring
CGST/SGST/IGST Split	❌	❌	✅ (Automatic)
Multiple Discounts	❌	❌	✅ (Sequential)
Header Discounts	❌	✅	✅ + Smart allocation
Multi-page PDFs	✅	⚠️	✅ (Optimized)
Confidence Scores	❌	❌	✅
Use Case	Debugging	Simple invoices	Production GST

Best Practices

Start with Raw Text for Unknown Formats

When processing invoices from a new vendor or format:

Call /api/ocr to verify text extraction quality
Check if all important fields are visible in the raw text
If text is missing, try different PDF engines (pdf-text, mistral-ocr, native)
Once satisfied, switch to structured V4 for production

Use Structured V4 for Production

For India GST invoices in production:

Always use /api/ocr-structured-v4
Provides the most accurate reconciliation
Handles edge cases like HSN tax tables, multi-discount structures
Returns confidence scores for validation

Leverage Annotations for PDFs

When processing large PDFs:

First request parses the PDF (slow, costs more)
OpenRouter returns annotations in the response
Subsequent requests with same PDF + annotations skip re-parsing
Significant cost and latency savings for iterative processing

Monitor Reconciliation Errors

Check reconciliation.error_absolute in V4 responses:

≤ ₹0.05: Perfect match, high confidence
₹0.05 - ₹1.00: Acceptable, likely rounding differences
> ₹1.00: Review required, possible extraction issues

Use reconciliation.alternates_considered to debug why specific interpretations were chosen.

Reconciliation Engine

Deep dive into V4 reconciliation logic

PDF Support

Configure PDF parsing engines and optimize performance

Get Started

Core Features

Guides

Configuration

Overview

Raw Text

Structured

Structured V4

Raw Text Mode

When to Use

API Endpoint

Request Format

Response

Example Usage

Structured Mode

When to Use

API Endpoint

Schema

Reconciliation Logic

Response Example

Structured V4 Mode

When to Use

API Endpoint

Key Features

Multi-Pass Reconciliation

HSN Tax Table Anchoring

Smart Discount Allocation

CGST/SGST/IGST Split

Schema

Request Format

Response Example

Mode Comparison

Best Practices

Reconciliation Engine

PDF Support

Build docs developers (and LLMs) love

Get Started

Core Features

Guides

Configuration

​Overview

Raw Text

Structured

Structured V4

​Raw Text Mode

​When to Use

​API Endpoint

​Request Format

​Response

​Example Usage

​Structured Mode

​When to Use

​API Endpoint

​Schema

​Reconciliation Logic

​Response Example

​Structured V4 Mode

​When to Use

​API Endpoint

​Key Features

Multi-Pass Reconciliation

HSN Tax Table Anchoring

Smart Discount Allocation

CGST/SGST/IGST Split

​Schema

​Request Format

​Response Example

​Mode Comparison

​Best Practices

​Related Topics

Reconciliation Engine

PDF Support

Build docs developers (and LLMs) love

Overview

Raw Text Mode

When to Use

API Endpoint

Request Format

Response

Example Usage

Structured Mode

When to Use

API Endpoint

Schema

Reconciliation Logic

Response Example

Structured V4 Mode

When to Use

API Endpoint

Key Features

Schema

Request Format

Response Example

Mode Comparison

Best Practices

Related Topics