​

LLM Accuracy Report

​

Executive Summary

​

Key Findings

​

Retrieval Accuracy

​

By Model Size

​

By Question Type

​

Key Observations

1. Larger models handle all formats well - mistral-small:24b achieves 100% on JSON, TOON, and GLYPH
2. Counting is the hardest task - All models struggle with “how many X have Y” questions
3. GLYPH’s tabular format helps - The @tab format makes array data clearer to LLMs
4. Boolean syntax matters - t/f (GLYPH) > T/F (ZON) for smaller models

​

Generation Quality

​

Results Across All Models

​

Analysis

​

Recommendations

​

Embedding Similarity (RAG)

​

Wire vs Semantic Comparison

​

The Bug

// This causes the "13% penalty" bug
const wireFormat = "{name=Alice age=28}";
const embedding = await embed(wireFormat);
// Result: Lower similarity due to syntax differences

​

Semantic Projection

employees=@tab{id,name,department,salary,remote}
1,"John Doe",Engineering,95000,t

{"employees":[{"id":1,"name":"John Doe","department":"Engineering","salary":95000,"remote":true}]}

employees.[array of 5 items]
employees.[0].id: 1
employees.[0].name: "John Doe"
employees.[0].department: "Engineering"
employees.[0].salary: 95000
employees.[0].remote: true

​

Correct RAG Architecture

data.json → GLYPH encode → blob → CID

data.json → semantic_view() → embed → vector_db
Link: vector_id → CID

"find employees" → embed → vector_search → CID
                → fetch GLYPH → decode → display

​

Size Comparison

​

Benefits of GLYPH for LLM Context

​

Context Window Math

Agent trace with 50 steps

GPT-4-turbo (128K context):
- Can fit: 8 full traces
- Cost: $0.155 per trace (input)

​

Recommendations

​

Architecture Pattern (Hybrid)

​

Use GLYPH Wire Format When

• ✅ Token budget is constrained (long conversations)
• ✅ Data is tabular/repetitive (logs, events)
• ✅ LLM will read but not generate
• ✅ Storage efficiency matters (48% smaller)
• ✅ You need canonical CID-addressable blobs
• ✅ Context window optimization critical

​

Use JSON When

• ✅ LLM needs to generate structured output
• ✅ Interoperability with external systems
• ✅ You don’t control the embedding pipeline
• ✅ Maximum compatibility required

​

Use Semantic Projection When

• ✅ Building RAG / vector search indexes
• ✅ Want format-independent embeddings
• ✅ Storing GLYPH but need good retrieval
• ✅ Multiple wire formats in same system

​

Implementation Example

​

Semantic Projection Function

/**
 * Creates a semantically-rich text view for embedding.
 * Ensures embeddings see same semantics regardless of wire format.
 */
function createSemanticView(data, prefix = '') {
  const lines = [];
  
  if (Array.isArray(data)) {
    lines.push(`${prefix}[array of ${data.length} items]`);
    data.forEach((item, i) => {
      if (typeof item === 'object' && item !== null) {
        lines.push(...createSemanticView(item, `${prefix}[${i}].`));
      } else {
        lines.push(`${prefix}[${i}]: ${formatValue(item)}`);
      }
    });
  } else if (typeof data === 'object' && data !== null) {
    for (const [key, value] of Object.entries(data)) {
      const fullKey = prefix ? `${prefix}${key}` : key;
      if (typeof value === 'object' && value !== null) {
        lines.push(...createSemanticView(value, `${fullKey}.`));
      } else {
        lines.push(`${fullKey}: ${formatValue(value)}`);
      }
    }
  }
  
  return lines;
}

function formatValue(value) {
  if (typeof value === 'boolean') return value ? 'true' : 'false';
  if (typeof value === 'string') return `"${value}"`;
  return String(value);
}

// Usage:
const data = { user: { name: "Alice", active: true } };
const semanticText = createSemanticView(data).join('\n');
// Output:
// user.name: "Alice"
// user.active: true

const embedding = await embed(semanticText);

​

Test Methodology

​

Models Tested

​

Embedding Model

• nomic-embed-text - 768-dim embeddings for semantic similarity

​

Test Categories

1. Direct lookup - “What is the user’s name?”
2. Nested access - “What is the user’s email?”
3. Boolean values - “Is the account verified?”
4. Counting - “How many users have admin role?”
5. Aggregation - “What is the average salary?”

​

Reproduction

cd sjson/benchmark/comparison/js

# Quick test (2 datasets, 3 codecs)
node codec_llm_accuracy_bench.mjs --quick --model=llama3.2:3b

# Full test (all datasets, all codecs)
node codec_llm_accuracy_bench.mjs --model=qwen3:8b

# With different model
node codec_llm_accuracy_bench.mjs --model=mistral-small:24b

​

The Key Insight

​

Related Documentation