Hardware Acceleration

​

Overview

• Android: OpenCL (GPU) for LLMs, Qualcomm QNN (NPU) for image generation
• iOS: Metal (GPU) for LLMs, Apple Neural Engine (ANE) for image generation

​

LLM Inference Acceleration

​

OpenCL GPU Offloading (Android)

llama.cpp (C++)
  ↓
ggml OpenCL backend
  ↓
OpenCL Runtime
  ↓
Qualcomm Adreno GPU Driver
  ↓
Adreno GPU (e.g., Adreno 740 on Snapdragon 8 Gen 2)

// User configures GPU layers in model settings
const modelParams = {
  nGpuLayers: 33, // Offload first 33 transformer layers to GPU
  contextSize: 2048,
  nBatch: 256,
  // ...
}

await llmService.initContext(modelParams);

// src/services/llmHelpers.ts
function getGpuLayersForDevice(
  totalMemoryBytes: number,
  requestedLayers: number
): number {
  if (totalMemoryBytes <= 4 * 1024 * 1024 * 1024) {
    // ≤4GB devices: disable GPU to prevent abort()
    return 0;
  }
  return requestedLayers;
}

// src/services/llm.ts
if (nGpuLayers > 0 && Platform.OS === 'android') {
  flashAttn = false; // Auto-disable flash attention
  console.warn('Flash attention disabled: incompatible with OpenCL backend');
}

​

Metal GPU Acceleration (iOS)

llama.cpp (C++)
  ↓
ggml Metal backend
  ↓
Metal API
  ↓
Apple GPU (e.g., A17 Pro 6-core GPU)

const modelParams = {
  nGpuLayers: 99, // Offload all layers to GPU (recommended)
  useMetalGpu: true,
  contextSize: 2048,
  // ...
}

await llmService.initContext(modelParams);

// src/services/llmHelpers.ts (iOS)
if (totalMemoryBytes <= 4 * 1024 * 1024 * 1024) {
  // ≤4GB devices (iPhone XS, iPhone 8): disable Metal
  return 0;
}

• Zero-copy tensor transfers (no CPU↔GPU memcpy)
• Lower latency for small batches
• Higher effective memory bandwidth

// src/services/llm.ts - initializeMultimodal()
const useGpuForClip = totalMemoryBytes > 4 * 1024 * 1024 * 1024;

await llmService.initMultimodal({
  mmProjPath: model.mmProjPath,
  useGpu: useGpuForClip,
});

​

Image Generation Acceleration

​

Qualcomm QNN NPU (Android)

local-dream (C++)
  ↓
QNN Backend (libQnnHtp.so)
  ↓
FastRPC (IPC to DSP)
  ↓
Hexagon cDSP (Compute DSP)
  ↓
HTP Accelerator (Hexagon Tensor Processor)

• unet.bin — UNet denoising network (HTP-optimized)
• vae_decoder.bin — VAE decoder (HTP-optimized)
• clip.bin or clip.mnn — Text encoder (CPU or MNN)

// LocalDreamModule.kt - buildCommand()
val command = mutableListOf(
    executable.absolutePath,
    "--backend", File(runtimeDir, "libQnnHtp.so").absolutePath,
    "--system_library", File(runtimeDir, "libQnnSystem.so").absolutePath,
    // ...
)

• Detects HTP version via /sys/devices/soc0/soc_id
• Loads corresponding libQnnHtpV73.so, libQnnHtpV75.so, etc.
• Falls back to oldest compatible version if exact match unavailable

// LocalDreamModule.kt:191-193
env["LD_LIBRARY_PATH"] = systemLibPaths.joinToString(":")
env["DSP_LIBRARY_PATH"] = runtimeDir.absolutePath
env["ADSP_LIBRARY_PATH"] = runtimeDir.absolutePath

• DSP_LIBRARY_PATH — Hexagon DSP firmware location
• ADSP_LIBRARY_PATH — Audio DSP library path (required for some SoCs)

1. Analyze compute graph
2. Compile ops for HTP
3. Cache compiled binaries to /data/local/tmp/qnn_cache/

​

Apple Neural Engine (iOS)

Core ML (Swift)
  ↓
ANE Compiler (on-device JIT)
  ↓
ANE Runtime
  ↓
Neural Engine Hardware

// CoreMLDiffusionModule.swift
config.computeUnits = .cpuAndNeuralEngine

• Op is convolution, matrix multiply, or supported activation
• Tensor shapes are ANE-compatible (batch size 1, certain dimension alignments)
• Precision is fp16 or int8

• Custom ops (e.g., some schedulers)
• Unsupported tensor shapes
• Dynamic shapes (rare in Stable Diffusion)

​

RAM-Based Limitations

​

Pre-Load Memory Checks

// src/services/activeModelService.ts
function estimateModelMemory(model: DownloadedModel, type: 'text' | 'image'): number {
  const fileSize = model.fileSize;
  
  if (type === 'text') {
    // Text models: file size × 1.5 (KV cache + activations)
    const baseRAM = fileSize * 1.5;
    
    if (model.isVisionModel && model.mmProjFileSize) {
      // Vision models: add mmproj overhead
      return baseRAM + (model.mmProjFileSize * 1.5);
    }
    
    return baseRAM;
  } else {
    // Image models: file size × 1.8 (MNN/QNN runtime + intermediate tensors)
    return fileSize * 1.8;
  }
}

const deviceRAM = await hardware.getTotalMemory();
const budget = deviceRAM * 0.60; // 60% of total RAM

if (estimatedRAM > budget) {
  throw new Error('Insufficient RAM');
}

if (estimatedRAM > deviceRAM * 0.50 && estimatedRAM <= deviceRAM * 0.60) {
  // Show yellow warning in UI
  return {
    canLoad: true,
    severity: 'warning',
    message: 'Model will use significant RAM. Close other apps before loading.'
  };
}

if (estimatedRAM > deviceRAM * 0.60) {
  // Block load entirely
  return {
    canLoad: false,
    severity: 'error',
    message: `Cannot load ${model.name} (~${estimatedGB}GB required) - would exceed device safe limit of ${budgetGB}GB. Unload current model or choose smaller.`
  };
}

​

Runtime Context Caps

// src/services/llmHelpers.ts
function getMaxContextForDevice(totalMemoryBytes: number): number {
  const totalGB = totalMemoryBytes / (1024 ** 3);
  
  if (totalGB <= 4) return 2048;
  if (totalGB <= 6) return 2048;
  if (totalGB <= 8) return 4096;
  return 8192;
}

// src/services/llm.ts - initWithAutoContext()
const requestedContext = userSettings.contextSize;
const cappedContext = Math.min(requestedContext, getMaxContextForDevice(deviceRAM));

if (cappedContext < requestedContext) {
  console.warn(`Context capped to ${cappedContext} (device limit)`);
}

​

Overload Prevention in ARCHITECTURE.md

RAM-Aware Runtime Safeguards: On low-RAM devices (e.g. iPhone XS with 4GB), llama.cpp and CLIP can call abort() during Metal GPU allocation, which is a POSIX signal that bypasses JavaScript try/catch and kills the app instantly. To prevent this, initWithAutoContext() applies device-RAM-based caps before any native call:

Device RAM	GPU Layers	Context Cap	CLIP GPU
≤4GB	0 (CPU-only)	2048	Off
4-6GB	Requested	2048	On
6-8GB	Requested	4096	On
>8GB	Requested	8192	On

// src/services/llmHelpers.ts
export function getGpuLayersForDevice(
  totalMemoryBytes: number,
  requestedLayers: number
): number {
  if (totalMemoryBytes <= 4 * 1024 * 1024 * 1024) {
    return 0; // Disable Metal/OpenCL on ≤4GB devices
  }
  return requestedLayers;
}

​

Performance Benchmarks

​

Text Generation (Tokens/Sec)

​

Image Generation (Seconds)

​

Vision Inference (Seconds)

​

Best Practices

​

For LLM Inference

1. Start conservative: Begin with 0 GPU layers, incrementally increase by 8-10 layers while monitoring stability
2. Monitor temperature: Sustained generation can trigger thermal throttling on mobile devices
3. Reduce context on ≤4GB devices: Use 2048 max context to prevent OOM
4. Disable flash attention with GPU: Auto-disabled on Android, but verify on custom builds

​

For Image Generation

1. Prefer NPU/ANE over CPU: 2-3x faster with lower power consumption
2. Use fp16 models on iOS: Palettized models are 2x slower due to dequantization overhead
3. Cache warmup on first run: Warn users first generation will be slow (QNN cache building)
4. Unload text model before image generation: Prevents RAM contention on 4-6GB devices

​

For Memory Management

1. Respect 60% RAM budget: Prevents OS kills
2. Warn at 50%: Give users option to close background apps
3. Unload before switching models: Releases GPU/ANE resources
4. Monitor with DeviceInfoScreen: Real-time RAM usage in settings

​

Debugging Acceleration Issues

​

OpenCL (Android)

# Enable OpenCL logging
adb shell setprop debug.ocl.log 1
adb logcat | grep -i opencl

• CL_OUT_OF_HOST_MEMORY — Reduce GPU layers
• CL_DEVICE_NOT_FOUND — GPU not detected, use CPU
• SIGSEGV during clBuildProgram — OpenCL compiler crash, disable GPU

​

Metal (iOS)

# Enable Metal validation
Product → Scheme → Edit Scheme → Run → Diagnostics
☑ Metal API Validation

• MTLBuffer allocation failed — Reduce context size or GPU layers
• abort() in mtl_backend_init — ≤4GB device, GPU layers should be 0

​

QNN (Android)

# Check DSP access
adb shell cat /sys/devices/soc0/soc_id
adb shell ls -l /dev/subsys_*

# QNN logs
adb logcat | grep -i qnn

• Failed to create backend — SELinux blocking DSP, use MNN fallback
• libQnnHtp.so not found — Runtime libs not extracted, check prepareRuntimeDir()

​

ANE (iOS)

1. Profile → Core ML
2. Run image generation
3. Check “ANE Utilization” track
4. Should be over 80% during UNet inference

​

References

• llama.cpp Backends: https://github.com/ggerganov/llama.cpp/tree/master/docs/backend
• Qualcomm AI Engine: https://www.qualcomm.com/developer/software/qualcomm-ai-engine-direct
• Apple Neural Engine: https://github.com/hollance/neural-engine (reverse engineering)
• Core ML Performance: https://developer.apple.com/documentation/coreml/optimizing_your_model_for_the_neural_engine
• Metal Performance Shaders: https://developer.apple.com/documentation/metalperformanceshaders