Skip to main content

​
Overview

JARVIS integrates with Meta Ray-Ban smart glasses using the Meta Wearables DAT SDK, enabling hands-free photo capture and real-time person identification. The glasses communicate with your iPhone app, which forwards images to the JARVIS backend for processing.
Meta Ray-Ban glasses require the companion iOS app and an iPhone with iOS 15.0 or later.

​
Architecture Flow

Meta Glasses → DAT SDK → iPhone App → JARVIS Backend → Identification Pipeline
    │                          │
    └── Photo Capture         └── Image Upload (Webhook/Telegram)
The system supports multiple capture channels:
  • Option A: Telegram Bot (glasses send photos via Telegram)
  • Option B: HTTP Webhook (POST /api/capture/webhook)
  • Option C: URL Import (POST /api/capture/url)

​
Prerequisites

Hardware

  • Meta Ray-Ban smart glasses
  • iPhone (iOS 15.0+)
  • Active Bluetooth connection

Software

  • Xcode 15+
  • Meta Wearables DAT SDK
  • JARVIS backend running

​
Setup Instructions

1

Install the DAT SDK

Add the Meta Wearables SDK to your Xcode project:
// Package.swift dependencies
dependencies: [
    .package(url: "https://github.com/meta-platforms/meta-wearables-sdk", from: "1.0.0")
]
Import the required modules:
import MWDATCore
#if canImport(MWDATMockDevice)
import MWDATMockDevice
#endif
2

Configure Wearables View Model

Set up the WearablesViewModel to manage device connections:
@MainActor
class WearablesViewModel: ObservableObject {
  @Published var devices: [DeviceIdentifier]
  @Published var registrationState: RegistrationState
  private let wearables: WearablesInterface

  init(wearables: WearablesInterface) {
    self.wearables = wearables
    self.devices = wearables.devices
    self.registrationState = wearables.registrationState

    // Listen for device changes
    Task {
      for await devices in wearables.devicesStream() {
        self.devices = devices
      }
    }
  }

  func connectGlasses() {
    Task {
      try await wearables.startRegistration()
    }
  }
}
View the complete implementation in source/samples/CameraAccess/CameraAccess/ViewModels/WearablesViewModel.swift
3

Monitor Device Compatibility

Add compatibility listeners to ensure glasses firmware is up to date:
private func monitorDeviceCompatibility(devices: [DeviceIdentifier]) {
  for deviceId in devices {
    guard let device = wearables.deviceForIdentifier(deviceId) else { continue }

    let token = device.addCompatibilityListener { compatibility in
      if compatibility == .deviceUpdateRequired {
        // Show update prompt to user
        self.showError("Device requires firmware update")
      }
    }
  }
}
4

Handle Camera Permissions

Request camera access when the user first connects their glasses:
static func requestPermission() async -> Bool {
  let status = AVCaptureDevice.authorizationStatus(for: .video)
  switch status {
  case .authorized:
    return true
  case .notDetermined:
    return await AVCaptureDevice.requestAccess(for: .video)
  default:
    return false
  }
}
5

Send Images to JARVIS

Configure the backend endpoint in your app:
enum Secrets {
  // JARVIS backend URL
  static let backendURL = "https://your-jarvis-backend.com"
  static let webhookToken = "your_webhook_token"
}
When a photo is captured, upload it to the webhook endpoint:
func uploadImage(_ imageData: Data) async throws {
  let base64Image = imageData.base64EncodedString()
  let payload = [
    "image_base64": base64Image,
    "source": "meta_glasses"
  ]

  var request = URLRequest(url: URL(string: "\(Secrets.backendURL)/api/capture/webhook")!)
  request.httpMethod = "POST"
  request.setValue("application/json", forHTTPHeaderField: "Content-Type")
  request.httpBody = try JSONSerialization.data(withJSONObject: payload)

  let (data, _) = try await URLSession.shared.data(for: request)
  // Handle response
}

​
Registration Flow

The glasses must be paired with your iPhone before use:

​
Testing with Mock Devices

During development, use MockDeviceKit for testing without physical glasses: 
#if canImport(MWDATMockDevice)
import MWDATMockDevice

func setupMockDevice() {
  let mockDevice = MockDeviceKit.shared.createDevice(
    name: "Test Glasses",
    serialNumber: "MOCK-001"
  )
  MockDeviceKit.shared.pairDevice(mockDevice)
}
#endif
Mock devices are only available in DEBUG builds and should not be shipped to production.

​
Configuration Options

SettingDescriptionDefault
sessionPresetVideo quality preset.medium
videoRotationAngleFrame orientation90 (portrait)
alwaysDiscardsLateVideoFramesDrop frames if processing is slowtrue
See Camera Setup for detailed camera configuration.

​
Troubleshooting

Symptoms: Registration fails or times outSolutions:
  • Ensure Bluetooth is enabled on iPhone
  • Check that glasses are charged (>20% battery)
  • Restart both glasses and iPhone app
  • Unpair and re-pair in iPhone Settings → Bluetooth
  • Verify glasses firmware is up to date in Meta View app
Symptoms: Images captured but not appearing in JARVISSolutions:
  • Verify backend URL is correct in Secrets.swift
  • Check network connectivity (WiFi or cellular)
  • Confirm webhook token is valid
  • Check backend logs for upload errors: docker logs jarvis-backend
  • Test with manual upload using Postman/curl
Symptoms: “Device requires update” messageSolutions:
  • Open Meta View app on iPhone
  • Navigate to Settings → Device Updates
  • Follow prompts to update glasses firmware
  • Wait for update to complete (may take 5-10 minutes)
  • Reconnect glasses in JARVIS app
Symptoms: Blurry or dark images from glassesSolutions:
  • Clean glasses lenses with microfiber cloth
  • Increase sessionPreset to .high in camera config
  • Ensure adequate lighting conditions
  • Check for lens scratches or damage
  • Adjust frame position on face for better angle

​
Next Steps

Camera Setup

Configure camera settings for both glasses and phone

Telegram Bot

Set up Telegram bot for alternative image capture

​
Reference

Build docs developers (and LLMs) love

Get started for freeTalk to us