Skip to main content
Refs are how you reference UI elements across commands. A snapshot assigns refs like @n1, @n2, @n3 to elements in the Accessibility tree. You then use these refs in subsequent commands to click, type, or inspect specific elements.

​
What are refs?

A ref (reference) is a short identifier like @n1, @n2, etc. that points to a specific UI element. Think of refs as temporary “handles” for elements discovered during a snapshot. 
# Take a snapshot
$ agent-native snapshot Safari --interactive
Snapshot: Safari (pid 1234) -- 12 elements
---------------------------------------------
  AXWindow "Safari" [ref=n1]
    AXButton "Back" [AXPress] [ref=n2]
    AXButton "Forward" [AXPress] [ref=n3]
    AXTextField [ref=n4]
    AXButton "Go" [AXPress] [ref=n5]

# Click using a ref
$ agent-native click @n2
OK Clicked: AXButton title="Back"
Refs are numbered sequentially starting from n1 in each snapshot. The @ prefix distinguishes refs from app names.

​
How snapshots work

The snapshot command walks the Accessibility tree and assigns refs to elements. From SnapshotCommand.swift:5-133:
1

Walk the tree

Traverse the app’s Accessibility tree to the specified depth:
let element = AXEngine.appElement(pid: pid)
let tree = AXEngine.walkTree(element, maxDepth: depth)
2

Filter elements

Optionally filter to only interactive elements with --interactive, or use --compact to skip empty structural nodes:
// SnapshotCommand.swift:26-33
static let interactiveRoles: Set<String> = [
    "AXButton", "AXTextField", "AXCheckBox",
    "AXRadioButton", "AXPopUpButton", "AXLink",
    "AXMenuItem", "AXTab", "AXSearchField", ...
]
3

Assign refs

Number each element sequentially:
var refCounter = 0
for (node, depth) in tree {
    refCounter += 1
    let ref = "n\(refCounter)"
    // Store for later resolution
}
4

Save to RefStore

Persist ref → element mappings to a temp JSON file:
RefStore.save(refEntries)

​
The RefStore

Refs are persisted to disk at /tmp/agent-native-refs.json so they survive between CLI invocations. Each command runs as a separate process, so refs must be saved externally. From RefStore.swift:4-35: 
struct RefEntry: Codable {
    let ref: String           // "n1", "n2", etc.
    let pid: Int32            // Process ID of the app
    let role: String          // AXButton, AXTextField, etc.
    let title: String?        // Button text, window title
    let label: String?        // Accessibility label
    let identifier: String?   // Programmatic identifier
    let pathHint: String      // XPath-like path in the tree
}

private static let storePath: URL = {
    let tmp = FileManager.default.temporaryDirectory
    return tmp.appendingPathComponent("agent-native-refs.json")
}()

static func save(_ entries: [RefEntry]) { ... }
static func load() -> [RefEntry] { ... }
The RefStore contains only the metadata needed to re-find elements, not live AXUIElement pointers (which can’t be serialized).

​
Ref resolution

When you use a ref like @n5, agent-native must resolve it back to a live AXUIElement. This happens by re-searching the tree using the stored attributes. From RefStore.swift:37-74: 
static func resolve(ref: String) throws -> (element: AXUIElement, node: AXNode) {
    let entries = load()
    guard let entry = entries.first(where: { $0.ref == cleanRef }) else {
        throw AXError.elementNotFound("Unknown ref: @\(cleanRef). Run `snapshot` first.")
    }
    
    let appElement = AXEngine.appElement(pid: entry.pid)
    let results = AXEngine.findElements(
        root: appElement,
        role: entry.role,
        title: entry.title,
        label: entry.label,
        identifier: entry.identifier,
        maxDepth: 15
    )
    
    // Return first exact match
    if let match = results.first(where: { /* exact attribute match */ }) {
        return match
    }
    
    throw AXError.elementNotFound(
        "Could not re-resolve @\(cleanRef). The UI may have changed -- run `snapshot` again.")
}
1

Load ref metadata

Read the entry from agent-native-refs.json.
2

Search the live tree

Use AXEngine.findElements() to search for elements matching the stored role, title, label, and identifier.
3

Match exactly

Return the element that exactly matches all stored attributes.
4

Error if not found

If the element no longer exists or has changed, throw elementNotFound.

​
Ref lifecycle and invalidation

Refs become invalid when:
  1. The UI changes - If a button is removed or its text changes, the ref can’t be resolved
  2. The app restarts - PIDs change, invalidating all refs for that app
  3. A new snapshot is taken - Each snapshot command overwrites the entire RefStore
Refs are ephemeral. They’re valid for the current UI state only. If your interaction changes the UI significantly (e.g., navigating to a new screen), take a fresh snapshot.

​
When refs fail

$ agent-native click @n8
Error: Could not re-resolve @n8. The UI may have changed -- run `snapshot` again.
This means the element with ref @n8 no longer exists or has different attributes. Solution: take a new snapshot.

​
Interactive vs full snapshots

You can control what gets refs:

​
Interactive snapshots

Use --interactive to only assign refs to elements you can interact with: 
$ agent-native snapshot Safari --interactive
This filters to interactiveRoles (buttons, text fields, checkboxes, links, etc.) and skips structural elements like groups and static text.
Use --interactive for agent workflows. It reduces noise and makes snapshots easier to parse.

​
Compact snapshots

Use --compact to remove empty structural elements: 
$ agent-native snapshot Safari --compact
From SnapshotCommand.swift:69-74, this skips elements with:
  • No title, label, or value
  • No actions
  • Has children (i.e., is just a container)

​
Full snapshots

By default, every element gets a ref. This is useful for debugging or when you need to inspect structural elements: 
$ agent-native snapshot Safari

​
Example workflow

1

Take initial snapshot

$ agent-native snapshot Safari --interactive
Snapshot: Safari (pid 1234) -- 8 elements
AXWindow "Safari" [ref=n1]
  AXTextField [ref=n2]
  AXButton "Go" [AXPress] [ref=n3]
2

Interact with refs

$ agent-native click @n2
OK Clicked: AXTextField

$ agent-native type @n2 "example.com"
OK Typed 11 chars into AXTextField

$ agent-native click @n3
OK Clicked: AXButton title="Go"
3

Re-snapshot after UI changes

After the page loads, the UI is different. Take a new snapshot:
$ agent-native snapshot Safari --interactive
Snapshot: Safari (pid 1234) -- 15 elements
# Now @n1, @n2, etc. refer to the NEW UI state

​
See also

Accessibility tree

Understanding the tree that refs point into

Workflow

The snapshot → interact → re-snapshot pattern

Build docs developers (and LLMs) love

Get started for freeTalk to us