Development - Agent Desktop

Quick Start

Prerequisites

Rust 1.78+

macOS 13.0+

Currently required for Phase 1 development

Clone and Build

git clone https://github.com/lahfir/agent-desktop
cd agent-desktop
cargo build --release

The binary will be at ./target/release/agent-desktop (under 15MB optimized).

Common Commands

cargo build

Running the Binary

After building, test with:

# Snapshot Finder with interactive refs
./target/release/agent-desktop snapshot --app Finder -i

# Check permissions
./target/release/agent-desktop permissions

# Show help
./target/release/agent-desktop --help

Grant Accessibility permission before running: System Settings > Privacy & Security > Accessibility → add your terminal app.

Project Structure

Workspace Members

Crate	Path	Purpose
`agent-desktop-core`	`crates/core/`	Platform-agnostic logic, `PlatformAdapter` trait
`agent-desktop-macos`	`crates/macos/`	macOS implementation (Phase 1)
`agent-desktop-windows`	`crates/windows/`	Windows stub (Phase 2)
`agent-desktop-linux`	`crates/linux/`	Linux stub (Phase 2)
`agent-desktop` (binary)	`src/`	CLI entry point, dispatch, JSON envelope

Shared Dependencies

Workspace-level dependencies in root Cargo.toml:

[workspace.dependencies]
clap               = { version = "4.5", features = ["derive"] }
serde              = { version = "1.0", features = ["derive"] }
serde_json         = "1.0"
thiserror          = "2.0"
tracing            = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
base64             = "0.22"
rustc-hash         = "2.1"
agent-desktop-core = { path = "crates/core" }

Coding Standards

File Rules

400 LOC hard limit per file. If approaching 400 lines, split by responsibility. No exceptions.

One struct/enum per file for domain types
One command per file under commands/
No inline comments — code must be self-documenting
Only doc-comments (///) on public items when name alone is insufficient
No God objects — max 7 fields per struct, max 5 params per function

Naming Conventions

Element	Convention	Example
Crate names	`agent-desktop-{name}`	`agent-desktop-core`
Module files	`snake_case`, singular	`snapshot.rs`
Structs	PascalCase, descriptive noun	`SnapshotEngine`
Traits	PascalCase, adjective	`PlatformAdapter`
Enums	PascalCase, variants PascalCase	`Action::Click`
Functions	`snake_case`, verb-first	`build_tree()`
Constants	`SCREAMING_SNAKE_CASE`	`MAX_TREE_DEPTH`
CLI flags	kebab-case	`--max-depth`
Ref IDs	`@e{n}` sequential	`@e1`, `@e14`

Error Handling

Zero unwrap()

All Results propagated with ? or matched explicitly. Panics are test-only.

Structured Errors

Every error has: ErrorCode, message, suggestion, platform_detail

Error codes:

PERM_DENIED, ELEMENT_NOT_FOUND, APP_NOT_FOUND, ACTION_FAILED,
ACTION_NOT_SUPPORTED, STALE_REF, WINDOW_NOT_FOUND,
PLATFORM_NOT_SUPPORTED, TIMEOUT, INVALID_ARGS, INTERNAL

Exit codes:

0 — success
1 — structured error (JSON on stdout)
2 — argument/parse error

Adding a New Command

Create command file

Add crates/core/src/commands/my_command.rs with an execute() function:

use crate::{PlatformAdapter, AppError};
use serde_json::Value;

pub fn execute(args: MyCommandArgs, adapter: &dyn PlatformAdapter) -> Result<Value, AppError> {
    // implementation
}

Add to crates/core/src/commands/mod.rs:

pub mod my_command;

Add CLI variant

Update src/cli.rs:

#[derive(Subcommand)]
pub enum Commands {
    // ...
    MyCommand(MyCommandArgs),
}

Add dispatch arm

Update src/dispatch.rs:

match cmd {
    // ...
    Commands::MyCommand(args) => commands::my_command::execute(args, adapter),
}

(Optional) Add Action variant

If the command needs a new Action type, add to crates/core/src/action.rs:

pub enum Action {
    // ...
    MyAction(String),
}

(Optional) Add adapter method

If the command needs a new platform capability, add to PlatformAdapter trait with default:

fn my_operation(&self) -> Result<(), AdapterError> {
    Err(AdapterError::not_supported())
}

No existing files are modified beyond these registration points. This is enforced via code review.

Testing Strategy

Unit Tests (Core)

Tests in crates/core/src/:

AccessibilityNode ser/de roundtrips
Ref allocator only assigns interactive roles
SnapshotEngine filtering logic
Error serialization format
MockAdapter for isolated testing

cargo test --lib -p agent-desktop-core

Platform Tests (macOS)

Tests in crates/macos/src/:

Tree traversal correctness
Role mapping coverage
Action execution
Permission detection

cargo test --lib -p agent-desktop-macos

Golden Fixtures

tests/fixtures/ contains real snapshots from Finder, TextEdit, etc. for regression testing:

cargo test --test fixtures

Integration Tests (macOS CI)

tests/integration/ runs full command scenarios:

Snapshot Finder → non-empty tree with refs
Click button in test app → verify action succeeded
Type text into TextEdit via ref → verify content changed
Clipboard get/set roundtrip
Permission denied scenario → correct error code

cargo test --test integration

Integration tests require Accessibility permission and running macOS 13.0+.

Linting & Formatting

Clippy Rules

Project uses strict linting (clippy.toml):

cargo clippy --all-targets -- -D warnings

Zero warnings allowed. CI enforces this.

Formatting

Standard Rust formatting:

# Check formatting
cargo fmt --all -- --check

# Auto-format
cargo fmt --all

Dependency Verification

CI enforces dependency inversion:

# Core must NOT depend on platform crates
cargo tree -p agent-desktop-core

If output contains macos, windows, or linux, the build fails.

Build Profiles

Debug (Default)

Fast compilation, large binary, no optimizations:

cargo build
# Binary: ./target/debug/agent-desktop (~50MB)

Release

Optimized for size (under 15MB target):

cargo build --release
# Binary: ./target/release/agent-desktop (~12MB)

Release profile settings:

[profile.release]
opt-level = "z"           # optimize for size
lto = true                # link-time optimization
codegen-units = 1         # single codegen unit
strip = true              # strip symbols
panic = "abort"           # smaller panic handler

Platform Crate Guidelines

Folder Placement

All platform crates follow identical structure:

crates/{macos,windows,linux}/src/
├── lib.rs              # mod declarations + re-exports only
├── adapter.rs          # PlatformAdapter trait impl
├── tree/               # Tree reading & traversal
├── actions/            # Element interactions
├── input/              # Keyboard, mouse, clipboard
└── system/             # App lifecycle, windows, permissions

Placement rules:

Tree reading → tree/
Element interaction → actions/
Raw OS input → input/
App lifecycle → system/

Safety Rules (macOS)

For AXElement wrapper:

Inner field must be pub(crate) not pub
Clone must call CFRetain
Drop must call CFRelease
No direct pointer extraction to prevent double-free

CI Requirements

GitHub Actions runs on every PR:

Dependency isolation

cargo tree -p agent-desktop-core must not contain platform crates

Lint check

cargo clippy --all-targets -- -D warnings must pass with zero warnings

Full test suite

cargo test --workspace must pass

Binary size check

Release binary must be under 15MB

Git Workflow

Conventional Commits

All commits must use a type prefix:

feat: — new feature (triggers minor version bump)
fix: — bug fix (triggers patch version bump)
feat!: or BREAKING CHANGE: — breaking change (major bump)
docs: — documentation only
refactor: — code change that neither fixes nor adds
chore: — maintenance tasks, dependencies
ci: — CI/CD changes
test: — adding or fixing tests

Format: type: concise imperative description (lowercase type, no capital after colon) Examples:

feat: add scroll-to command
fix: prevent stale ref on window resize
ci: add binary size check
docs: update architecture diagram

Focus on “why” not “what”. The diff shows what changed; the message explains why.

Performance Tips

macOS Tree Traversal

Batch Attributes

Use AXUIElementCopyMultipleAttributeValues for 3-5x speedup vs. individual calls

Max Depth

Default --max-depth 10 prevents infinite loops. Adjust if needed.

Ancestor Paths

Use ancestor-path sets, not global visited sets (macOS reuses pointers)

Interactive-Only

Use -i flag to skip static elements and reduce tree size by ~70%

Debugging

Enable Tracing

RUST_LOG=debug ./target/debug/agent-desktop snapshot --app Finder -i

Levels: error, warn, info, debug, trace

Inspect JSON Output

agent-desktop snapshot -i | jq .

Check RefMap

Ref storage:

cat ~/.agent-desktop/last_refmap.json | jq .

Contributing

Fork and clone

Fork the repo and clone your fork locally

Create feature branch

git checkout -b feat/my-feature

Make changes

Follow coding standards, add tests, run clippy

Commit with conventional format

git commit -m "feat: add new command"

Push and create PR

Push to your fork and open a pull request

Next Steps

Architecture

Understand the workspace structure and design patterns

Troubleshooting

Common issues and solutions

Get Started

Core Concepts

Command Categories

Guides

Advanced

​Quick Start

​Prerequisites

Rust 1.78+

macOS 13.0+

​Clone and Build

​Common Commands

​Running the Binary

​Project Structure

​Workspace Members

​Shared Dependencies

​Coding Standards

​File Rules

​Naming Conventions

​Error Handling

Zero unwrap()

Structured Errors

​Adding a New Command

​Testing Strategy

​Unit Tests (Core)

​Platform Tests (macOS)

​Golden Fixtures

​Integration Tests (macOS CI)

​Linting & Formatting

​Clippy Rules

​Formatting

​Dependency Verification

​Build Profiles

​Debug (Default)

​Release

​Platform Crate Guidelines

​Folder Placement

​Safety Rules (macOS)

​CI Requirements

​Git Workflow

​Conventional Commits

​Performance Tips

​macOS Tree Traversal

Batch Attributes

Max Depth

Ancestor Paths

Interactive-Only

​Debugging

​Enable Tracing

​Inspect JSON Output

​Check RefMap

​Contributing

​Next Steps

Architecture

Troubleshooting

Build docs developers (and LLMs) love

Quick Start

Prerequisites

Clone and Build

Common Commands

Running the Binary

Project Structure

Workspace Members

Shared Dependencies

Coding Standards

File Rules

Naming Conventions

Error Handling

Adding a New Command

Testing Strategy

Unit Tests (Core)

Platform Tests (macOS)

Golden Fixtures

Integration Tests (macOS CI)

Linting & Formatting

Clippy Rules

Formatting

Dependency Verification

Build Profiles

Debug (Default)

Release

Platform Crate Guidelines

Folder Placement

Safety Rules (macOS)

CI Requirements

Git Workflow

Conventional Commits

Performance Tips

macOS Tree Traversal

Debugging

Enable Tracing

Inspect JSON Output

Check RefMap

Contributing

Next Steps