Skip to main content
Mouse commands provide low-level cursor control. Use these when accessibility APIs are insufficient or when you need pixel-precise positioning.

​
Overview

hover

Move cursor to element center or coordinates

drag

Drag from one element or point to another

mouse-move

Move cursor to absolute screen coordinates

mouse-click

Click at absolute screen coordinates

mouse-down

Press mouse button at coordinates (without releasing)

mouse-up

Release mouse button at coordinates

​
When to Use Mouse Commands

Prefer interaction commands (like click @e3) over mouse commands. Use mouse commands only when:
  • Element has no accessibility representation
  • Pixel-precise positioning is required
  • Custom drag-and-drop interactions
  • Game or canvas automation

​
Common Patterns

​
Hover Over Element

agent-desktop hover @e3
Moves cursor to element center. Useful for revealing tooltips or hover states.

​
Hover at Coordinates

agent-desktop hover --xy 500,300
Moves cursor to absolute screen coordinates (x=500, y=300).

​
Drag Between Elements

agent-desktop drag @e3 --to @e8
Drags from element @e3 center to element @e8 center.

​
Drag Between Coordinates

agent-desktop drag --xy 100,200 --to-xy 400,200
Drags from (100, 200) to (400, 200).

​
Click at Coordinates

agent-desktop mouse-click --xy 500,300
Clicks at absolute coordinates. Supports --button (left, right, middle) and --count (1, 2, 3).

​
Double-Click at Coordinates

agent-desktop mouse-click --xy 500,300 --count 2

​
Right-Click at Coordinates

agent-desktop mouse-click --xy 500,300 --button right

​
Manual Drag Sequence

For complex drag gestures: 
agent-desktop mouse-down --xy 100,200
agent-desktop wait 100
agent-desktop mouse-move --xy 400,200
agent-desktop wait 100
agent-desktop mouse-up --xy 400,200

​
Examples

# Hover over element
agent-desktop hover @e3

# Hover at coordinates
agent-desktop hover --xy 500,300

# Drag between elements
agent-desktop drag @e3 --to @e8

# Drag between coordinates
agent-desktop drag --xy 100,200 --to-xy 400,200

# Click at coordinates
agent-desktop mouse-click --xy 500,300

# Double-click at coordinates
agent-desktop mouse-click --xy 500,300 --count 2

# Right-click at coordinates
agent-desktop mouse-click --xy 500,300 --button right

# Move cursor to coordinates
agent-desktop mouse-move --xy 500,300

# Press mouse button
agent-desktop mouse-down --xy 500,300

# Release mouse button
agent-desktop mouse-up --xy 500,300

​
Use Cases

Hover over elements to trigger tooltips:
agent-desktop hover @e3
agent-desktop wait 500
agent-desktop snapshot --surface popover
Drag files, reorder lists, move windows:
agent-desktop drag @e3 --to @e8
Interact with drawing apps, games, or custom canvas elements:
agent-desktop mouse-down --xy 100,100
agent-desktop mouse-move --xy 200,200
agent-desktop mouse-up --xy 200,200
When accessibility bounds are unreliable:
agent-desktop mouse-click --xy 500,300

​
Coordinate System

  • Origin (0, 0) is top-left of the primary screen
  • X increases to the right
  • Y increases downward
  • Multi-monitor setups: coordinates extend into negative space or beyond primary screen bounds
To get element bounds: 
agent-desktop get @e3 bounds
Returns {"x": 100, "y": 200, "width": 80, "height": 40}.

​
Mouse Buttons

Supported values for --button:
  • left (default)
  • right
  • middle

​
Error Handling

Common error codes:
  • INVALID_ARGS: Coordinates out of bounds or malformed
  • ACTION_FAILED: OS rejected the mouse event
  • ELEMENT_NOT_FOUND: Ref doesn’t exist (when using element-based hover/drag)
All mouse commands return structured JSON with error codes and recovery hints.

Build docs developers (and LLMs) love

Get started for freeTalk to us