Skip to main content

​
Overview

The observe() method identifies actionable elements on a page and returns Action objects that can be previewed, cached, or executed later with act(). This is useful for:
  • Previewing actions before execution
  • Caching actions to replay later
  • Hiding sensitive data from LLM inference
  • Building action libraries for testing

​
Method Signature

observe(instruction?: string, options?: ObserveOptions): Promise<Action[]>

​
Parameters

​
instruction
string
Natural language description of elements to find (e.g., “find the submit button”). If not provided, returns all actionable elements on the page.
​
options
ObserveOptions
Optional configuration for observation.

​
Return Value

Returns a Promise<Action[]> - an array of Action objects: 
interface Action {
  selector: string;      // XPath selector for the element
  description: string;   // Human-readable description
  method?: string;       // Playwright method to use (click, fill, etc.)
  arguments?: string[];  // Arguments for the method
}

​
Usage Examples

​
Basic Observation

import { Stagehand } from "@stagehand/api";

const stagehand = new Stagehand({
  env: "BROWSERBASE",
  apiKey: process.env.BROWSERBASE_API_KEY,
});

await stagehand.init();
const page = stagehand.context.pages()[0];

await page.goto("https://www.apartments.com/san-francisco-ca/");

// Observe a specific element
const [action] = await stagehand.observe("find the 'all filters' button");

console.log("Found:", action.description);
console.log("Selector:", action.selector);
console.log("Method:", action.method);

// Execute the observed action
await stagehand.act(action);

​
Preview Before Execution

// Find the action
const [loginAction] = await stagehand.observe(
  "find the login button"
);

// Review what will happen
console.log("Will execute:", loginAction.description);
console.log("Using method:", loginAction.method);

// Confirm with user or validate
const userConfirmed = await promptUser(
  `Execute ${loginAction.description}?`
);

if (userConfirmed) {
  await stagehand.act(loginAction);
}

​
Caching Actions

import fs from "fs/promises";

// Observe and cache actions
const actions = [
  await stagehand.observe("find the username field"),
  await stagehand.observe("find the password field"),
  await stagehand.observe("find the login button"),
];

// Save to file
await fs.writeFile(
  "login-actions.json",
  JSON.stringify(actions, null, 2)
);

// Later, load and execute without LLM inference
const cached = JSON.parse(
  await fs.readFile("login-actions.json", "utf-8")
);

for (const action of cached.flat()) {
  await stagehand.act(action);
}

​
Find All Actionable Elements

// Without instruction - returns all actionable elements
const allActions = await stagehand.observe();

console.log(`Found ${allActions.length} actionable elements:`);
allActions.forEach((action) => {
  console.log(`- ${action.description} (${action.method})`);
});

​
Focused Observation

// Observe only within a specific section
const sidebarActions = await stagehand.observe(
  "find navigation links",
  {
    selector: "aside.sidebar",
  }
);

// Or use XPath
const formActions = await stagehand.observe(
  "find all form fields",
  {
    selector: "xpath=//form[@id='contact']",
  }
);

​
Multi-Step Form Workflow

await page.goto("https://www.apartments.com/san-francisco-ca/");

// Observe each step
let observation: Action;

[observation] = await stagehand.observe(
  "find the 'all filters' button"
);
await stagehand.act(observation);
await new Promise((resolve) => setTimeout(resolve, 2000));

[observation] = await stagehand.observe(
  "find the '1+' button in the 'beds' section"
);
await stagehand.act(observation);
await new Promise((resolve) => setTimeout(resolve, 2000));

[observation] = await stagehand.observe(
  "find the 'apartments' button in the 'home type' section"
);
await stagehand.act(observation);
await new Promise((resolve) => setTimeout(resolve, 2000));

[observation] = await stagehand.observe(
  "find the pet policy dropdown"
);
await stagehand.act(observation);

​
Multi-Page Observation

const page1 = stagehand.context.pages()[0];
const page2 = await stagehand.context.newPage();

await page2.goto("https://example.com/form");

// Observe on specific page
const actions = await stagehand.observe(
  "find all form fields",
  { page: page2 }
);

​
Building Action Libraries

// Create reusable action library
class ActionLibrary {
  private actions: Map<string, Action> = new Map();

  async register(
    name: string,
    instruction: string,
    stagehand: Stagehand
  ) {
    const [action] = await stagehand.observe(instruction);
    this.actions.set(name, action);
  }

  get(name: string): Action | undefined {
    return this.actions.get(name);
  }

  async execute(name: string, stagehand: Stagehand) {
    const action = this.get(name);
    if (!action) throw new Error(`Action ${name} not found`);
    await stagehand.act(action);
  }
}

// Usage
const lib = new ActionLibrary();
await lib.register("login", "find login button", stagehand);
await lib.register("submit", "find submit button", stagehand);

// Execute by name
await lib.execute("login", stagehand);

​
With Timeout

try {
  const actions = await stagehand.observe(
    "find the button",
    { timeout: 15000 } // 15 seconds
  );
} catch (error) {
  if (error instanceof ObserveTimeoutError) {
    console.error("Observation timed out");
  }
}

​
Filtering Actions

const allActions = await stagehand.observe();

// Filter by method
const clickableElements = allActions.filter(
  (action) => action.method === "click"
);

const inputFields = allActions.filter(
  (action) => action.method === "fill"
);

console.log(`Found ${clickableElements.length} clickable elements`);
console.log(`Found ${inputFields.length} input fields`);

​
How It Works

  1. Snapshot: Captures an accessibility tree of the page
  2. LLM Processing: AI analyzes the tree to find actionable elements
  3. Mapping: Converts element IDs to XPath selectors
  4. Return: Returns Action objects with method and arguments

​
Action Methods

Observed actions can include these Playwright methods:
  • click - Click on element
  • fill - Type into input field
  • selectOption - Select from dropdown
  • check / uncheck - Toggle checkbox
  • hover - Hover over element
  • dragAndDrop - Drag to target (target in arguments)
  • press - Press keyboard key
  • not-supported - Element in shadow DOM or not actionable

​
Use Cases

​
1. Action Preview

Show users what will happen before executing: 
const [action] = await stagehand.observe("find submit button");
console.log(`Ready to: ${action.description}`);
// User confirms, then execute
await stagehand.act(action);

​
2. Action Caching

Reduce LLM calls by caching actions: 
// Cache during development
const actions = await stagehand.observe("find all buttons");
saveToCache(actions);

// Use in production without LLM
const cached = loadFromCache();
await stagehand.act(cached[0]);

​
3. Sensitive Data Handling

Keep credentials out of LLM context: 
// Observe without sensitive data
const [usernameField] = await stagehand.observe(
  "find username field"
);
const [passwordField] = await stagehand.observe(
  "find password field"
);

// Execute with variables (not sent to LLM)
await stagehand.act(usernameField, {
  variables: { value: process.env.USERNAME }
});
await stagehand.act(passwordField, {
  variables: { value: process.env.PASSWORD }
});

​
4. Testing & Validation

Build test suites with observed actions: 
const actions = await stagehand.observe();

// Validate expected elements exist
expect(actions.some(
  (a) => a.description.includes("submit")
)).toBe(true);

​
Best Practices

  1. Be specific with instructions - Get targeted results 
    // Good
await stagehand.observe("find the blue submit button in the footer");

// Less specific
await stagehand.observe("find button");
  2. Review before executing - Check action descriptions 
    const [action] = await stagehand.observe(instruction);
console.log("Will perform:", action.description);
await stagehand.act(action);
  3. Cache for repeated workflows - Save on LLM costs 
    // Cache once
const actions = await observeAndCache();

// Reuse many times
for (let i = 0; i < 100; i++) {
  await stagehand.act(actions[0]);
}
  4. Use focused observation - Improve accuracy on complex pages 
    await stagehand.observe(instruction, {
  selector: ".main-form"
});

​
Error Handling

try {
  const actions = await stagehand.observe("find button");
  
  if (actions.length === 0) {
    console.log("No actions found");
  } else if (actions[0].method === "not-supported") {
    console.log("Element not actionable");
  } else {
    await stagehand.act(actions[0]);
  }
} catch (error) {
  if (error instanceof ObserveTimeoutError) {
    console.error("Observation timed out");
  } else {
    console.error("Observation failed:", error);
  }
}

​
Related Methods

  • act() - Execute actions (including observed actions)
  • extract() - Extract data from pages
  • agent() - Autonomous multi-step automation

Build docs developers (and LLMs) love

Get started for freeTalk to us