Browser Automation: Playwright Web Scraping

PocketPaw includes a browser automation tool powered by Playwright. Unlike screenshot-based approaches, it uses accessibility tree snapshots for reliable, fast interaction.

How It Works

The BrowserDriver provides:

1. Navigation - Open URLs and navigate between pages
2. Accessibility tree - Get a structured representation of the page
3. Ref map - Maps reference numbers to CSS selectors for clicking/typing
4. Interaction - Click elements, type text, scroll, using ref numbers

Why Accessibility Tree?

Instead of taking screenshots and using vision models, PocketPaw reads the page’s accessibility tree. This is:

• Faster - No image processing or vision API calls
• More reliable - Structured data instead of visual interpretation
• Cheaper - No vision model tokens
• Accessible - Works with any page, including dynamic SPAs

Usage

User: Go to news.ycombinator.com and find the top story

Agent: [uses browser tool]
  → navigate to "https://news.ycombinator.com"
  → accessibility tree shows: [1] "Story Title" link [2] "comments" link ...
  → click ref 1
  → read page content

Configuration

The browser tool uses Playwright. Install it with:

curl -fsSL https://pocketpaw.xyz/install.sh | sh

# Or add the browser extra manually
pip install pocketpaw[browser]
playwright install chromium

NavigationResult

Each browser action returns a NavigationResult:

@dataclass
class NavigationResult:
    url: str                    # Current URL
    title: str                  # Page title
    accessibility_tree: str     # Structured page content
    refmap: dict[int, str]      # Ref number → CSS selector mapping

Policy Group

The browser tool belongs to group:browser. It’s included in the coding profile.

Installation

curl -fsSL https://pocketpaw.xyz/install.sh | sh

# Or add the browser extra manually
pip install pocketpaw[browser]

This installs playwright as an optional dependency. You also need to install browser binaries:

playwright install chromium

Related