Playwright Browser Integration

​

What Playwright Provides

• Full JavaScript Rendering - Execute and wait for JavaScript to complete before capturing content
• High-Quality Screenshots - Capture full-page screenshots with configurable quality and format (PNG/JPEG)
• Browser Steps Support - Automate interactions like clicking buttons, filling forms, and logging in
• Visual Selector - Point-and-click element selection for precise monitoring
• XPath Element Data - Extract structured data from page elements
• Advanced Screenshot Stitching - Automatically stitch large pages up to 16,000px height
• Network Control - Access response headers, status codes, and network events
• Proxy Support - Full proxy configuration including authentication

​

Docker Configuration

​

Using docker-compose.yml

services:
  changedetection:
    image: ghcr.io/dgtlmoon/changedetection.io
    environment:
      # Connect to the Playwright browser service
      - PLAYWRIGHT_DRIVER_URL=ws://browser-sockpuppet-chrome:3000
    depends_on:
      browser-sockpuppet-chrome:
        condition: service_started

  # Sockpuppetbrowser - Chrome wrapped in API for fast fetching
  browser-sockpuppet-chrome:
    hostname: browser-sockpuppet-chrome
    image: dgtlmoon/sockpuppetbrowser:latest
    cap_add:
      - SYS_ADMIN
    restart: unless-stopped
    environment:
      - SCREEN_WIDTH=1920
      - SCREEN_HEIGHT=1024
      - SCREEN_DEPTH=16
      - MAX_CONCURRENT_CHROME_PROCESSES=10

​

Environment Variables

​

Required Configuration

PLAYWRIGHT_DRIVER_URL=ws://browser-sockpuppet-chrome:3000

​

Optional Configuration

PLAYWRIGHT_BROWSER_TYPE=chromium  # or firefox, webkit

WEBDRIVER_DELAY_BEFORE_CONTENT_READY=5

SCREENSHOT_QUALITY=72

SCREENSHOT_MAX_HEIGHT=16000

SCREENSHOT_CHUNK_HEIGHT=10000

PLAYWRIGHT_SERVICE_WORKERS=block  # Useful for sites like YouTube

​

Proxy Configuration

playwright_proxy_server=http://proxy.example.com:8080
playwright_proxy_username=user
playwright_proxy_password=pass
playwright_proxy_bypass=localhost,127.0.0.1

​

When to Use Playwright

​

JavaScript-Heavy Websites

• Content is loaded dynamically via AJAX or React/Vue/Angular frameworks
• Prices, stock status, or data appear after page load
• Content requires scrolling or user interaction to appear

​

Single Page Applications (SPAs)

• Sites that don’t use traditional page navigation
• Modern web apps that update content without refreshing

​

Authentication Required

• Logging into websites before monitoring protected content
• Using Browser Steps to automate login flows
• Maintaining session state across checks

​

Visual Monitoring

• Tracking layout changes or visual elements
• Monitoring images, charts, or dynamically rendered graphics
• Using screenshots for visual comparison

​

Anti-Bot Protection

• Sites using Cloudflare or similar protection
• Pages that block simple HTTP requests
• Sites checking for browser fingerprints

​

JavaScript Rendering

​

Custom JavaScript Execution

// Wait for specific element
await new Promise(r => setTimeout(r, 2000));
document.querySelector('.load-more').click();

// Scroll to load lazy content
window.scrollTo(0, document.body.scrollHeight);

// Modify page before capture
document.querySelector('.annoying-popup').remove();

​

Render Delays

1. Page navigation to complete
2. DOM content to be loaded
3. Network requests to settle
4. Additional delay configured via WEBDRIVER_DELAY_BEFORE_CONTENT_READY

​

Screenshots and Visual Comparison

​

Screenshot Formats

• Smaller file size
• Configurable quality (1-100)
• Faster processing
• Good for content monitoring

• Lossless quality
• Larger file size
• Better for visual comparison
• Preserves transparency

​

Full-Page Screenshots

1. Single Viewport - For pages smaller than viewport height, captures in one shot
2. Stitched Pages - For tall pages, captures in chunks and stitches seamlessly
3. Memory Optimized - Uses subprocess isolation to prevent memory leaks
4. Height Limit - Respects SCREENSHOT_MAX_TOTAL_HEIGHT to prevent excessive memory use

​

Screenshot Stitching Process

Page Height: 25,000px
Viewport: 1920x1024px
Chunk Size: 10,000px

→ Capture 3 chunks:
  - Chunk 1: 0-10,000px
  - Chunk 2: 10,000-20,000px  
  - Chunk 3: 20,000-16,000px (limited by SCREENSHOT_MAX_HEIGHT)

→ Stitch in separate process to avoid memory leaks
→ Return final JPEG/PNG screenshot

​

Visual Selector Integration

1. Click elements on the screenshot to select them
2. Generate XPath automatically
3. Preview extracted content in real-time
4. Refine selections with multiple elements

​

Performance Considerations

​

Resource Usage

• Browser processes: ~200-500MB per concurrent watch
• Screenshots: ~5-50MB depending on page size and quality
• Stitching subprocess: temporary memory spike, then released

• JavaScript execution and rendering
• Screenshot encoding (JPEG/PNG)
• Image stitching for large pages

• WebSocket connection to browser service
• Screenshots transferred as base64 (use lower quality to reduce overhead)
• All page resources (images, CSS, JS) are fetched

​

Optimization Tips

1. Lower Screenshot Quality - Set SCREENSHOT_QUALITY=40 for smaller transfers
2. Limit Screenshot Height - Reduce SCREENSHOT_MAX_HEIGHT if not needed
3. Block Service Workers - Set PLAYWRIGHT_SERVICE_WORKERS=block for media sites
4. Concurrent Processes - Adjust MAX_CONCURRENT_CHROME_PROCESSES based on RAM
5. Use Viewport Sizing - Configure smaller viewport in sockpuppetbrowser for faster renders

​

Speed Comparison

​

Browser Steps Integration

​

Supported Actions

• Click elements
• Fill text fields
• Select dropdowns
• Wait for elements
• Execute custom JavaScript
• Take screenshots at each step
• Navigate between pages

​

Example: Login Flow

1. Click "Login" button
2. Fill username field: [email protected]
3. Fill password field: ********
4. Click "Submit"
5. Wait 2 seconds
6. Execute JS: console.log('Logged in')

​

Troubleshooting

​

Connection Issues

# Check browser service is running
docker ps | grep sockpuppet

# Check WebSocket connection
curl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" \
  http://browser-sockpuppet-chrome:3000

# Verify environment variable
docker exec changedetection env | grep PLAYWRIGHT

​

Memory Issues

# Reduce concurrent processes
MAX_CONCURRENT_CHROME_PROCESSES=5

# Lower screenshot limits
SCREENSHOT_MAX_HEIGHT=10000
SCREENSHOT_QUALITY=50

​

Slow Performance

# Reduce wait time
WEBDRIVER_DELAY_BEFORE_CONTENT_READY=2

# Block unnecessary resources (add to Browser Steps)
Execute JS: 
navigator.serviceWorker.getRegistrations().then(r => 
  r.forEach(reg => reg.unregister())
);

​

Screenshot Quality Issues

# Use PNG for lossless quality
Select PNG format in watch settings

# Increase JPEG quality
SCREENSHOT_QUALITY=90

​

Comparison with Other Fetchers

​

vs. HTTP Requests Fetcher

• Executes JavaScript
• Gets real rendered content
• Handles modern web apps
• Takes screenshots
• Supports automation

• Much faster (100x)
• Lower resource usage
• Works for static content
• No browser service needed

​

vs. Selenium WebDriver

• Better screenshot support (full page)
• Reports HTTP status codes
• More reliable
• Supports Browser Steps
• Better error handling
• Active development

• Deprecated in changedetection.io
• No Browser Steps support
• Limited screenshot quality
• No status code reporting
• Legacy technology

​

vs. Puppeteer Direct

• More stable
• Better maintained
• Cross-browser support
• Better error messages

• Slightly faster startup
• Can be enabled with FAST_PUPPETEER_CHROME_FETCHER=True
• Same underlying browser (Chrome)

​

Advanced Configuration

​

Custom Browser Connection

PLAYWRIGHT_DRIVER_URL=wss://external-browser.example.com:3000

​

Viewport Sizing

# Mobile viewport
SCREEN_WIDTH=375
SCREEN_HEIGHT=667

# Desktop
SCREEN_WIDTH=1920  
SCREEN_HEIGHT=1080

# Tablet
SCREEN_WIDTH=768
SCREEN_HEIGHT=1024

​

Multiple Browser Types

# Use Firefox instead of Chrome
PLAYWRIGHT_BROWSER_TYPE=firefox

# Use WebKit (Safari engine)
PLAYWRIGHT_BROWSER_TYPE=webkit

​

Best Practices

1. Start with HTTP Requests - Only use Playwright when JavaScript is needed
2. Use Visual Selector - Easier than writing XPath manually
3. Optimize Screenshots - Balance quality vs. performance for your needs
4. Monitor Resource Usage - Watch memory and CPU when scaling
5. Test Browser Steps - Verify automation works before relying on it
6. Set Appropriate Delays - Too short misses content, too long wastes time
7. Use Proxies Wisely - Rotate IPs for high-volume monitoring
8. Regular Cleanup - Restart browser service periodically to free memory

​

Related Documentation

• Browser Steps - Automate interactions before monitoring
• Visual Selector - Point-and-click element selection
• Proxy Configuration - Using proxies with Playwright
• Selenium WebDriver - Legacy browser option
• Chrome Extension - Add watches from your browser