Playwright Local Browser Automation

Status: Production Ready ✅ Last Updated: 2026-01-21 Dependencies: Node.js 20+ (Node.js 18 deprecated) or Python 3.9+ Latest Versions: playwright@1.57.0, playwright-stealth@0.0.1, puppeteer-extra-plugin-stealth@2.11.2 Browser Versions: Chromium 143.0.7499.4 | Firefox 144.0.2 | WebKit 26.0

⚠️ v1.57 Breaking Change: Playwright now uses Chrome for Testing builds instead of Chromium. This provides more authentic browser behavior but changes the browser icon and title bar.

---

Quick Start (5 Minutes)

1. Install Playwright

Node.js:

npm install -D playwright
npx playwright install chromium

Python:

pip install playwright
playwright install chromium

Why this matters:

• playwright install downloads browser binaries (~400MB for Chromium)
• Install only needed browsers: chromium, firefox, or webkit
• Binaries stored in ~/.cache/ms-playwright/

2. Basic Page Scrape

import { chromium } from 'playwright';

const browser = await chromium.launch({ headless: true });
const page = await browser.newPage();

await page.goto('https://example.com', { waitUntil: 'networkidle' });
const title = await page.title();
const content = await page.textContent('body');

await browser.close();
console.log({ title, content });

CRITICAL:

• Always close browser with await browser.close() to avoid zombie processes
• Use waitUntil: 'networkidle' for dynamic content (SPAs)
• Default timeout is 30 seconds - adjust with timeout: 60000 if needed

3. Test Locally

# Node.js
npx tsx scrape.ts

# Python
python scrape.py

---

Why Playwright Local vs Cloudflare Browser Rendering

Feature	Playwright Local	Cloudflare Browser Rendering
IP Address	Residential (your ISP)	Datacenter (easily detected)
Stealth Plugins	Full support	Not available
Rate Limits	None	2,000 requests/day free tier
Cost	Free (your CPU)	$5/10k requests after free tier
Browser Control	All Playwright features	Limited API
Concurrency	Your hardware limit	Account-based limits
Session Persistence	Full cookie/storage control	Limited session management
Use Case	Bot-protected sites, auth flows	Simple scraping, serverless

When to use Cloudflare: Serverless environments, simple scraping, cost-efficient at scale When to use Local: Anti-bot bypass needed, residential IP required, complex automation

---

The 7-Step Stealth Setup Process

⚠️ 2025 Reality Check: Stealth plugins work well against basic anti-bot measures, but advanced detection systems (Cloudflare Bot Management, PerimeterX, DataDome) have evolved significantly. The detection landscape now includes:

• Behavioral analysis (mouse patterns, scroll timing, keystroke dynamics)
• TLS fingerprinting (JA3/JA4 signatures)
• Canvas and WebGL fingerprinting
• HTTP/2 fingerprinting

Recommendations:

• Stealth plugins are a good starting point, not a complete solution
• Combine with realistic user behavior simulation (use steps option)
• Consider residential proxies for heavily protected sites
• "What works today may not work tomorrow" - test regularly
• For advanced scenarios, research alternatives like nodriver or undetected-chromedriver

Step 1: Install Stealth Plugin (Node.js)

npm install playwright-extra playwright-stealth

For puppeteer-extra compatibility:

npm install puppeteer-extra puppeteer-extra-plugin-stealth

Step 2: Configure Stealth Mode

playwright-extra:

import { chromium } from 'playwright-extra';
import stealth from 'puppeteer-extra-plugin-stealth';

chromium.use(stealth());

const browser = await chromium.launch({
  headless: true,
  args: [
    '--disable-blink-features=AutomationControlled',
    '--no-sandbox',
    '--disable-setuid-sandbox',
  ],
});

const context = await browser.newContext({
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
  viewport: { width: 1920, height: 1080 },
  locale: 'en-US',
  timezoneId: 'America/New_York',
});

Key Points:

• --disable-blink-features=AutomationControlled removes navigator.webdriver flag
• Randomize viewport sizes to avoid fingerprinting
• Match user agent to browser version (Chrome 120 example above)

Step 3: Mask WebDriver Detection

await page.addInitScript(() => {
  // Remove webdriver property
  Object.defineProperty(navigator, 'webdriver', {
    get: () => undefined,
  });

  // Mock plugins
  Object.defineProperty(navigator, 'plugins', {
    get: () => [1, 2, 3, 4, 5],
  });

  // Mock languages
  Object.defineProperty(navigator, 'languages', {
    get: () => ['en-US', 'en'],
  });

  // Consistent permissions
  const originalQuery = window.navigator.permissions.query;
  window.navigator.permissions.query = (parameters) => (
    parameters.name === 'notifications' ?
      Promise.resolve({ state: Notification.permission }) :
      originalQuery(parameters)
  );
});

Step 4: Human-Like Mouse Movement

// Simulate human cursor movement
async function humanClick(page, selector) {
  const element = await page.locator(selector);
  const box = await element.boundingBox();

  if (box) {
    // Move to random point within element
    const x = box.x + box.width * Math.random();
    const y = box.y + box.height * Math.random();

    await page.mouse.move(x, y, { steps: 10 });
    await page.mouse.click(x, y, { delay: 100 });
  }
}

Step 5: Rotate User Agents

const userAgents = [
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
  'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
];

const randomUA = userAgents[Math.floor(Math.random() * userAgents.length)];

const context = await browser.newContext({
  userAgent: randomUA,
});

Step 6: Cookie and Session Persistence

import { chromium } from 'playwright';
import fs from 'fs/promises';

// Save session
const context = await browser.newContext();
const page = await context.newPage();

// ... perform login ...

const cookies = await context.cookies();
await fs.writeFile('session.json', JSON.stringify(cookies, null, 2));
await context.close();

// Restore session
const savedCookies = JSON.parse(await fs.readFile('session.json', 'utf-8'));
const newContext = await browser.newContext();
await newContext.addCookies(savedCookies);

Step 7: Verify Stealth

Test your setup at: https://bot.sannysoft.com/

const page = await context.newPage();
await page.goto('https://bot.sannysoft.com/', { waitUntil: 'networkidle' });
await page.screenshot({ path: 'stealth-test.png', fullPage: true });

What to check:

• navigator.webdriver should be undefined (not false)
• Chrome should be detected
• Plugins should be populated
• No red flags on the page

---

Critical Rules

Always Do

✅ Use waitUntil: 'networkidle' for SPAs (React, Vue, Angular) ✅ Close browsers with await browser.close() to prevent memory leaks ✅ Wrap automation in try/catch/finally blocks ✅ Set explicit timeouts for unreliable sites ✅ Save screenshots on errors for debugging ✅ Use page.waitForSelector() before interacting with elements ✅ Rotate user agents for high-volume scraping ✅ Test with headless: false first, then switch to headless: true

Never Do

❌ Use page.click() without waiting for element (waitForSelector first) ❌ Rely on fixed setTimeout() for waits (use waitForSelector, waitForLoadState) ❌ Scrape without rate limiting (add delays between requests) ❌ Use same user agent for all requests (rotate agents) ❌ Ignore navigation errors (catch and retry with backoff) ❌ Run headless without testing headed mode first (visual debugging catches issues) ❌ Store credentials in code (use environment variables)

---

Debug Methods (v1.56+)

Playwright v1.56 introduced new methods for capturing debug information without setting up event listeners:

Console Messages

import { test, expect } from '@playwright/test';

test('capture console output', async ({ page }) => {
  await page.goto('https://example.com');

  // Get all recent console messages
  const messages = page.consoleMessages();

  // Filter by type
  const errors = messages.filter(m => m.type() === 'error');
  const logs = messages.filter(m => m.type() === 'log');

  console.log('Console errors:', errors.map(m => m.text()));
});

Page Errors

test('check for JavaScript errors', async ({ page }) => {
  await page.goto('https://example.com');

  // Get all page errors (uncaught exceptions)
  const errors = page.pageErrors();

  // Fail test if any errors occurred
  expect(errors).toHaveLength(0);
});

Network Requests

test('inspect API calls', async ({ page }) => {
  await page.goto('https://example.com');

  // Get all recent network requests
  const requests = page.requests();

  // Filter for API calls
  const apiCalls = requests.filter(r => r.url().includes('/api/'));
  console.log('API calls made:', apiCalls.length);

  // Check for failed requests
  const failed = requests.filter(r => r.failure());
  expect(failed).toHaveLength(0);
});

When to use: Debugging test failures, verifying no console errors, auditing network activity.

---

Advanced Mouse Control (v1.57+)

The steps option provides fine-grained control over mouse movement, useful for:

• Appearing more human-like to anti-bot detection
• Testing drag-and-drop with smooth animations
• Debugging visual interactions

Click with Steps

// Move to element in 10 intermediate steps (smoother, more human-like)
await page.locator('button.submit').click({ steps: 10 });

// Fast click (fewer steps)
await page.locator('button.cancel').click({ steps: 2 });

Drag with Steps

const source = page.locator('#draggable');
const target = page.locator('#dropzone');

// Smooth drag animation (20 steps)
await source.dragTo(target, { steps: 20 });

// Quick drag (5 steps)
await source.dragTo(target, { steps: 5 });

Anti-detection benefit: Many bot detection systems look for instantaneous mouse movements. Using steps: 10 or higher simulates realistic human mouse behavior.

---

Known Issues Prevention

This skill prevents 10 documented issues:

Issue #1: Target Closed Error

Error: Protocol error (Target.sendMessageToTarget): Target closed. Source: https://github.com/microsoft/playwright/issues/2938 Why It Happens: Page was closed before action completed, or browser crashed Prevention:

try {
  await page.goto(url, { timeout: 30000 });
} catch (error) {
  if (error.message.includes('Target closed')) {
    console.log('Browser crashed, restarting...');
    await browser.close();
    browser = await chromium.launch();
  }
}

Issue #2: Element Not Found

Error: TimeoutError: waiting for selector "button" failed: timeout 30000ms exceeded Source: https://playwright.dev/docs/actionability Why It Happens: Element doesn't exist, selector is wrong, or page hasn't loaded Prevention:

// Use waitForSelector with explicit timeout
const button = await page.waitForSelector('button.submit', {
  state: 'visible',
  timeout: 10000,
});
await button.click();

// Or use locator with auto-wait
await page.locator('button.submit').click();

Issue #3: Navigation Timeout

Error: TimeoutError: page.goto: Timeout 30000ms exceeded. Source: https://playwright.dev/docs/navigations Why It Happens: Slow page load, infinite loading spinner, blocked by firewall Prevention:

try {
  await page.goto(url, {
    waitUntil: 'domcontentloaded', // Less strict than networkidle
    timeout: 60000, // Increase for slow sites
  });
} catch (error) {
  if (error.name === 'TimeoutError') {
    console.log('Navigation timeout, checking if page loaded...');
    const title = await page.title();
    if (title) {
      console.log('Page loaded despite timeout');
    }
  }
}

Issue #4: Detached Frame Error

Error: Error: Execution context was destroyed, most likely because of a navigation. Source: https://github.com/microsoft/playwright/issues/3934 Why It Happens: SPA navigation re-rendered the element Prevention:

// Re-query element after navigation
async function safeClick(page, selector) {
  await page.waitForSelector(selector);
  await page.click(selector);
  await page.waitForLoadState('networkidle');
}

Issue #5: Bot Detection (403/Captcha)

Error: Page returns 403 or shows captcha Source: https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth Why It Happens: Site detects navigator.webdriver, datacenter IP, or fingerprint mismatch Prevention: Use stealth mode (Step 2-7 above) + residential IP

Issue #6: File Download Not Completing

Error: Download starts but never finishes Source: https://playwright.dev/docs/downloads Why It Happens: Download event not awaited, file stream not closed Prevention:

const [download] = await Promise.all([
  page.waitForEvent('download'),
  page.click('a.download-link'),
]);

const path = await download.path();
await download.saveAs('./downloads/' + download.suggestedFilename());

Issue #7: Infinite Scroll Not Loading More

Error: Scroll reaches bottom but no new content loads Source: https://playwright.dev/docs/input#scrolling Why It Happens: Scroll event not triggered correctly, or scroll too fast Prevention:

let previousHeight = 0;
while (true) {
  const currentHeight = await page.evaluate(() => document.body.scrollHeight);

  if (currentHeight === previousHeight) {
    break; // No more content
  }

  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
  await page.waitForTimeout(2000); // Wait for new content to load
  previousHeight = currentHeight;
}

Issue #8: WebSocket Connection Failed

Error: WebSocket connection to 'ws://...' failed Source: https://playwright.dev/docs/api/class-browser Why It Happens: Browser launched without --no-sandbox in restrictive environments Prevention:

const browser = await chromium.launch({
  args: ['--no-sandbox', '--disable-setuid-sandbox'],
});

Issue #9: page.pause() Disables Timeout in Headless Mode

Error: Tests hang indefinitely in CI when page.pause() is present Source: GitHub Issue #38754 Why It Happens: page.pause() is ignored in headless mode but disables test timeout, causing subsequent failing assertions to hang forever Prevention:

// Conditional debugging - only pause in local development
if (!process.env.CI && !process.env.HEADLESS) {
  await page.pause();
}

// Or use environment variable
const shouldPause = process.env.DEBUG_MODE === 'true';
if (shouldPause) {
  await page.pause();
}

Impact: HIGH - Can cause CI pipelines to hang indefinitely on failing assertions

Issue #10: Permission Prompts Block Extension Testing in CI

Error: Tests hang on permission prompts when testing browser extensions Source: GitHub Issue #38670 Why It Happens: launchPersistentContext with extensions shows non-dismissible permission prompts (clipboard-read/write, local-network-access) that cannot be auto-granted Prevention:

// Don't use persistent context for extensions in CI
// Use regular context instead
const context = await browser.newContext({
  permissions: ['clipboard-read', 'clipboard-write']
});

// For extensions, pre-grant permissions where possible
const context = await browser.newContext({
  permissions: ['notifications', 'geolocation']
});

Impact: HIGH - Blocks automated extension testing in CI/CD environments

---

Configuration Files Reference

playwright.config.ts (Full Example)

import { defineConfig, devices } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  timeout: 30000,
  expect: {
    timeout: 5000,
  },
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,
  reporter: 'html',

  use: {
    baseURL: 'http://localhost:3000',
    trace: 'on-first-retry',
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',

    // Anti-detection settings
    userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    viewport: { width: 1920, height: 1080 },
    locale: 'en-US',
    timezoneId: 'America/New_York',
  },

  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    },
    {
      name: 'stealth',
      use: {
        ...devices['Desktop Chrome'],
        launchOptions: {
          args: [
            '--disable-blink-features=AutomationControlled',
            '--no-sandbox',
          ],
        },
      },
    },
  ],
});

Why these settings:

• trace: 'on-first-retry' - Captures full trace for debugging failed tests
• screenshot: 'only-on-failure' - Saves disk space
• viewport: { width: 1920, height: 1080 } - Common desktop resolution
• --disable-blink-features=AutomationControlled - Removes webdriver flag

Dynamic Web Server Configuration (v1.57+)

Wait for web server output before starting tests using regular expressions:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  webServer: {
    command: 'npm run dev',
    // Wait for server to print port
    wait: {
      stdout: '/Server running on port (?<SERVER_PORT>\\d+)/'
    },
  },
  use: {
    // Use captured port in tests
    baseURL: `http://localhost:${process.env.SERVER_PORT ?? 3000}`
  }
});

Benefits:

• Handles dynamic ports from dev servers (Vite, Next.js dev mode)
• No need for HTTP readiness checks
• Named capture groups become environment variables
• Works with services that only log readiness messages

When to Use:

• Dev servers with random ports
• Services without HTTP endpoints
• Containerized environments with port mapping

---

Common Patterns

Pattern 1: Authenticated Session Scraping

import { chromium } from 'playwright';
import fs from 'fs/promises';

async function scrapeWithAuth() {
  const browser = await chromium.launch({ headless: false });
  const context = await browser.newContext();
  const page = await context.newPage();

  // Login
  await page.goto('https://example.com/login');
  await page.fill('input[name="email"]', process.env.EMAIL);
  await page.fill('input[name="password"]', process.env.PASSWORD);
  await page.click('button[type="submit"]');
  await page.waitForURL('**/dashboard', { timeout: 10000 });

  // Save session
  const cookies = await context.cookies();
  await fs.writeFile('session.json', JSON.stringify(cookies));

  // Navigate to protected page
  await page.goto('https://example.com/protected-data');
  const data = await page.locator('.data-table').textContent();

  await browser.close();
  return data;
}

When to use: Sites requiring login, scraping user-specific content

Pattern 2: Infinite Scroll with Deduplication

async function scrapeInfiniteScroll(page, selector) {
  const items = new Set();
  let previousCount = 0;
  let noChangeCount = 0;

  while (noChangeCount < 3) {
    const elements = await page.locator(selector).all();

    for (const el of elements) {
      const text = await el.textContent();
      items.add(text);
    }

    if (items.size === previousCount) {
      noChangeCount++;
    } else {
      noChangeCount = 0;
    }

    previousCount = items.size;

    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
    await page.waitForTimeout(1500);
  }

  return Array.from(items);
}

When to use: Twitter feeds, product listings, news sites with infinite scroll

Pattern 3: Multi-Tab Orchestration

async function scrapeMultipleTabs(urls) {
  const browser = await chromium.launch();
  const context = await browser.newContext();

  const results = await Promise.all(
    urls.map(async (url) => {
      const page = await context.newPage();
      await page.goto(url);
      const title = await page.title();
      await page.close();
      return { url, title };
    })
  );

  await browser.close();
  return results;
}

When to use: Scraping multiple pages concurrently, comparison shopping

Pattern 4: Screenshot Full Page

async function captureFullPage(url, outputPath) {
  const browser = await chromium.launch();
  const page = await browser.newPage({
    viewport: { width: 1920, height: 1080 },
  });

  await page.goto(url, { waitUntil: 'networkidle' });

  await page.screenshot({
    path: outputPath,
    fullPage: true,
    type: 'png',
  });

  await browser.close();
}

When to use: Visual regression testing, page archiving, documentation

Pattern 5: PDF Generation

async function generatePDF(url, outputPath) {
  const browser = await chromium.launch();
  const page = await browser.newPage();

  await page.goto(url, { waitUntil: 'networkidle' });

  await page.pdf({
    path: outputPath,
    format: 'A4',
    printBackground: true,
    margin: {
      top: '1cm',
      right: '1cm',
      bottom: '1cm',
      left: '1cm',
    },
  });

  await browser.close();
}

When to use: Report generation, invoice archiving, content preservation

Pattern 6: Form Automation with Validation

async function fillFormWithValidation(page) {
  // Fill fields
  await page.fill('input[name="firstName"]', 'John');
  await page.fill('input[name="lastName"]', 'Doe');
  await page.fill('input[name="email"]', 'john@example.com');

  // Handle dropdowns
  await page.selectOption('select[name="country"]', 'US');

  // Handle checkboxes
  await page.check('input[name="terms"]');

  // Wait for validation
  await page.waitForSelector('input[name="email"]:valid');

  // Submit
  await page.click('button[type="submit"]');

  // Wait for success message
  await page.waitForSelector('.success-message', { timeout: 10000 });
}

When to use: Account creation, form testing, data entry automation

Pattern 7: Retry with Exponential Backoff

async function retryWithBackoff(fn, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await fn();
    } catch (error) {
      if (i === maxRetries - 1) throw error;

      const delay = Math.pow(2, i) * 1000; // 1s, 2s, 4s
      console.log(`Retry ${i + 1}/${maxRetries} after ${delay}ms`);
      await new Promise(resolve => setTimeout(resolve, delay));
    }
  }
}

// Usage
await retryWithBackoff(async () => {
  await page.goto('https://unreliable-site.com');
});

When to use: Flaky networks, rate-limited APIs, unreliable sites

---

Using Bundled Resources

Templates (templates/)

All templates are ready-to-use TypeScript files. Copy from ~/.claude/skills/playwright-local/templates/: