Chrome — Python Manipulation Guide

This skill teaches setup-gen (create initial browser state) and reward-gen (verify browser state after agent actions) how to work with Chrome programmatically. Unlike file-based domains, Chrome tasks involve three data layers: config files (JSON/SQLite), live browser via CDP, and accessibility tree.

• Libraries: playwright, sqlite3, json, beautifulsoup4
• Install: pip3 install playwright beautifulsoup4 lxml rapidfuzz tldextract

---

0. GUI Startup on VM (for setup-gen)

After preparing browser state, setup-gen should leave Chrome visibly opened for the GUI agent.

CRITICAL VM LIMIT: GUI launches must set DISPLAY=:0.

import os
import shlex
import subprocess
import time

def launch_gui(command: str, delay_sec: float = 1.0):
    env = os.environ.copy()
    env["DISPLAY"] = ":0"
    subprocess.Popen(
        shlex.split(command),
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
        env=env,
    )
    time.sleep(delay_sec)

# Open Chrome with default profile
launch_gui("google-chrome", delay_sec=2.0)

# Optional URL preload
# launch_gui('google-chrome "https://example.com"', delay_sec=2.0)

Guidelines:

• Use non-blocking launch (Popen) so initial_setup.py exits cleanly.
• If the task needs multiple apps, launch all required windows in order with small sleeps.

---

1. Setting Up Browser State (setup-gen)

Chrome Config File Paths (Linux VM)

import os, json, sqlite3, shutil, time

# Standard paths on the OSWorld Linux VM
CHROME_USER_DATA = os.path.expanduser("~/.config/google-chrome")
CHROME_DEFAULT = os.path.join(CHROME_USER_DATA, "Default")

PREFS_FILE = os.path.join(CHROME_DEFAULT, "Preferences")
LOCAL_STATE = os.path.join(CHROME_USER_DATA, "Local State")
BOOKMARKS_FILE = os.path.join(CHROME_DEFAULT, "Bookmarks")
HISTORY_DB = os.path.join(CHROME_DEFAULT, "History")
COOKIES_DB = os.path.join(CHROME_DEFAULT, "Cookies")
EXTENSIONS_DIR = os.path.join(CHROME_DEFAULT, "Extensions")

# ARM/Chromium alternative
CHROMIUM_DEFAULT = os.path.expanduser("~/snap/chromium/common/chromium/Default")

Modifying Preferences (JSON)

def read_prefs():
    with open(PREFS_FILE, 'r') as f:
        return json.load(f)

def write_prefs(prefs):
    with open(PREFS_FILE, 'w') as f:
        json.dump(prefs, f, indent=2)

prefs = read_prefs()

# Do Not Track
prefs['enable_do_not_track'] = True

# Default font size
prefs.setdefault('webkit', {}).setdefault('webprefs', {})
prefs['webkit']['webprefs']['default_font_size'] = 16
prefs['webkit']['webprefs']['default_fixed_font_size'] = 13
prefs['webkit']['webprefs']['minimum_font_size'] = 6

# Safe browsing
prefs.setdefault('safebrowsing', {})['enhanced'] = True
prefs['safebrowsing']['enabled'] = True

# Color scheme: 0=system, 1=light, 2=dark
prefs.setdefault('browser', {}).setdefault('theme', {})['color_scheme2'] = 2

# Profile name
prefs['profile'] = prefs.get('profile', {})
prefs['profile']['name'] = "Custom Profile"

# Startup: 5 = open new tab page (fresh start)
prefs.setdefault('session', {})['restore_on_startup'] = 5

# Default search engine
prefs.setdefault('default_search_provider_data', {}).setdefault('template_url_data', {})
prefs['default_search_provider_data']['template_url_data']['short_name'] = "DuckDuckGo"

write_prefs(prefs)

Modifying Local State (JSON)

def read_local_state():
    with open(LOCAL_STATE, 'r') as f:
        return json.load(f)

def write_local_state(state):
    with open(LOCAL_STATE, 'w') as f:
        json.dump(state, f, indent=2)

state = read_local_state()

# Chrome language
state['intl'] = state.get('intl', {})
state['intl']['app_locale'] = 'en-US'

# Chrome Labs experiments
state.setdefault('browser', {})['enabled_labs_experiments'] = [
    'smooth-scrolling@1', 'tab-hover-card-images@2'
]

write_local_state(state)

Setting Up Bookmarks (JSON)

import time as _time

def create_bookmark_entry(name, url):
    return {
        "date_added": str(int(_time.time() * 1e6)),  # Chrome microsecond timestamp
        "date_last_used": "0",
        "guid": "",  # Chrome generates this
        "id": str(int(_time.time() * 1e3)),
        "name": name,
        "type": "url",
        "url": url
    }

def create_bookmark_folder(name, children):
    return {
        "children": children,
        "date_added": str(int(_time.time() * 1e6)),
        "date_modified": str(int(_time.time() * 1e6)),
        "guid": "",
        "id": str(int(_time.time() * 1e3)),
        "name": name,
        "type": "folder"
    }

bookmarks = {
    "checksum": "",
    "roots": {
        "bookmark_bar": {
            "children": [
                create_bookmark_entry("Google", "https://www.google.com"),
                create_bookmark_folder("Liked Authors", [
                    create_bookmark_entry("Author1", "https://example.com/author1"),
                    create_bookmark_entry("Author2", "https://example.com/author2"),
                ]),
            ],
            "date_added": str(int(_time.time() * 1e6)),
            "date_modified": str(int(_time.time() * 1e6)),
            "guid": "",
            "id": "1",
            "name": "Bookmarks bar",
            "type": "folder"
        },
        "other": {"children": [], "type": "folder", "name": "Other bookmarks"},
        "synced": {"children": [], "type": "folder", "name": "Mobile bookmarks"}
    },
    "version": 1
}

with open(BOOKMARKS_FILE, 'w') as f:
    json.dump(bookmarks, f, indent=2)

Setting Up Browsing History (SQLite)

import datetime

def chrome_timestamp(dt=None, seconds_ago=0):
    """Chrome timestamps: microseconds since 1601-01-01 (Windows FILETIME)."""
    epoch_1601 = datetime.datetime(1601, 1, 1)
    if dt is None:
        dt = datetime.datetime.utcnow() - datetime.timedelta(seconds=seconds_ago)
    delta = dt - epoch_1601
    return int(delta.total_seconds() * 1_000_000)

def setup_history(history_items):
    """history_items: list of {url, title, visit_time_from_now_in_seconds}"""
    # MUST close Chrome before modifying History DB
    conn = sqlite3.connect(HISTORY_DB)
    c = conn.cursor()
    # Ensure tables exist (normally they do in an existing profile)
    c.execute("""CREATE TABLE IF NOT EXISTS urls (
        id INTEGER PRIMARY KEY, url TEXT, title TEXT, visit_count INTEGER DEFAULT 1,
        typed_count INTEGER DEFAULT 0, last_visit_time INTEGER, hidden INTEGER DEFAULT 0
    )""")
    c.execute("""CREATE TABLE IF NOT EXISTS visits (
        id INTEGER PRIMARY KEY, url INTEGER, visit_time INTEGER, from_visit INTEGER DEFAULT 0,
        transition INTEGER DEFAULT 805306368, segment_id INTEGER DEFAULT 0,
        visit_duration INTEGER DEFAULT 0
    )""")
    for item in history_items:
        ts = chrome_timestamp(seconds_ago=item['visit_time_from_now_in_seconds'])
        c.execute("INSERT INTO urls (url, title, last_visit_time) VALUES (?, ?, ?)",
                  (item['url'], item['title'], ts))
        url_id = c.lastrowid
        c.execute("INSERT INTO visits (url, visit_time) VALUES (?, ?)", (url_id, ts))
    conn.commit()
    conn.close()

setup_history([
    {"url": "https://www.youtube.com/watch?v=abc", "title": "Video", "visit_time_from_now_in_seconds": 3600},
    {"url": "https://news.example.com", "title": "News", "visit_time_from_now_in_seconds": 7200},
])

Opening Tabs via Playwright CDP

from playwright.sync_api import sync_playwright

def open_chrome_tabs(urls, host="localhost", port=9222):
    """Open URLs in existing Chrome (must be running with --remote-debugging-port)."""
    with sync_playwright() as p:
        browser = p.chromium.connect_over_cdp(f"http://{host}:{port}")
        context = browser.contexts[0]  # use existing default profile context
        for url in urls:
            page = context.new_page()
            page.goto(url, timeout=60000, wait_until='networkidle')
        # Close default blank tab if present
        for page in context.pages:
            if page.url == "about:blank" or page.url == "chrome://newtab/":
                page.close()
                break
        # Do NOT call browser.close() — keep Chrome open for agent

open_chrome_tabs(["https://www.google.com", "https://www.github.com"])

---

2. Reading & Verifying Browser State (reward-gen)

Reading Preferences

prefs = read_prefs()  # reuse from above

# Do Not Track
do_not_track = prefs.get('enable_do_not_track', False)

# Font size
font_size = prefs.get('webkit', {}).get('webprefs', {}).get('default_font_size', 16)

# Safe browsing
safe_browsing = prefs.get('safebrowsing', {}).get('enabled', False)
enhanced_safe = prefs.get('safebrowsing', {}).get('enhanced', False)

# Color scheme
scheme = prefs.get('browser', {}).get('theme', {}).get('color_scheme2',
         prefs.get('browser', {}).get('theme', {}).get('color_scheme', 0))
# 0=system, 1=light, 2=dark

# Startup page
startup = prefs.get('session', {}).get('restore_on_startup', None)
# 5 or missing = open new tab page

# Search engine
search = prefs.get('default_search_provider_data', {}).get('template_url_data', {}).get('short_name', '')

Reading Local State

state = read_local_state()
language = state.get('intl', {}).get('app_locale', '')
experiments = state.get('browser', {}).get('enabled_labs_experiments', [])
exp_names = [e.split("@")[0] for e in experiments]

Verifying Bookmarks

with open(BOOKMARKS_FILE, 'r') as f:
    bm = json.load(f)

bar = bm['roots']['bookmark_bar']['children']
folder_names = [b['name'] for b in bar if b['type'] == 'folder']
bookmark_urls = [b['url'] for b in bar if b['type'] == 'url']

# Check specific folder contents
def get_folder_urls(bar_children, folder_name):
    for b in bar_children:
        if b['type'] == 'folder' and b['name'] == folder_name:
            return [c['url'] for c in b['children'] if c['type'] == 'url']
    return []

Verifying History (SQLite)

def read_history():
    conn = sqlite3.connect(HISTORY_DB)
    c = conn.cursor()
    c.execute("SELECT url, title, last_visit_time FROM urls")
    rows = c.fetchall()
    conn.close()
    return rows  # list of (url, title, timestamp)

def check_history_keyword_deleted(keyword):
    """True if no history URL contains the keyword."""
    return all(keyword not in url for url, _, _ in read_history())

Verifying Cookies (SQLite)

def read_cookies():
    conn = sqlite3.connect(COOKIES_DB)
    c = conn.cursor()
    c.execute("SELECT * FROM cookies")
    rows = c.fetchall()
    conn.close()
    return rows

def check_cookie_domain_deleted(domain):
    """True if no cookies from this domain exist."""
    cookies = read_cookies()
    return all(domain not in str(row[1]) for row in cookies)

Verifying Extensions

def get_installed_extensions():
    """Read extension names from manifest.json files."""
    names = []
    if not os.path.isdir(EXTENSIONS_DIR):
        return names
    for ext_id in os.listdir(EXTENSIONS_DIR):
        ext_path = os.path.join(EXTENSIONS_DIR, ext_id)
        if not os.path.isdir(ext_path):
            continue
        for version in os.listdir(ext_path):
            manifest = os.path.join(ext_path, version, "manifest.json")
            if os.path.exists(manifest):
                with open(manifest, 'r') as f:
                    data = json.load(f)
                    names.append(data.get('name', ''))
    return names

# Extension name aliases (same extension, different names across versions)
EXTENSION_ALIASES = [{"Zoom Chrome Extension", "Zoom for Google Chrome"}]

def canonicalize_ext(name):
    for group in EXTENSION_ALIASES:
        if name in group:
            return sorted(group)[0]
    return name

Verifying Active Tab URL via CDP

def get_active_tab_url(host="localhost", port=9222):
    with sync_playwright() as p:
        browser = p.chromium.connect_over_cdp(f"http://{host}:{port}")
        context = browser.contexts[0]
        # Last page is usually the active one
        if context.pages:
            page = context.pages[-1]
            return page.url
    return None

Verifying Page Content via CDP

def get_page_info(url, host="localhost", port=9222):
    """Navigate to URL and return {title, url, content}."""
    with sync_playwright() as p:
        browser = p.chromium.connect_over_cdp(f"http://{host}:{port}")
        context = browser.contexts[0]
        page = context.new_page()
        page.goto(url, timeout=60000, wait_until='networkidle')
        return {"title": page.title(), "url": page.url, "content": page.content()}

def get_all_open_tabs(host="localhost", port=9222):
    """Return list of {title, url} for all open tabs."""
    with sync_playwright() as p:
        browser = p.chromium.connect_over_cdp(f"http://{host}:{port}")
        tabs = []
        for ctx in browser.contexts:
            for page in ctx.pages:
                try:
                    page.wait_for_load_state('networkidle', timeout=30000)
                    tabs.append({"title": page.title(), "url": page.url})
                except:
                    tabs.append({"title": "", "url": page.url})
        return tabs

URL Comparison Helper

from urllib.parse import urlparse, urlunparse
import tldextract

def normalize_url(url):
    """Normalize URL: drop www, ignore TLD suffix differences, strip trailing slash."""
    if not url:
        return ""
    import re
    if not re.match(r'^[a-zA-Z][a-zA-Z0-9+\-.]*://', url):
        url = f"http://{url}"
    parsed = urlparse(url)
    extracted = tldextract.extract(parsed.netloc.lower())
    subdomain = '' if extracted.subdomain == 'www' else extracted.subdomain
    netloc = f"{subdomain}.{extracted.domain}" if subdomain else extracted.domain
    path = parsed.path if parsed.path != '/' else ''
    return urlunparse((parsed.scheme.lower(), netloc, path, parsed.params, parsed.query, parsed.fragment))

def compare_urls(url1, url2):
    return normalize_url(url1) == normalize_url(url2)

Verifying Desktop Shortcuts (.desktop files)

def check_desktop_shortcut(expected_name):
    """Check if a .desktop shortcut exists on Desktop."""
    desktop_dir = os.path.expanduser("~/Desktop")
    for fname in os.listdir(desktop_dir):
        if fname.endswith('.desktop'):
            with open(os.path.join(desktop_dir, fname), 'r') as f:
                content = f.read()
            for line in content.split('\n'):
                if line.startswith('Name='):
                    actual = line[5:]
                    if expected_name.lower() in actual.lower() or actual.lower() in expected_name.lower():
                        return True
    return False

---

3. Bitter Lessons

1. Chrome must be killed before modifying SQLite databases. History, Cookies, and other SQLite files are locked while Chrome is running. Setup scripts must pkill chrome, modify files, then relaunch with --remote-debugging-port=1337. Forgetting this causes "database is locked" errors.

2. Chrome timestamps are NOT Unix timestamps. Chrome uses microseconds since 1601-01-01 (Windows FILETIME epoch). datetime(1601,1,1) + timedelta(microseconds=chrome_ts) gives the real time. Writing Unix timestamps will produce dates in the 1600s.

3. connect_over_cdp reuses existing contexts. browser.contexts[0] is the real Chrome profile with cookies, history, and extensions. Do NOT create browser.new_context() — it creates an incognito-like context that shares nothing with the user's profile.

4. URL comparison must normalize aggressively. www.google.com and google.com are the same. airbnb.com and airbnb.com.sg may be the same. Always strip www, ignore TLD suffix, and normalize trailing slashes before comparing.

5. Preferences file has async writes. Chrome may not flush settings to Preferences immediately. After changing settings via the Chrome UI, poll the file up to 5 times with 1-second delays. Similarly, after writing Preferences, Chrome may overwrite your changes on next save. Write BEFORE Chrome starts.

6. Accessibility tree URL lacks protocol. The URL from AT-SPI's "Address and search bar" entry has no https:// prefix. Always prepend a goto_prefix (typically "https://") before using it.

7. Extension names have aliases. The same Chrome extension can have different manifest.json names across versions (e.g., "Zoom Chrome Extension" vs "Zoom for Google Chrome"). Canonicalize names before comparison using alias groups.

8. page.goto() with networkidle can hang on streaming pages. Some pages (YouTube live, infinite feeds) never reach networkidle. Use timeout=60000 and catch TimeoutError gracefully — the page may still be usable.

9. socat bridges port 9222 to 1337 inside the VM. Chrome runs with --remote-debugging-port=1337 internally. An socat tcp-listen:9222,fork tcp:localhost:1337 bridge exposes it on port 9222 for external access. Setup scripts must launch both processes.

10. ARM machines use chromium, x86 uses google-chrome. Binary name, user data path, and accessibility tree application name all differ. Check platform.machine() to determine which binary to use. ARM: ~/snap/chromium/common/chromium/Default/, x86: ~/.config/google-chrome/Default/.

11. HTML parsing needs shadow DOM traversal for Chrome settings. Chrome settings pages (chrome://settings/*) use nested shadow DOMs. Regular CSS selectors don't penetrate shadow roots. Use page.evaluate() with JavaScript that calls .shadowRoot.querySelector() chains.

12. compare_urls drops TLD suffixes entirely. The URL normalizer uses tldextract and ignores the suffix, so airbnb.com and airbnb.co.uk normalize to the same domain. This is intentional for cross-region tolerance but can cause false matches.

13. Cookie values are encrypted on disk. Chrome encrypts cookie values using platform-specific keys (DPAPI on Windows, keychain on macOS, gnome-keyring on Linux). The cookies table has encrypted value and encrypted_value columns. For verification, check domain/name presence rather than decrypting values.

14. Post-config must kill and restart Chrome before reading settings. Many evaluators use a postconfig sequence: pkill chrome → sleep 3s → relaunch Chrome → sleep 3s → then read. This ensures settings are flushed to disk and Chrome re-reads the state cleanly.