browser-ladder

What This Skill Does

The browser-ladder skill is a sophisticated resource management strategy for web scraping and automation within the OpenClaw agent ecosystem. Rather than defaulting to expensive or resource-heavy browser environments, this skill implements a tiered approach, or 'ladder', that starts with lightweight, free-to-use methods and progressively scales up as technical requirements increase. By starting at Rung 1, agents attempt to fetch data using simple HTTP requests. If content is dynamic (requires JavaScript), the agent ascends to Rung 2 using local Playwright containers. If local resources are restricted or complex bot-detection is encountered, the agent pushes tasks to cloud providers like BrowserCat or Browserless.io. This systematic approach saves costs, preserves local CPU resources, and ensures high success rates for complex web navigation tasks.

Installation

To integrate this skill, use the ClawHub CLI: clawhub install openclaw/skills/skills/ktpriyatham/browser-ladder. After installation, execute the setup script located at ./skills/browser-ladder/scripts/setup.sh. It is highly recommended to configure your environment variables in your .env file, specifically BROWSERCAT_API_KEY for Rung 3 and BROWSERLESS_TOKEN for Rung 4, to ensure the agent has the necessary credentials to bypass advanced security hurdles when prompted.

Use Cases

This skill is indispensable for developers and researchers who need to aggregate data from a wide variety of sources. Common scenarios include: scraping legacy static websites where a simple GET request is sufficient, rendering high-end Single Page Applications (SPAs) built with React or Vue that require browser engines, and navigating protected websites that employ Cloudflare or other bot-detection mechanisms. It is also ideal for automated testing pipelines where you only want to spin up a full headless browser instance when absolutely necessary for rendering complex UI interactions.

Example Prompts

1. "Fetch the source code of the documentation page at https://docs.example.com and tell me if it contains a login button."
2. "Extract all stock prices from this dashboard https://finance-app.io; note that it requires JavaScript to render the charts."
3. "The website at https://protected-site.com is blocking me with a CAPTCHA. Please use the browser ladder to bypass it and extract the product catalog."

Tips & Limitations

Always start with the lowest rung. Over-utilizing Rung 4 when Rung 1 or 2 is sufficient will quickly exhaust your paid API credits. If you find your agent is stuck, check your Docker installation, as Rung 2 requires a healthy local engine. Ensure that your .env file is properly loaded before execution. Note that while Rung 4 handles most CAPTCHAs, persistent bot blocking may occasionally require manual intervention or proxy rotation configurations not handled by the default ladder setup.