playwright-scraper-skill
Playwright-based web scraping OpenClaw Skill with anti-bot protection. Successfully tested on complex sites like Discuss.com.hk.
Why use this skill?
Master web scraping with the Playwright Scraper Skill. Bypass anti-bot protections, handle dynamic JS sites, and extract data efficiently with OpenClaw.
Install via CLI (Recommended)
clawhub install openclaw/skills/skills/itsjustfred/playwright-scraper-skill-1-2-0What This Skill Does
The playwright-scraper-skill is a robust, modular web scraping toolkit for the OpenClaw platform designed to handle everything from basic static pages to complex, heavily protected websites. By leveraging the power of Playwright, it provides three distinct tiers of scraping technology: lightweight native fetching, standard JavaScript-enabled rendering, and advanced stealth mode to bypass aggressive anti-bot protections like Cloudflare. Whether you are extracting data for market research, monitoring forum threads on sites like Discuss.com.hk, or gathering information from dynamic dashboards, this skill provides the necessary flexibility to ensure your automation succeeds where standard scrapers fail.
Installation
To get started, clone the repository into your skills directory. Ensure you have Node.js installed, then run the following commands in your terminal within the skill folder:
cd playwright-scraper-skillnpm installnpx playwright install chromiumThis installs the required dependencies and the headless Chromium browser engine needed for the dynamic and stealth scraping scripts.
Use Cases
This skill is perfect for developers and data analysts needing reliable access to web data.
- Regular Dynamic Sites: Use the simple script for pages that require JS to render content.
- Protected Environments: Utilize the stealth script for websites employing bot detection, such as Cloudflare or Incapsula. It masks your automation signature by rotating User-Agents and setting navigation flags.
- Mass Data Collection: Integrate these scripts into your OpenClaw agent workflows to automate recurring tasks such as price tracking or sentiment analysis.
Example Prompts
- "Fetch the latest hot topic headlines from https://m.discuss.com.hk/#hot using the stealth scraper."
- "Use the playwright-simple skill to extract the main content from https://example-dynamic-site.com and summarize it for me."
- "OpenClaw, scrape the landing page of https://news-site.com to check for any updates on the current event."
Tips & Limitations
Always start with the built-in web_fetch tool before escalating to Playwright scripts to save system resources. If you face 403 Forbidden errors or anti-bot challenges, immediately switch to the playwright-stealth.js script. Note that stealth browsing takes longer—expect 5-20 seconds per request. For specialized platforms like YouTube or Reddit, consider installing dedicated tools like deep-scraper to avoid unnecessary overhead and improve data extraction quality.
Metadata
Not sure this is the right skill?
Describe what you want to build — we'll match you to the best skill from 16,000+ options.
Find the right skillPaste this into your clawhub.json to enable this plugin.
{
"plugins": {
"official-itsjustfred-playwright-scraper-skill-1-2-0": {
"enabled": true,
"auto_update": true
}
}
}Tags(AI)
Flags: network-access, file-write, file-read, code-execution
Related Skills
n8n-workflow-automation
Designs and outputs n8n workflow JSON with robust triggers, idempotency, error handling, logging, retries, and human-in-the-loop review queues. Use when you need an auditable automation that won’t silently fail.
elite-longterm-memory
Ultimate AI agent memory system for Cursor, Claude, ChatGPT & Copilot. WAL protocol + vector search + git-notes + cloud backup. Never lose context again. Vibe-coding ready.
playwright-mcp
Browser automation via Playwright MCP server. Navigate websites, click elements, fill forms, extract data, take screenshots, and perform full browser automation workflows.
backtest-expert
Expert guidance for systematic backtesting of trading strategies. Use when developing, testing, stress-testing, or validating quantitative trading strategies. Covers "beating ideas to death" methodology, parameter robustness testing, slippage modeling, bias prevention, and interpreting backtest results. Applicable when user asks about backtesting, strategy validation, robustness testing, avoiding overfitting, or systematic trading development.
polymarket
Query Polymarket prediction markets - check odds, trending markets, search events, track prices and momentum. Includes watchlist alerts, resolution calendar, momentum scanner, and paper trading (simulated, no real money).