web-scraper

What This Skill Does

The web-scraper skill for OpenClaw is a sophisticated data engineering tool designed to navigate the complexities of modern web structures. It operates on a robust 5-stage pipeline, prioritizing efficiency by utilizing a cascade fallback method: starting with lightweight HTTP requests and only escalating to headless browser automation or LLM-driven comprehension when absolutely necessary. This prevents wasted resources and minimizes the risk of IP-based blocking.

The tool is built to handle content cleaning, boilerplate removal, and structured metadata extraction. It distinguishes between raw HTML and semantically meaningful text, ensuring that any subsequent AI entity extraction is performed on high-quality, sanitized data rather than noisy DOM elements.

Installation

To integrate this capability into your OpenClaw environment, execute the following command in your terminal: clawhub install openclaw/skills/skills/guifav/web-scraper Ensure you have Python 3.10+ and the necessary system dependencies for Playwright, as the tool uses headless browser rendering for JavaScript-heavy targets.

Use Cases

• Competitive Intelligence: Monitoring product pricing, feature updates, and press releases across multiple competitor domains.
• Market Research: Aggregating news articles from various sources and using LLMs to extract key entities, sentiment, and trends into a structured JSON database.
• Content Migration: Extracting clean text from legacy documentation sites or blogs for migration into knowledge bases or LLM fine-tuning datasets.
• Lead Generation: Identifying contact information and business metadata from directory listings while maintaining compliance with local access protocols.

Example Prompts

1. "Scrape the latest blog posts from https://example.com/blog and save the extracted titles, authors, and publish dates into a structured JSON file."
2. "Go to https://techcrunch.com, identify the main article content, remove all navigation and footer boilerplate, and extract all mentioned company names and funding amounts."
3. "Perform a bulk crawl of the documentation pages listed in this text file, normalize the content to Markdown, and ensure all internal links are preserved in the metadata."

Tips & Limitations

• Planning First: Always allow the agent to execute its Planning Protocol. Skipping this leads to inefficient crawling and increased risk of being rate-limited.
• Resource Management: For large-scale crawls, monitor your local disk space and memory usage, as headless browser sessions consume significantly more RAM than simple HTTP clients.
• Credential Security: The skill is designed to reference OPENROUTER_API_KEY only within template code. Never hardcode credentials into your scraping scripts. The skill will never read your .env files directly.
• Ethical Crawling: Always check robots.txt before initiating a scrape. If a site explicitly forbids bot access, the agent's safety protocols will prevent the action to ensure ethical compliance.