web-scraping

What This Skill Does

The web-scraping skill empowers your OpenClaw agent to interact directly with the live web. It provides a robust set of tools for fetching content from URLs, extracting relevant data, and discovering navigational structures across multiple domains. Whether you are analyzing a single technical document, tracking news updates across various sources, or performing complex research that requires cross-referencing multiple URLs, this skill provides the necessary interface to convert raw HTML into clean, usable formats like text or markdown. By handling the complexities of fetching, deduplicating links, and filtering content, the agent can focus on synthesizing information rather than wrestling with raw markup.

Installation

To integrate this skill into your OpenClaw environment, execute the following command in your terminal: clawhub install openclaw/skills/skills/paulgnz/xpr-web-scraping This installs the necessary dependencies authored by paulgnz from the official openclaw/skills repository.

Use Cases

1. Market Research: Gather data from various competitor websites concurrently using scrape_multiple to identify trends.
2. Document Retrieval: Use extract_links with regex patterns to isolate and download all PDF reports from a corporate investor relations page.
3. Content Summarization: Fetch a long-form article using scrape_url in markdown format to maintain context while generating a concise summary.
4. Database Population: Extract structured data points from a series of product pages to create a CSV file for your project.

Example Prompts

1. "Scrape the documentation pages at the following three URLs and summarize the key security updates for each: [URL1, URL2, URL3]."
2. "Go to the official project website, extract all links ending in .pdf, and compile a list of their titles and absolute URLs."
3. "Fetch the content of this landing page as markdown and tell me what the primary call-to-action is."

Tips & Limitations

• Rate Limiting: Adhere to a maximum of 5 requests per minute per domain to ensure reliability and respect server resources.
• Format Selection: Always choose format="text" for heavy analysis to save tokens, and use format="markdown" only when preserving headers and structure is vital.
• Data Size: Be aware that individual page content is capped at 5MB. If a page exceeds this, you may need to focus the agent on specific sub-pages.
• Persistence: Always pair the scraping results with store_deliverable to ensure the scraped data persists as evidence after the session ends.