Scrapling Fetch

What This Skill Does

Scrapling Fetch is a high-performance, robust web scraping solution designed specifically for OpenClaw. It serves as a specialized tool for bypassing complex anti-scraping mechanisms that often render standard HTTP requests useless. By utilizing an advanced engine, this skill handles websites protected by Cloudflare Turnstile, as well as complex platforms like WeChat Official Accounts (mp.weixin.qq.com), Medium, and Substack. It intelligently extracts clean, readable text or Markdown content, making it perfect for preparing web data for LLM consumption. It offers two modes: a highly adaptive 'Scrapling' mode for hard-to-access sites and a lightning-fast 'Jina' mode for standard technical documentation or static blogs.

Installation

To install this skill, run the following command in your terminal: clawhub install openclaw/skills/skills/imgolye/scrapling-fetch. Ensure your environment has the necessary dependencies installed within the OpenClaw workspace. For premium features or large-scale data harvesting, configure the billing module by editing fetch_paid.py with your BILLING_API_KEY obtained from the SkillPay dashboard.

Use Cases

1. AI Content Research: Automatically fetch in-depth technical articles from WeChat or Substack to provide context for AI-generated summaries.
2. Competitive Intelligence: Monitor updates on platforms that aggressively block standard scrapers.
3. Knowledge Base Synchronization: Convert live web documentation into structured Markdown files for local or cloud knowledge management systems.
4. Fallback Mechanism: Use it as a secondary data collection layer when primary, simpler fetchers fail due to 403 Forbidden errors or JavaScript-based protection.

Example Prompts

1. "Scrapling, can you fetch the article from this WeChat link and summarize the key technical points in Markdown? [URL]"
2. "Go to this Medium URL, bypass the anti-scraping, and extract the full text for me to read later."
3. "Attempt to scrape this blog page. If it's a technical site, use fast mode, but if it's protected by Cloudflare, use the full Scrapling engine."

Tips & Limitations

• Rate Limiting: To ensure long-term stability and avoid IP bans, maintain a 2-3 second delay between consecutive requests.
• Authentication: This tool cannot access content behind a login wall, such as private Twitter threads or private Weibo posts.
• Billing: Keep an eye on your usage; ensure you have sufficient balance in your SkillPay wallet when using the paid scraping mode.
• Output Control: Use the --text flag if you prefer raw text output instead of the default JSON format, which is easier for some external text processing pipelines.