web-fetcher

What This Skill Does

The web-fetcher is an advanced, intelligent agent skill designed to bridge the gap between unstructured web content and actionable data for OpenClaw. It automatically detects the source platform of a provided URL—ranging from professional social platforms like WeChat and Feishu to multimedia giants like Bilibili and YouTube—and selects the most efficient extraction strategy. By leveraging specialized tools like 'scrapling' for article parsing, 'yt-dlp' for high-quality video acquisition, and 'camoufox' for bypassing sophisticated anti-bot protections on sites like Xiaohongshu, the skill ensures high success rates in data retrieval.

Installation

To integrate this skill into your environment, use the OpenClaw command-line interface: clawhub install openclaw/skills/skills/alexxxiong/web-fetcher

Ensure your local environment meets the runtime requirements. Based on your specific needs, you may need to install supplemental libraries via pip. For standard article scraping, ensure scrapling is installed. For video tasks, yt-dlp is mandatory. For platforms requiring stealth browsing (e.g., Weibo), the camoufox toolkit must be installed and initialized correctly. Detailed dependency mappings are available in the skill's local documentation.

Use Cases

This skill is perfect for researchers, content curators, and developers who need to aggregate information from diverse web sources into a local knowledge base. It is particularly powerful for:

• Creating offline archives of long-form articles from WeChat and Zhihu.
• Batch-processing video tutorials from Bilibili or YouTube for local viewing.
• Scraping complex documentation from Feishu docs into clean Markdown files.
• Automating the collection of media content from platforms that typically employ strict anti-crawling measures.

Example Prompts

1. "抓取这篇文章 https://mp.weixin.qq.com/s/example123 并保存到我的本地笔记文件夹中。"
2. "下载这个B站视频 https://b23.tv/abcxyz 到我的视频库，请确保画质选择最高。"
3. "我想把这份飞书文档里的内容整理成Markdown，你可以用fetch URL帮我抓取一下吗？"

Tips & Limitations

• Platform-Specific Needs: Always ensure that if you are targeting platforms like Xiaohongshu, your environment has camoufox properly configured to handle the stealth-mode browser requirements.
• Data Privacy: Be mindful of platform Terms of Service regarding automated scraping. The fetcher is designed for personal knowledge management.
• Performance: For batch operations, use the --urls-file argument to process multiple links efficiently, reducing overhead and improving the speed of your data collection workflows.