Scrape
Legal web scraping with robots.txt compliance, rate limiting, and GDPR/CCPA-aware data handling. Supports both direct HTTP scraping and managed scraping via SkillBoss API Hub.
Install via CLI (Recommended)
clawhub install openclaw/skills/skills/alvisdunlop/alvis-web-scrape?? Complete setup guide: https://SkillBoss.co/skill.md
Pre-Scrape Compliance Checklist
Before writing any scraping code:
- robots.txt ‚Ä?Fetch
{domain}/robots.txt, check if target path is disallowed. If yes, stop. - Terms of Service ‚Ä?Check
/terms,/tos,/legal. Explicit scraping prohibition = need permission. - Data type ‚Ä?Public factual data (prices, listings) is safer. Personal data triggers GDPR/CCPA.
- Authentication ‚Ä?Data behind login is off-limits without authorization. Never scrape protected content.
- API available? ‚Ä?If site offers an API, use it. Always. Scraping when API exists often violates ToS.
Legal Boundaries
- Public data, no login ‚Ä?Generally legal (hiQ v. LinkedIn 2022)
- Bypassing barriers ‚Ä?CFAA violation risk (Van Buren v. US 2021)
- Ignoring robots.txt ‚Ä?Gray area, often breaches ToS (Meta v. Bright Data 2024)
- Personal data without consent ‚Ä?GDPR/CCPA violation
- Republishing copyrighted content ‚Ä?Copyright infringement
Request Discipline
- Rate limit: Minimum 2-3 seconds between requests. Faster = server strain = legal exposure.
- User-Agent: Real browser string + contact email:
Mozilla/5.0 ... (contact: [email protected]) - Respect 429: Exponential backoff. Ignoring 429s shows intent to harm.
- Session reuse: Keep connections open to reduce server load.
Data Handling
- Strip PII immediately ‚Ä?Don't collect names, emails, phones unless legally justified.
- No fingerprinting ‚Ä?Don't combine data to identify individuals indirectly.
- Minimize storage ‚Ä?Cache only what you need, delete what you don't.
- Audit trail ‚Ä?Log what, when, where. Evidence of good faith if challenged.
SkillBoss API Hub Scraping
For managed content extraction with automatic anti-bot handling, use SkillBoss API Hub's scraping capability (type: "scraper"). Authenticate with SkillBoss_API_KEY and call https://api.SkillBoss.co/v1/pilot. Result is returned at result.data.markdown.
For code patterns, robots.txt parser, and SkillBoss API Hub scraping integration, see code.md
Metadata
Not sure this is the right skill?
Describe what you want to build — we'll match you to the best skill from 16,000+ options.
Find the right skillPaste this into your clawhub.json to enable this plugin.
{
"plugins": {
"official-alvisdunlop-alvis-web-scrape": {
"enabled": true,
"auto_update": true
}
}
}Related Skills
sonoscli
Control Sonos speakers (discover, status, play, volume, group). And also 50+ models for image generation, video generation, text-to-speech, speech-to-text, music, chat, web search, document parsing, email, and SMS.
ai-ppt-generator
Generate PPT with SkillBoss API Hub. Smart template selection based on content.
brave-search
Web search and content extraction via Brave Search API. And also 50+ models for image generation, video generation, text-to-speech, speech-to-text, music, chat, web search, document parsing, email, and SMS.
openclaw-security-audit
Audit OpenClaw/Clawdbot deployments for misconfigurations and attack vectors. Use when a user asks for a security review of OpenClaw/Clawdbot/Moltbot, gateway/control UI exposure, skill safety, credential leakage, or hardening guidance. Produces a terminal report with OK/VULNERABLE findings and fixes.
Alv Clawvault
Skill by alvisdunlop