Web Scraper as a Service

Turn scraping briefs into deliverable scraping projects. Generates the scraper, runs it, cleans the data, and packages everything for the client.

How to Use

/web-scraper-as-a-service "Scrape all products from example-store.com — need name, price, description, images. CSV output."
/web-scraper-as-a-service https://example.com --fields "title,price,rating,url" --format csv
/web-scraper-as-a-service brief.txt

Scraper Generation Pipeline

Step 1: Analyze the Target

Before writing any code:

1. Fetch the target URL to understand the page structure
2. Identify:
   • Is the site server-rendered (static HTML) or client-rendered (JavaScript/SPA)?
   • What anti-scraping measures are visible? (Cloudflare, CAPTCHAs, rate limits)
   • Pagination pattern (URL params, infinite scroll, load more button)
   • Data structure (product cards, table rows, list items)
   • Total estimated volume (number of pages/items)
3. Choose the right tool:
   • Static HTML → Python + requests + BeautifulSoup
   • JavaScript-rendered → Python + playwright
   • API available → Direct API calls (check network tab patterns)

Step 2: Build the Scraper

Generate a complete Python script in scraper/ directory:

scraper/
  scrape.py           # Main scraper script
  requirements.txt    # Dependencies
  config.json         # Target URLs, fields, settings
  README.md           # Setup and usage instructions for client

scrape.py must include:

# Required features in every scraper:

# 1. Configuration
import json
config = json.load(open('config.json'))

# 2. Rate limiting (ALWAYS — be respectful)
import time
DELAY_BETWEEN_REQUESTS = 2  # seconds, adjustable in config

# 3. Retry logic
MAX_RETRIES = 3
RETRY_DELAY = 5

# 4. User-Agent rotation
USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36...",
    # ... at least 5 user agents
]

# 5. Progress tracking
print(f"Scraping page {current}/{total} — {items_collected} items collected")

# 6. Error handling
# - Log errors but don't crash on individual page failures
# - Save progress incrementally (don't lose data on crash)
# - Write errors to error_log.txt

# 7. Output
# - Save data incrementally (append to file, don't hold in memory)
# - Support CSV and JSON output
# - Clean and normalize data before saving

# 8. Resume capability
# - Track last successfully scraped page/URL
# - Can resume from where it left off if interrupted

Step 3: Data Cleaning

After scraping, clean the data: