ddg-search

What This Skill Does

ddg-search is a robust command-line interface tool designed to perform web searches via DuckDuckGo without the overhead of a graphical browser. It is specifically engineered for data extraction, allowing users to pull search results directly into their terminal or pipe them into downstream analytical tools. The skill supports multiple output formats, including JSON, CSV, and Markdown, making it highly versatile for both automated scripts and manual research workflows. By handling the intricacies of HTML scraping and implementing random delays, ddg-search ensures a smoother interaction with DuckDuckGo's search engine infrastructure while minimizing the likelihood of triggering anti-bot protections.

Installation

To integrate this skill into your environment, use the OpenClaw hub command: clawhub install openclaw/skills/skills/camohiddendj/ddg-search-cli

Use Cases

• LLM Context Gathering: Use the 'compact' output format to fetch relevant, token-efficient search results for feeding into AI prompts.
• Content Curation: Generate markdown-formatted search results to quickly build research reports or documentation bases.
• Data Mining: Use JSON or CSV output to feed large-scale search data into spreadsheets or databases for market analysis or competitive intelligence.
• Automated Monitoring: Schedule queries to track trends or specific keywords, outputting to JSONL for easy integration with streaming data pipelines.

Example Prompts

1. "Search for the latest research papers on transformer architecture published in the past month using ddg-search and give me the titles and links in markdown format."
2. "Perform a search for 'best open source LLM frameworks' and output the results in a compact format so I can use them as context for a follow-up summary."
3. "Scrape the first 20 results for 'Python automation libraries' and save them as a CSV file to my workspace."

Tips & Limitations

• Rate Limiting: The tool automatically inserts random delays (800ms–2900ms) to respect target servers. Do not attempt to bypass these or remove them, as it increases the risk of being blocked by DuckDuckGo.
• Bot Detection: Even with delays, DuckDuckGo may implement CAPTCHAs or temporary blocks if high volumes of requests are sent from a single IP. If you encounter errors, consider reducing the number of pages scraped (-p) or the frequency of your queries.
• Progress Tracking: Always remember that status messages are sent to stderr. If you are piping output to another tool, you don't need to worry about progress logs contaminating your data stream.
• Precision: For complex data extraction tasks, use the default JSON format combined with jq to isolate specific fields like snippets or URLs.