X Twitter Scraper

What This Skill Does

The X Twitter Scraper is a sophisticated, agent-driven automation tool designed to extract public profile information from X (formerly Twitter) without the need for official API keys or expensive subscriptions. Leveraging the Playwright engine, this skill simulates real human browser behavior to navigate, discover, and extract rich metadata from public accounts. Whether you are looking to aggregate contact information, analyze influencer engagement, or map social ecosystems, this tool provides a robust framework for ethical, browser-based data collection.

Installation

To integrate this skill into your environment, use the OpenClaw command-line interface. Ensure your system has Python3 and Chromium installed, as these are required for browser rendering. Run the following command in your terminal:

clawhub install openclaw/skills/skills/arulmozhiv/x-twitter-scraper

Once installed, you can configure your optional Google Custom Search API credentials in the configuration file to enhance discovery capabilities. If you choose not to use Google, the tool defaults to DuckDuckGo for search queries.

Use Cases

This skill is ideal for professionals across various domains. Market researchers use it to identify trending voices in specific niches like Web3 or AI. Recruiters use it to locate and qualify candidates based on their public social footprint. Data analysts utilize the JSON and CSV export features to feed social sentiment data into larger machine learning pipelines. Finally, community managers use the tool to audit brand mentions and analyze competitor follower bases.

Example Prompts

1. "Find me the top 10 tech influencers based in San Francisco and save their profile data to a CSV file."
2. "Go to the profile of @naval and extract his recent tweet activity and engagement metrics."
3. "Discover new accounts tweeting about crypto in New York and add them to my leads queue for further analysis."

Tips & Limitations

To ensure success, keep your scrape operations staggered to mimic natural browsing patterns. The tool includes advanced anti-detection mechanisms, but excessive, rapid-fire requests may still trigger rate limits on the X platform side. We recommend using a residential proxy if you plan on scraping large volumes of data. Always respect X’s robots.txt and privacy policies regarding public versus private data. Note that private accounts will be automatically skipped by the scraper to maintain ethical operation standards.