spark-engineer
Use when building Apache Spark applications, distributed data processing pipelines, or optimizing big data workloads. Invoke for DataFrame API, Spark SQL, RDD operations, performance tuning, streaming analytics.
Why use this skill?
Build scalable data pipelines, optimize Spark SQL performance, and manage large-scale ETL workloads with the expert Spark Engineer skill for OpenClaw.
Install via CLI (Recommended)
clawhub install openclaw/skills/skills/veeramanikandanr48/spark-engineerWhat This Skill Does
The spark-engineer skill transforms your OpenClaw agent into a high-performance Apache Spark consultant. It provides deep architectural expertise for distributed data processing, allowing you to build, debug, and optimize complex ETL pipelines. Whether you are dealing with multi-terabyte datasets or fine-tuning cluster configurations, this skill leverages years of industry experience to guide you through efficient DataFrame manipulation, Spark SQL query optimization, and Structured Streaming logic. It bridges the gap between raw data and production-grade systems by providing actionable insights into partition strategies, broadcast join selection, and memory management.
Installation
To integrate this expert capability into your workspace, run the following command in your terminal:
clawhub install openclaw/skills/skills/veeramanikandanr48/spark-engineer
Use Cases
- Pipeline Construction: Architecting end-to-end distributed data pipelines that are fault-tolerant and scalable.
- Performance Tuning: Analyzing Spark UI metrics to diagnose shuffles, data spills, and garbage collection overhead.
- Data Skew Remediation: Implementing advanced techniques like salting or key salting to balance workload distribution across executors.
- API Migration: Modernizing legacy RDD-based applications to high-performance DataFrame and Dataset APIs.
- Production Readiness: Defining schemas, designing robust error-handling mechanisms, and ensuring cluster resource efficiency.
Example Prompts
- "I'm experiencing a massive data skew when joining my 500GB user logs table with the store dimensions. Can you help me implement a broadcast hash join or suggest an alternative skew-handling strategy?"
- "How should I tune my executor memory and shuffle partitions for a Spark application running on a 20-node cluster processing 50TB of data?"
- "Write a structured streaming job in PySpark that consumes from Kafka, applies windowing for a 5-minute rolling average, and writes the output to S3 in Parquet format."
Tips & Limitations
- Best Practices: Always prefer the DataFrame/Dataset API over RDDs, as the Catalyst Optimizer can perform significant code generation and query optimization that RDDs cannot access.
- Resource Management: Remember that excessive caching can lead to OOM errors. Only cache DataFrames that are referenced multiple times in an execution plan.
- Constraints: This skill is strictly for architectural and coding assistance. It cannot directly access your private production clusters unless you provide the specific environment metadata or logs required for analysis. Always scrub PII (Personally Identifiable Information) from any code snippets or log data shared with the AI.
Metadata
Not sure this is the right skill?
Describe what you want to build — we'll match you to the best skill from 16,000+ options.
Find the right skillPaste this into your clawhub.json to enable this plugin.
{
"plugins": {
"official-veeramanikandanr48-spark-engineer": {
"enabled": true,
"auto_update": true
}
}
}Tags(AI)
Flags: code-execution
Related Skills
earnings-calendar
This skill retrieves upcoming earnings announcements for US stocks using the Financial Modeling Prep (FMP) API. Use this when the user requests earnings calendar data, wants to know which companies are reporting earnings in the upcoming week, or needs a weekly earnings review. The skill focuses on mid-cap and above companies (over $2B market cap) that have significant market impact, organizing the data by date and timing in a clean markdown table format. Supports multiple environments (CLI, Desktop, Web) with flexible API key management.
better-auth
Self-hosted auth for TypeScript/Cloudflare Workers with social auth, 2FA, passkeys, organizations, RBAC, and 15+ plugins. Requires Drizzle ORM or Kysely for D1 (no direct adapter). Self-hosted alternative to Clerk/Auth.js. Use when: self-hosting auth on D1, building OAuth provider, multi-tenant SaaS, or troubleshooting D1 adapter errors, session caching, rate limits, Expo crashes, additionalFields bugs.
dividend-growth-pullback-screener
Use this skill to find high-quality dividend growth stocks (12%+ annual dividend growth, 1.5%+ yield) that are experiencing temporary pullbacks, identified by RSI oversold conditions (RSI ≤40). This skill combines fundamental dividend analysis with technical timing indicators to identify buying opportunities in strong dividend growers during short-term weakness.
cli-developer
Use when building CLI tools, implementing argument parsing, or adding interactive prompts. Invoke for CLI design, argument parsing, interactive prompts, progress indicators, shell completions.
options-strategy-advisor
Options trading strategy analysis and simulation tool. Provides theoretical pricing using Black-Scholes model, Greeks calculation, strategy P/L simulation, and risk management guidance. Use when user requests options strategy analysis, covered calls, protective puts, spreads, iron condors, earnings plays, or options risk management. Includes volatility analysis, position sizing, and earnings-based strategy recommendations. Educational focus with practical trade simulation.