senior-data-engineer

What This Skill Does

The Senior Data Engineer skill is a specialized OpenClaw agent designed to act as a production-grade data architect. It handles the full lifecycle of data infrastructure, from initial requirement gathering and schema modeling to deployment and performance tuning. This skill leverages industry-standard patterns to help you build, maintain, and scale data pipelines. Whether you are dealing with massive historical datasets that require batch ETL processing, or high-velocity event streams that necessitate sub-second latency, this skill provides the architectural oversight and code-generation capabilities to execute effectively. It integrates seamlessly with modern data stack components including Apache Spark, Airflow, dbt, and cloud-native warehousing solutions like Snowflake and BigQuery.

Installation

To add this skill to your OpenClaw environment, execute the following command in your terminal:

clawhub install openclaw/skills/skills/alirezarezvani/senior-data-engineer

Ensure that you have the appropriate environment variables configured for your cloud provider (e.g., AWS, GCP, or Azure) to allow the skill to interface with your data infrastructure components.

Use Cases

• Pipeline Orchestration: Architecting complex Airflow DAGs with dependency management, backfilling logic, and task retries.
• Modern Data Stack Implementation: Building and managing dbt models for modular and documented SQL transformations.
• Performance Tuning: Diagnosing slow-running Spark jobs by analyzing execution plans and partition strategies.
• Data Quality Assurance: Implementing proactive data contracts and validation checks to prevent data drift and downstream failures.
• Architectural Migration: Assessing the trade-offs between Lambda and Kappa architectures for real-time streaming requirements.

Example Prompts

1. "I have a 5TB daily log file in S3. Design an ETL pipeline using Spark and Airflow to aggregate this data into a star schema in Snowflake, including error handling for late-arriving records."
2. "Can you review my dbt project structure and suggest improvements for modularity and incremental processing of our user_activity table?"
3. "My current Kafka-to-BigQuery streaming pipeline is experiencing latency spikes. How should I approach monitoring to identify whether the issue is at the consumer group or the load step?"

Tips & Limitations

• Proactive Monitoring: Always define your data quality requirements before building the pipeline to ensure that freshness and schema validation are baked in from day one.
• Environment Safety: This skill generates code that interacts with your data infrastructure. While highly effective, always review generated configurations in a development or staging environment before applying them to production systems.
• Cost Awareness: Be mindful that high-throughput streaming architectures often incur significantly higher cloud costs than batch-based alternatives; use the built-in Decision Framework provided by this skill to evaluate your cost-to-value ratio.