kafka-architecture

What This Skill Does

The kafka-architecture skill serves as a specialized knowledge repository and decision-support engine for Apache Kafka infrastructure. It provides technical insights into cluster design, capacity planning, and operational best practices. Users can leverage this agent to architect robust streaming platforms, determine hardware requirements for specific throughput needs, choose between KRaft or legacy ZooKeeper configurations, and optimize partition strategies to maximize parallelism. By interpreting your specific traffic volume and retention requirements, the skill outputs precise architectural recommendations including broker counts, CPU/RAM/Disk allocations, and high-availability setups.

Installation

You can integrate this skill into your environment by executing the following command in your terminal via the OpenClaw CLI: clawhub install openclaw/skills/skills/anton-abyzov/sw-kafka-architecture

Use Cases

• Cluster Sizing: Calculate necessary broker resources based on projected throughput (MB/s) and message retention policies.
• Migration Guidance: Get step-by-step best practices for migrating from legacy ZooKeeper-based clusters to modern KRaft mode clusters (Kafka 3.6+).
• High Availability Design: Receive architectural patterns for cross-datacenter replication and partition placement strategies.
• Performance Tuning: Identify bottlenecks in existing clusters, such as overloaded brokers, partition count limits, or I/O saturation.
• Scalability Planning: Receive advice on when to scale out by adding brokers versus scaling up existing hardware nodes.

Example Prompts

1. "I am planning a production cluster for 200MB/s throughput with 30 days of retention. What should my broker count and hardware profile look like?"
2. "We are currently running on ZooKeeper. Walk me through the recommended approach to migrate a 10-broker cluster to KRaft mode without downtime."
3. "Explain why my consumer groups are lagging; I have 50 partitions but the data is only being processed by a single consumer. How should I adjust my partitioning strategy?"

Tips & Limitations

• Tip: Always prioritize KRaft mode for new installations to avoid technical debt associated with deprecated ZooKeeper dependencies.
• Tip: Remember that partition count is a primary performance factor; avoid 'over-partitioning' to keep controller overhead manageable.
• Limitation: This skill provides design and architectural guidance. It does not directly execute infrastructure changes or perform server provisioning. Always validate hardware sizing in a staging environment under load before production deployment.