confluent-kafka-connect
Kafka Connect integration expert. Covers source and sink connectors, JDBC, Elasticsearch, S3, Debezium CDC, SMT (Single Message Transforms), connector configuration, and data pipeline patterns. Activates for kafka connect, connectors, source connector, sink connector, jdbc connector, debezium, smt, data pipeline, cdc.
Why use this skill?
Master Kafka Connect with OpenClaw. Streamline JDBC, Debezium CDC, and SMT configurations for robust, scalable data pipelines across your infrastructure.
Install via CLI (Recommended)
clawhub install openclaw/skills/skills/anton-abyzov/sw-confluent-kafka-connectWhat This Skill Does
The confluent-kafka-connect skill is an advanced integration tool designed to streamline the lifecycle of Kafka Connect ecosystems. It provides expert guidance for configuring, deploying, and troubleshooting both source and sink connectors, enabling seamless data movement between Apache Kafka and external systems like relational databases (MySQL, PostgreSQL), NoSQL stores (Elasticsearch), object storage (S3), and Hadoop HDFS. By leveraging this skill, you can implement robust Change Data Capture (CDC) patterns using Debezium, apply complex Single Message Transforms (SMTs) to cleanse or route data on the fly, and resolve common connector failures with architectural precision.
Installation
To integrate this skill into your environment, use the OpenClaw CLI tool. Run the following command in your terminal:
clawhub install openclaw/skills/skills/anton-abyzov/sw-confluent-kafka-connect
Ensure that your OpenClaw agent has the necessary permissions to access your Kafka Connect cluster endpoints, as this skill interacts with your infrastructure's configuration APIs to generate valid JSON manifests.
Use Cases
- Database Synchronization: Stream raw database logs into Kafka for downstream analytics without placing excessive load on production DBs.
- CDC Implementation: Capture full insert, update, and delete events from legacy databases to maintain a synchronized audit log in Kafka topics.
- Data Transformation: Mask PII (Personally Identifiable Information) or rename fields in real-time as data moves between connectors using SMTs.
- Infrastructure Migration: Move data from on-premise JDBC sources to cloud-native storage like AWS S3 or Elasticsearch sink connectors for searchability.
- Troubleshooting: Analyze stack traces and task failures within your Kafka Connect cluster to identify configuration gaps or serialization issues.
Example Prompts
- "Configure a JDBC source connector to ingest data from my PostgreSQL database starting from the latest incrementing ID."
- "Show me how to set up a Debezium CDC connector for MySQL and explain how to handle schema history topics."
- "I need an SMT configuration to mask the 'social_security_number' field in my incoming Kafka records before they hit the Elasticsearch sink."
Tips & Limitations
- Tip: Always use the 'timestamp+incrementing' mode in JDBC connectors for the highest reliability in data capture.
- Tip: When using Debezium, ensure that your database binlogs or WAL logs are correctly configured for replication slots.
- Limitation: This skill generates configurations based on industry standards but cannot verify your specific network firewalls or database access credentials.
- Limitation: Complex SMT chains can significantly impact throughput; test performance in a staging environment before pushing to high-volume production streams.
Metadata
Not sure this is the right skill?
Describe what you want to build — we'll match you to the best skill from 16,000+ options.
Find the right skillPaste this into your clawhub.json to enable this plugin.
{
"plugins": {
"official-anton-abyzov-sw-confluent-kafka-connect": {
"enabled": true,
"auto_update": true
}
}
}Tags(AI)
Flags: network-access, external-api
Related Skills
network-engineer
Cloud network architect for VPC design, service mesh, zero-trust networking, load balancers, and CDN optimization. Use for network troubleshooting or connectivity issues.
jira-multi-project-mapper
Expert in mapping SpecWeave specs to multiple JIRA projects with intelligent project detection and cross-project coordination. Use when syncing to multiple JIRA projects (project-per-team, component-based), or managing bidirectional sync across team boundaries.
helm-chart-scaffolding
Design, organize, and manage Helm charts for templating and packaging Kubernetes applications with reusable configurations. Use when creating Helm charts, packaging Kubernetes applications, or implementing templated deployments.
performance-optimization
React Native performance with Hermes V1, FlashList, expo-image v2, concurrent rendering. Use for slow app, memory leaks, or FPS issues.
release-strategy-advisor
Release strategy advisor - detects brownfield patterns (tags, CI/CD, changelogs), recommends versioning strategy based on architecture. Creates release-strategy.md.