Job Summary
We are seeking an experienced Senior Kafka Engineer to design, develop, implement, and maintain enterprise-grade event streaming platforms using Apache Kafka.
The ideal candidate will have strong hands-on expertise in Kafka architecture, real-time data streaming, distributed systems, and cloud-native technologies. This role will be instrumental in building scalable, highly available, and secure event-driven solutions while establishing engineering standards and best practices across the organization.
The candidate will also provide technical leadership, drive architectural decisions, optimize platform performance, and collaborate with cross-functional teams to support business-critical real-time data integration initiatives.
Key Responsibilities
- Design, develop, and implement scalable, fault-tolerant event-driven architectures using Apache Kafka.
- Build and manage real-time data streaming pipelines and distributed streaming applications.
- Administer, configure, monitor, and optimize Kafka clusters for performance, availability, and security.
- Implement and manage Kafka ecosystem components, including Kafka Connect, Kafka Streams, Schema Registry, and Confluent Platform.
- Establish standards for topic management, partitioning, replication, retention policies, and data governance.
- Perform capacity planning, performance tuning, cluster upgrades, and infrastructure optimization.
- Troubleshoot complex production issues and conduct root cause analysis.
- Implement automation using Infrastructure as Code and CI/CD pipelines.
- Collaborate with application development, DevOps, cloud, infrastructure, and data engineering teams.
- Drive architectural reviews, technical discussions, and engineering best practices.
- Mentor junior engineers and provide technical guidance to project teams.
- Ensure compliance with enterprise security, operational, and data governance standards.
Required Technical Skills
1. Apache Kafka – Primary Skill
- Minimum 5 years of hands-on experience with Apache Kafka.
- Strong understanding of Kafka architecture, brokers, topics, partitions, replication, and consumer groups.
- Expertise in producer and consumer APIs, offset management, and message delivery semantics.
- Hands-on experience with Kafka Connect, Kafka Streams, and Schema Registry.
- Knowledge of Confluent Platform and ksqlDB.
- Experience in Kafka cluster administration, monitoring, troubleshooting, and performance tuning.
2. Programming & Development
- Strong programming experience in Java and Spring Boot.
- Experience developing event-driven microservices and distributed applications.
- Python knowledge is an added advantage.
3. Cloud & DevOps
- Experience with at least one major cloud platform: AWS, Azure, or GCP.
- Hands-on experience with Docker and Kubernetes/OpenShift.
- Experience with Terraform or other Infrastructure as Code tools.
- Familiarity with CI/CD tools such as Jenkins, GitHub Actions, or Azure DevOps.
4. Monitoring & Observability
- Experience with Prometheus and Grafana.
- Knowledge of ELK Stack and Splunk.
- Ability to monitor Kafka performance, troubleshoot bottlenecks, and ensure platform reliability.
Leadership & Technical Responsibilities
- Lead Kafka architecture and solution design discussions.
- Define platform engineering standards, governance, and best practices.
- Conduct code reviews and ensure adherence to engineering standards.
- Guide development teams in implementing event-driven solutions.
- Engage with business and technology stakeholders to translate requirements into scalable streaming architectures.
Qualifications & Experience
- Bachelor's or Master's degree in Computer Science, Information Technology, or a related discipline.
- 8–12 years of overall IT experience.
- Minimum 5 years of hands-on Apache Kafka experience.
- Proven experience supporting mission-critical, enterprise-scale applications.
- Strong experience in distributed systems, event-driven architecture, and real-time data processing.
Preferred Certifications
- Confluent Certified Developer for Apache Kafka.
- Confluent Certified Administrator for Apache Kafka.
- AWS, Azure, or GCP certifications.
- Certified Kubernetes Administrator (CKA) or Certified Kubernetes Application Developer (CKAD)
