Job Summary
We are looking for a skilled Data & AI Engineer with 3+ years of experience in data engineering and AI-driven solutions to join our team. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines and infrastructure that support AI-powered chatbots and agentic AI use cases.
The role requires strong expertise in modern data technologies, big data processing, and data orchestration, along with the ability to understand business requirements and translate them into technical solutions for AI chatbot and AI agent development.
Key Responsibilities
- Design, develop, and maintain scalable data pipelines to support AI chatbot and agentic AI applications.
- Build and manage data processing workflows using Apache Airflow and Apache Spark.
- Work with big data technologies, including Hadoop HDFS, Hive Metastore (HMS), Iceberg, Trino, and Hadoop Catalog.
- Develop and optimize data ingestion and streaming solutions using Apache Kafka.
- Manage and maintain data storage, processing, and query infrastructure to ensure data availability and performance.
- Implement and maintain data monitoring, logging, and troubleshooting solutions using ELK Stack and Apache Doris.
- Collaborate with business stakeholders and technical teams to gather, analyze, and document business requirements for AI chatbot and agentic use cases.
- Translate business requirements into technical specifications and data architecture solutions.
- Identify opportunities to leverage AI agents and chatbot technologies to automate business processes and improve operational efficiency.
- Ensure data quality, reliability, scalability, and security across data platforms.
Technical Requirements (Must-Have)
- 3+ years of experience in Data Engineering, Big Data, or a related field.
- Strong hands-on experience with:
- Apache Airflow: Workflow orchestration and pipeline automation.
- Apache Spark: Distributed data processing and analytics.
- Hadoop HDFS: Distributed data storage and management.
- Hive Metastore (HMS): Metadata management.
- Apache Iceberg: Open table formats and data lake management.
- Trino: Distributed SQL query processing.
- Apache Kafka: Real-time data streaming and event processing.
- ELK Stack: Centralized logging, monitoring, and analytics.
- Apache Doris: Distributed analytical database.
- Hadoop Catalog: Data catalog and metadata management.
