Job SummaryKey Responsibilities
- * Design, build and maintain batch and real-time data pipelines.
- Develop data products using Databricks Lakeflow Declarative Pipelines, PySpark and SQL.
- Build streaming solutions using Spark Structured Streaming and Lakeflow declarative pipelines
- Optimise pipelines for scalability, reliability and performance.
- Develop automated tests and participate in code reviews.
- Contribute to CI/CD and engineering best practices.
- Monitor, troubleshoot and support production workloads.
- Use AI-assisted development tools to improve productivity while maintaining high engineering standards.
Essential Skills
- * Databricks
- Lakeflow Declarative Pipelines and DXQ, DB Genie, MCP Server
- Apache Spark (PySpark)
- Spark Structured Streaming
- Auto Loader
- Unity Catalog
- Python
- SQL
- Delta Lake
- Azure Event Hubs or equivalent event streaming platforms
- Git and collaborative development workflows
- Unit testing
- CI/CD
- Strong analytical and problem-solving skills
Nice to Have
- * Performance tuning and Spark optimisation
- Data modelling
- Azure DevOps
- AI coding assistants (GitHub Copilot, Cursor or similar)
- Experience working in Agile or SAFe environments
We're Looking For An Engineer Who
- * Takes ownership from design through production support.
- Writes clean, maintainable and well-tested code.
- Builds reusable, production-ready solutions.
- Collaborates effectively with teammates and stakeholders.
- Continuously learns and embraces modern engineering practices.
