Role Objectives:
Data Engineer II is responsible for owning a defined data domain within a business stream, including its data pipelines, models, quality checks, documentation, & production operations. The role works across the Company's BigQuery and Trino + Apache Iceberg lakehouse environments, solving data engineering challenges related to incremental model design, data layout, SQL dialect differences, reliability, and cost optimization.
The role operates with a high degree of independence, translating business and analytical requirements into well-scoped technical solutions, defining appropriate acceptance criteria, and owning outcomes through production
Roles & Responsibilities:
- Own the design, development, and operation of ingestion and transformation pipelines for a defined data domain, including dependencies, sequencing, backfills, and production support.
- Design reliable and scalable pipelines using appropriate architectural patterns, including idempotency, error handling, retries, recovery, and handling of late or out-of-order data.
- Design and maintain data models for the assigned domain, applying appropriate modeling patterns, automated tests, and documentation to ensure data quality and usability.
- Maintain data quality and pipeline reliability through automated testing, monitoring, gap detection, and recovery mechanisms, and resolve issues through root-cause analysis.
- Partner directly with analysts and business stakeholders to translate requirements into well-scoped technical solutions, define acceptance criteria, and communicate data limitations and dependencies.
- Partner with Product and Engineering teams to ensure new features and changes generate complete and reliable data for downstream analytics and data products.
- Diagnose and optimize pipeline and query performance, assessing correctness, scalability, and cost before and after changes.
- Maintain monitoring, alerting, and runbooks for owned data pipelines, and drive improvements that reduce recurring operational issues and manual effort.
- Review teammates’ data engineering solutions and code, providing constructive technical feedback and promoting maintainable engineering practices.
- Collaborate with Data Platform Engineering and other engineering teams on dependencies and infrastructure requirements affecting the owned domain.
Required Education, Knowledge, And Skills:
- 2-4 years in data engineering or an adjacent production engineering role.
- Strong SQL, including performance. You can read a query plan, spot a bad join, and reason about partitioning and data layout.
- Production Python. Modules, tests, error handling and dependency management, not just scripts.
- Data modelling for a real domain. Dimensional or equivalent models you designed, with tests and documentation that others relied on.
- dbt, or an equivalent transformation framework, in production.
- A workflow orchestrator in production, such as Dagster or Airflow.
- Git and code review as routine practice.
- Docker. Building, running and debugging containerised jobs.
