Find a job, a company, or an NGO Key Responsibilities
- Design and develop scalable batch, near-real-time, and streaming data pipelines on GCP.
- Build and manage ETL/ELT pipelines and analytics-ready data products.
- Work with BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Composer, and Cloud Storage.
- Design modern Data Lake, Lakehouse, Data Warehouse, and Data Mesh solutions.
- Develop data models, dimensional models, transformations, and optimized SQL.
- Implement data quality, validation, reconciliation, observability, metadata, and lineage.
- Ensure data solutions meet security, privacy, governance, compliance, and performance standards.
- Implement CI/CD, Git, automated testing, monitoring, and production support.
- Optimize pipelines for performance, reliability, scalability, and cloud cost.
- Use approved AI-assisted engineering tools for development, testing, documentation, SQL optimization, and troubleshooting.
- Provide technical leadership, mentor engineers, and contribute to architecture/design reviews.
Mandatory Skills
- 8+ years in Data Engineering / Data Architecture.
- Strong GCP experience.
- Strong Python and SQL.
- Hands-on ETL/ELT and data pipeline development.
- Experience with BigQuery + Dataflow / Dataproc / Pub/Sub / Cloud Composer.
- Strong data modelling / dimensional modelling experience.
- Knowledge of data governance, quality, lineage, and observability.
- Experience with Git, CI/CD, and automated testing.
- Strong understanding of modern cloud data architecture.
