Offer summary

Qualifications:

5+ years of experience in building production-grade data pipelines and implementing data quality frameworks using Spark, Python, and SQL., Expertise in Databricks, including Delta Lake and AWS integrations, with a focus on data observability tools., Bachelor's or Master's degree in Computer Science or equivalent, along with relevant certifications such as Databricks Certified Professional or AWS/Azure Data Engineering certs., Strong communication skills in English, both oral and written, to effectively collaborate with engineering and data science teams..

Key responsabilities:

Review and redesign data ingestion pipelines, ensuring schema validation and duplicate detection.

Implement end-to-end data quality checks and monitoring using tools like Great Expectations and Apache Airflow.

Develop fault-tolerant data pipelines with error handling mechanisms and centralized logging.

Conduct post-mortems for critical failures and maintain service level agreements (SLA/SLO) for data quality.

Job description

This is a remote position.

About Us:

CloudGeometry is a leading Silicon Valley-based cloud-native system integrator with deep expertise in AWS and the CNCF technology stack. We partner with industry leaders like AWS, Google and Databricks to deliver solutions across all layers of the modern technology stack, from generative UI to Kubernetes-powered MLOps workloads. Our distributed team of top technology experts from the US, Europe, and LATAM collaborates on innovative commercial and open-source projects. Our commitment to innovation and excellence makes us a trusted partner for leading technology companies and VC-funded startups. By pushing the boundaries of what's possible, we help our clients stay ahead in a rapidly evolving technological landscape

Cloudgeometry Engineering Profile:

We are in search of experienced senior software engineers who possess a deep passion for technology and an eagerness to tackle complex technical challenges. You have a strong interest in leveraging AI and cutting-edge methodologies to drive efficiency and precision in your work. A strong commitment to the organization's growth and the advancement of its internal ventures is critical. You care about your project, but also about CloudGeometry communities and their development. Sharing best practices with engineering communities across projects is paramount to you. You must demonstrate a commitment to continuous learning, allocating time to acquire new skills and obtain certifications to validate your expertise and experience.

Project Description:

We seek a hands-on Senior Data Engineer to overhaul our Databricks-based data infrastructure and establish robust data quality frameworks. You will architect reliable data pipelines, implement error handling/alerting systems and enforce engineering best practices.

Responsibilities:

Review and Redesign data ingestion pipelines (with schema validation, duplicate detection, automated encoding correction, missed data enrichment, etc)
Implement end-to-end data quality checks and monitoring (Great Expectations, dbt, Deequ, Monte Carlo, Apache Airflow, etc)
Improve CI/CD pipelines with unit/integration testing, code reviews, and deployment rollback capabilities -Participate engineering tasks like pull request reviews and documentation templates
Create playbooks for common scenarios (schema evolution, backfills, CDC implementations) Error Handling & Reliability Engineering
Develop fault-tolerant pipelines with retry mechanisms, dead-letter queues, and checkpointing
Create centralized error taxonomy and logging and severity-based alerts -Implement data lineage tracking (OpenLineage, Marquez) for root cause analysis of quality issues -Conduct post-mortems for critical failures and maintain SLA/SLO

Requirements

Required Skills:

Technical -5+ years building production-grade data pipelines and implementing data quality frameworks (Spark, Python, SQL)
Expert in Databricks (Delta Lake, CDC, Workflows, Unity Catalog) and AWS integrations -Proven experience with data observability tools
Deep knowledge of distributed systems failure modes and recovery patterns
Ability to balance strategic planning with tactical firefighting
Exceptional communication skills for bridging engineering/data science teams Qualifications -BS/MS in Computer Science or equivalent
Databricks Certified Professional or AWS/Azure Data Engineering certs -C1 English
Excellent oral and written communication skills

Benefits

Benefits:

Remote anywhere
Coworking space financial coverage
Flexible working hours
B2B with multiple benefits
Paid days off annually
Workspace program: $2500 for work equipment of your choice.
English language lessons on all levels
Performance financial incentives for the people who demonstrate interest in the company’s development.
Paid courses and certifications: example AWS, CKA, ML certifications
Participation at international conferences: like CNCF Summits, Kubecon, others