Senior Data Engineer

Description

• As a Sr. Data Engineer, you will lead short- and long-term data strategy to deliver value incrementally while partnering with data scientists and engineers
• Build and maintain data pipeline architecture, optimize data flow, and collect data from cross-functional teams. This includes designing and implementing a new scalable data platform using modern data technologies
• Champion data validity and accuracy tests to ensure high-quality data
• Be one step ahead of data scientists and machine learning engineers by providing infrastructure and tools they can leverage to deliver ML solutions
• Work in a fast-paced Agile environment with a proven ML product already in the market, a strong team, and seniors to assist you in taking the next steps in your career.

Conexiom embraces diversity and equal opportunity. We are committed to building a team that represents a variety of backgrounds, perspectives, and skills. We are working to ensure that the profile of our staff reflects the profile of our customers and the communities they serve. For that reason, we seek resumes and expressions of interest from a broad and diverse talent pool. Strength comes from the inclusion of diverse perspectives and experiences.

• Bachelor’s or Master’s degree in a quantitative field such as Physics, Engineering, Computer Science, or equivalent technical fields
• 4+ years’ experience in building a scalable data platform
• Advanced skills in SQL/ Python to script, automate, and integrate data pipelines
• Design, build, and maintain scalable batch and streaming data pipelines in Databricks to support analytics and machine learning needs.
• Play a key role in architectural decisions around data infrastructure and workflows.
• Write clean, well-tested Python code for processing large and complex datasets.
• Work with unstructured data and document databases like MongoDB.
• Build and manage CI/CD pipelines to streamline deployment and version control for data workflows.
• Set up automated monitoring and alerting to ensure pipelines run reliably and efficiently.
• Ensure data and code quality through robust unit testing, integration testing, and validation.
• Document workflows and processes to ensure they are easy to understand and maintain.

• Hands-on experience with Spark Structured Streaming
• Familiarity with vector search tools like Pinecone for working with embeddings.
• Terraform experience
• Azure experience (AWS or GCP okay too)