Match score not available

Data Ingestion Engineer

Remote:

Full Remote

Contract:

Full time

Experience:

Mid-level (2-5 years)

Work from:

Texas (USA), United States

Offer summary

Qualifications:

3+ years of experience in data engineering, Bachelor's degree in related field, Hands-on with Spark/Scala, SQL, Python/PySpark, Experience with ETL/ELT tools and data quality guidelines, Familiarity with CI/CD and Agile practices.

Key responsabilities:

Ingest data from various source systems
Manage and optimize ETL/ELT pipelines
Document ingestion and integration processes
Ensure design meets security and compliance guidelines
Maintain pipeline infrastructure and implement monitoring

Encora Inc. XLarge http://www.encora.com/

5001 - 10000 Employees

See more Encora Inc. offers

Job description

Role Overview:

This role will be responsible for conducting data ingestion and platform management on the Databricks platform. This role requires a deep understanding of:

Data Lake ingestion processes and best practices
ETL/ELT implementation
CI/CD
System integration tools
Data pipeline management

About the role:

Ingest data from a variety of source systems and tailor ingestion approaches on a per-system basis
Manage, maintain, and oversee ETL/ELT pipelines on the Databricks platform
Optimize data pipelines for scalability and speed
Document ingestion and integration flows and pipelines
Use Airflow to schedule and automate ingestion jobs
Manage metadata and master data in technical data catalog
Ensure ELT/ETL design meets required security and compliance guidelines, and ensure PII management, flagging and risk assessment during ingestion
Maintain ETL/ELT pipeline infrastructure and implement automated monitoring strategies
Ensure adherence to SDLC best practices

About you:

Bring 3+ years of experience in data engineering, ingestion pipelining, and ETL/ELT
Hold a bachelor's degree in computer science, engineering, statistics, or related field
Have hands-on experience with and understanding of the following:

Spark/Scala
SQL
Python/PySpark or similar programming language
Databricks
Unity Catalog
ETL/ELT development, monitoring and pipelining using tools such as Apache Airflow
Ingestion tools such as Dell Boomi
Data quality guidelines
CI/CD pipelines
Agile
Git and version control

About Encora

Encora is the preferred digital engineering and modernization partner of some of the world's leading enterprises and digital native companies. With over 9,000 experts in 47+ offices and innovation labs worldwide, Encora's technology practices include Product Engineering & Development, Cloud Services, Quality Engineering, DevSecOps, Data & Analytics, Digital Experience, Cybersecurity, and AI & LLM Engineering.

At Encora, we hire professionals based solely on their skills and qualifications, and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.