Match score not available

Data Ingestion Engineer

Remote:

Full Remote

Contract:

Full time

Experience:

Mid-level (2-5 years)

Work from:

Mexico

Offer summary

Qualifications:

Bachelor's degree in a related field, 3+ years of experience in data engineering, Strong knowledge of data lake ingestion, Hands-on experience with Spark/Scala, SQL, Python, Expertise in CI/CD and version control systems like Git.

Key responsabilities:

Ingest data from various source systems
Manage and optimize ETL/ELT pipelines on Databricks
Document workflows and integration flows
Schedule and automate ingestion jobs using Apache Airflow
Ensure compliance with data quality and security guidelines

Encora Inc. XLarge http://www.encora.com/

5001 - 10000 Employees

See more Encora Inc. offers

Job description

Important Information:

Years of Experience: 3+ years of experience in data engineering, ingestion pipelining, and ETL/ELT.
Job Mode: Full-time.
Work Mode: Remote.

Job Summary:

The Data Ingestion Engineer will be responsible for managing data ingestion processes, platform management, and optimizing ETL/ELT pipelines on the Databricks platform. The role involves ensuring compliance with data quality, security, and PII management standards while utilizing system integration tools and CI/CD methodologies. This position also focuses on documenting workflows, managing metadata, and supporting the overall data pipeline infrastructure.

Responsibilities and Duties:

Ingest data from various source systems, adapting ingestion strategies as needed.
Manage and oversee ETL/ELT pipelines on the Databricks platform.
Optimize data pipelines for scalability, speed, and performance.
Document ingestion, integration flows, and pipelines for clarity and future reference.
Schedule and automate ingestion jobs using Apache Airflow.
Manage metadata and master data within the technical data catalog.
Ensure adherence to security, compliance guidelines, and PII management during data ingestion.
Maintain pipeline infrastructure and implement automated monitoring strategies.
Follow SDLC best practices to ensure quality and consistency.

Qualifications and Skills:

Bachelor's degree in Computer Science, Engineering, Statistics, or a related field.
3+ years of experience in data engineering, ingestion pipelines, and ETL/ELT processes.
Strong knowledge of data lake ingestion processes and best practices.
Expertise in CI/CD pipelines and version control systems like Git.
Proficient in agile methodologies.

Role-specific Requirements:

Hands-on experience with Spark/Scala, SQL, and Python/PySpark.
Proficiency in working with Databricks and Unity Catalog.
Experience with ETL/ELT development, monitoring, and pipelining tools such as Apache Airflow.
Knowledge of ingestion tools such as Dell Boomi.
Strong understanding of data quality guidelines and best practices for managing large data sets.

Technologies:

Databricks, Spark/Scala, SQL, Python/PySpark.
Apache Airflow, Unity Catalog, Dell Boomi.
CI/CD tools, Git, and version control.

Skillset Competencies:

Data pipeline management and optimization.
ETL/ELT pipeline development and monitoring.
System integration and metadata management.
Compliance with security and PII guidelines.
Automated monitoring and CI/CD implementation.

About Encora:

Encora is the preferred digital engineering and modernization partner of some of the world's leading enterprises and digital-native companies. With over 9,000 experts in 47+ offices and innovation labs worldwide, Encora's technology practices include Product Engineering & Development, Cloud Services, Quality Engineering, DevSecOps, Data & Analytics, Digital Experience, Cybersecurity, and AI & LLM Engineering.

At Encora, we hire professionals based solely on their skills and qualifications, and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.