JOB DESCRIPTION
Your Mission
We are seeking a highly skilled Senior Data Engineer to join our team and play a crucial role in developing and managing critical components of our data platform like Data API (Rest API), Apache Airflow and data engineering artifacts like data ingestion and data curation pipelines. This role demands a strong proficiency in Python, Apache Airflow, REST APIs, PySpark and Azure cloud services.
For this opportunity, fluent English is essential, as the professional will work with people from all over the world on a daily basis.
We are looking for people who reside in Europe, in locations that share the same time zone as the United Kingdom.
It will be necessary to have a Brazilian CNPJ or be available to issue an invoice for payment.
RESPONSIBILITIES AND ASSIGNMENTS
Activities
- Work closely with the Data Engineering Lead to devise effective data ingestion strategies aligned with business objectives.
- Define and implement robust data ingestion patterns and processes to ensure efficient and reliable data flow into the organization's data platform.
- Develop and maintain reusable data engineering or ETL pipelines and codebase using Python, PySpark, Databricks, Airflow, REST APIs, and the Azure cloud platform.
- Design and implement robust data APIs using Python frameworks such as FastAPI or Flask and deploy them on Azure App Service.
- Collaborate with cross-functional teams to understand data requirements and provide scalable data engineering solutions.
- Design and implement batch and streaming data architectures leveraging Azure cloud services like Azure Data Factory and Azure Databricks.
- Contribute to the development and deployment of Apache Airflow on Azure Kubernetes Service (AKS).
- Ensure adherence to software engineering best practices, including version control, testing, and continuous integration/continuous deployment (CI/CD) processes.
- Participate in code reviews, technical discussions, and knowledge-sharing sessions within the team.
REQUIREMENTS AND QUALIFICATIONS
Technical Requirements:
- Good experience in Python programming language.
- Proven experience with Apache Airflow and its deployments on Azure Kubernetes Service.
- Hands-on experience with Python frameworks like FastAPI or Flask and deploying REST APIs on Azure App Service.
- Strong understanding of core data engineering concepts and principles.
- Solid experience with Azure cloud services, including Azure Data Factory and Azure Databricks.
- Proficiency in writing PySpark code for data processing and transformation.
- Ability to design and implement reusable and scalable data engineering pipelines and codebase.
- Sound understanding of software engineering development practices, including version control, testing, and continuous integration/continuous deployment (CI/CD).
- Experience in designing batch and streaming data architectures using Azure cloud platform services.
- Effective problem-solving skills and the ability to troubleshoot complex data engineering issues.
- Commitment to continuous learning and staying updated with industry trends and best practices in data engineering.
- Strong communication and collaboration skills with English language proficiency. Ability to work effectively in a team environment, provide training, and document processes.
Differentiator:
- Familiarity with Python tools like Pytest, tox, poetry.
- Familiarity with Python testing frameworks like pytest and build tools like tox and poetry.
- Experience in building CI/CD pipelines and working with Git-based version control systems.
- Knowledge of containerization technologies such as Docker and Kubernetes.
Behavioral Requirements:
- Effective problem-solving skills and the ability to troubleshoot complex data engineering issues.
- Commitment to continuous learning and staying updated with industry trends and best practices in data engineering.
- Strong communication and collaboration skills with English language proficiency. Ability to work effectively in a team environment, provide training, and document processes.
ADDITIONAL INFORMATION
Advantages of service provision:
🧡 Anywhere office culture: we like to have the freedom to work with autonomy, flexibility, and trust;
🍀 Psychological support: for taking care of our mental health;
🎓 Certification assistance;
🎓 FIAP: Undergraduate and postgraduate courses - 10% OFF on undergraduate courses and 20% OFF on MBA, POSTECH, and SHIFT courses, both in-person and online.
SER #teamMGInfo é ser especialista em dados
Nossa missão é potencializar os resultados das empresas para a tomada de decisões orientadas a dados, de forma confiável, ágil, inovadora e com alto valor agregado.
Para esse desafio, contamos com um time incrível de especialistas em dados que atuam em projetos globais, conectando conhecimento com parcerias de sucesso e soluções personalizadas para a necessidade de cada cliente.
Vem conhecer um pouco mais sobre nossa atuação aqui.
Nosso DNA Data Driven se baseia em valores que inspiram nosso ambiente e ideais:
- Bem-Estar: acolhemos e cuidamos do bem-estar das pessoas.
- Aprendizado Contínuo: somos incansáveis na arte de aprender. De forma contínua, estimulamos nosso time a desenvolver novas habilidades, aprender novos conceitos e propor novas soluções.
- Colaboração: temos um forte senso de comunidade (e camaradagem), somos orientados para o compartilhamento de aprendizado e ajuda mútua.
- Agilidade: aprendemos com as mudanças, nos adaptamos rapidamente aproveitando as oportunidades e superando os desafios.
- Orientação a Dados: nos guiamos por dados confiáveis e relevantes para perseguir nosso objetivo de impulsionar a tomada de decisão de forma mais assertiva e eficiente.
PESSOAS INCRÍVEIS FAZEM UM LUGAR INCRÍVEL!
Orgulho de ser e pertencer. Aqui na MG Info nos comprometemos a criar, juntos, um ambiente plural e de acolhimento com oportunidades para todos os talentos se desenvolverem e crescerem conosco!
Já deu match? Então vem ser #teamMGINFO!
