Data/Platform Engineer (Remote)

Requirements

Job Overview

The primary goal of developers is efficiency, consistency, scalability and reliability.

 We are responsible for the Platform … all the tooling integrations, security, access-control, data classification/management, orchestration, self-service “lab” concept, observability, reliability … as well as data availability (data ingestion)

 We are NOT responsible for – Data Modeling, Data Warehousing, Reporting (PowerBI)

… although we do work with PBI team for access control from PBI to Snowflake.

 Everything we do is achieved through code – nothing is manual (or ClickOps) – everything is automated – through the effectiveness of our CI/CD framework – GitHub, GitHub-Actions, Terraform, Python.

Orchestration is centrally managed using Managed Airflow

We manage RBAC / Access Control

We’re responsible for Tooling Integrations and all the connectivity and authentication requirements

·         Ingestion Methods/Patterns

o    Fivetran

o    Snowflake-Snowpipe (File-Based sources)

o    Snowflake-Secure Data Share

·         Solid Software Development (Full SDLC) Experience with excellent Coding skills:

o    Python (required).

o    Good knowledge of Git and GitHub (required)

o    Good code-management experience/best practices (required).

·         Understanding of CI/CD to automate and improve the efficiency, speed, and reliability of software delivery.

• Best Practices/principals
  
• Github Actions
  
• automate workflows directly from their GitHub repositories.
  
• Automation of building, testing, and deploying code – inc. code linting, security scanning, and version management.
  
• Experience with testing frameworks
  
• Good knowledge of IaC (Infrastructure as Code) using Terraform (required)
  
• EVERYTHING we do is IaC
  
• Strong verbal and written skills are a must, ideally with the ability to communicate in both technical and some business language
  

• A good level of experience with cloud technologies – AWS - namely S3, Lambda, SQS, SNS, API Gateway (API Development), Networking (VPCs), PrivateLink and Secrets Manager.
  
• Extensive hands-on experience engineering data pipelines and a solid understanding of the full data supply chain, from discovery & analysis, data ingestion, processing & transformation, to consumption/downstream data-integration.
  

·         A passion for continuous improvement and learning, for optimization both in terms of cost and efficiency, as well as ways of working. Obsessed with data observability (aka data reconciliation), ensuring pipeline and data integrity.

• Experience working with large structured/semi-structured datasets
  
• A good understanding of Parquet, Avro, JSON/XML
  
• Experience with Apache Airflow / MWAA or similar orchestration tooling.
  
• Experience with Snowflake as a Data Platform
  
• Solid understanding of Snowflake Architecture – compute, storage, partitioning etc.
  
• Key features - such as COPY-INTO, Snowpipe, object-level tagging and masking policies
  
• RBAC (security model) – design and administration – intermediate skill required
  

o    query performance tuning, and zero copy clone – nice to have

o    virtual warehouse (compute) sizing

• TSQL experience – ability to understand complex queries and think about optimisation - advantageous
  
• Data Modelling experience – advantageous
  
• Exposure to dbt (data build tool) for data transformations – advantageous
  
• Exposure to Alation or other Enterprise Metadata Management (EMM) tooling – advantageous
  
• Documentation: architectural designs, operational procedures, and platform configurations to ensure smooth onboarding and troubleshooting for team members.