Position Summary

We are seeking a highly skilled and experienced Senior Data Engineer with a consulting background to join our team. In this role, you will design, build, and optimise scalable data solutions that drive business insights for our clients. You will work with modern cloud technologies, specifically leveraging Microsoft Azure products, and apply your expertise in SQL, Python, Spark, and Terraform. The ideal candidate has a deep understanding of lakehouse and medallion architecture, data modelling, creating and configuring ETL/ELT pipelines, and securing networks for private connectivity.

This role requires a hands-on engineer who can lead data projects, consult with stakeholders, and deliver high-performance data platforms that meet security, compliance, and business requirements.

The Senior Data Engineer has responsibility for the following:

• Consulting & Stakeholder Collaboration: Work closely with clients and internal teams to gather requirements in technical and pre-sales engagements, define project scope, and deliver data solutions that align with business objectives. You may also be required to lead or support the delivery of technical workshops to customers.
  
  
• Data Pipeline Development & Optimization: Design, implement, and optimise robust ETL/ELT workflows to ingest, transform, and process large volumes of data, utilizing tools like Azure Data Factory, Azure Synapse Analytics, and Databricks.
  
  
• Cloud Platform Expertise: Architect, deploy, and manage data solutions on Azure, including Azure Data Lake, Azure Databricks, Azure Synapse, and Azure Log Analytics.
  
  
• Data Modelling & Architecture: Implement scalable data models and architectures, with a strong focus on lakehouse and medallion architectures to enable efficient and structured data analysis.
  
  
• CI/CD & DevOps: Lead efforts to automate data pipeline deployments using Azure DevOps, CI/CD pipelines, and Terraform for infrastructure-as-code.
  
  
• Networking: Design and configure secure networking for data solutions, ensuring data protection, compliance, and best practices across the stack.
  
  
• Performance Tuning: Monitor and optimise system performance, ensuring high availability and efficient processing times for all data platforms.
  
  
• Cross-functional Collaboration: Work closely with other team members, such as data analysts, analytics engineers, and FinOps analysts, to ensure the delivery of high-quality, reliable, and scalable data products, facilitating seamless access to clean and well-modelled data for reporting and advanced analytics.
  
  About You
  
  Behavioural competencies – Organisational and Behavioural Fit
  
  
• You have a positive mindset: you're excited by unfamiliar challenges and learning new things
  
  
• You’re an enthusiastic self-starter, eager to learn a wide variety of technologies and apply them to real-world customer problems
  
  
• You are comfortable and experienced in talking to and working with senior stakeholders
  
  
• You keep up to date on new technologies and trends
  
  Critical competencies – Technical Fit
  
  
• Extensive experience with Power BI, with understanding of workspace administration, capacity management, and licensing
  
  
• Strong knowledge of SQL and data warehouse and lakehouse design methodologies (i.e. STAR schema approach and medallion architecture); and experience applying this to real-world data modelling scenarios by developing raw data to create databases, tables, and views for analytics
  
  
• Background working with programming (such as Python and/or R) and scripting languages (e.g. PowerShell) for statistical analysis, automation, and working with APIs
  
  
• Comfortable accessing, transforming, analysing, and modelling data using big data frameworks such as Spark with e.g. PySpark or Sparklyr
  
  
• Capable of researching and developing machine learning models and approaches for deploying models into production for real-world applications
  
  
• Understanding of a variety of approaches to data platforms (e.g. Data Lake, Data Mesh, Data Warehouse, streaming, batch processing)
  
  
• Knowledge of security, GDPR and PII data handling principles
  
  Experience with the following technologies is desirable:
  
  
• 5+ years of experience in data engineering or a related role, with consulting experience preferred.
  
  
• Expertise in SQL, Python, and Spark for data processing and analysis.
  
  
• Strong experience with Azure Data services
  
  
• Deep understanding of data warehousing, data lakes, data mesh, lakehouse architecture (with medallion architecture design principles), and streaming and batch processing.
  
  
• Extensive experience with data modeling, ETL/ELT pipeline design, and pipeline optimisation.
  
  
• Proficiency with CI/CD pipelines, Azure DevOps, and infrastructure automation tools like Terraform.
  
  The following would also be beneficial knowledge and experience:
  
  
• Microsoft Fabric and Power BI, including: preparing data for use in semantic models and dataflows, configuring data source connections and dataflows, and deploying and managing data gateways.
  
  
• Deploying and configuring Microsoft Purview and Profisee.
  
  
• Azure Monitor for monitoring and securing data platforms.
  
  
• Familiarity with containerization technologies (e.g., Docker, Kubernetes) in a cloud environment