Jobs

Senior Data Engineer


Job details
  • London
  • 2 months ago
Applications closed

Position Summary

We are seeking a highly skilled and experienced Senior Data Engineer with a consulting background to join our team. In this role, you will design, build, and optimise scalable data solutions that drive business insights for our clients. You will work with modern cloud technologies, specifically leveraging Microsoft Azure products, and apply your expertise in SQL, Python, Spark, and Terraform. The ideal candidate has a deep understanding of lakehouse and medallion architecture, data modelling, creating and configuring ETL/ELT pipelines, and securing networks for private connectivity.

This role requires a hands-on engineer who can lead data projects, consult with stakeholders, and deliver high-performance data platforms that meet security, compliance, and business requirements.

The Senior Data Engineer has responsibility for the following:

  • Consulting & Stakeholder Collaboration: Work closely with clients and internal teams to gather requirements in technical and pre-sales engagements, define project scope, and deliver data solutions that align with business objectives. You may also be required to lead or support the delivery of technical workshops to customers.

  • Data Pipeline Development & Optimization: Design, implement, and optimise robust ETL/ELT workflows to ingest, transform, and process large volumes of data, utilizing tools like Azure Data Factory, Azure Synapse Analytics, and Databricks.

  • Cloud Platform Expertise: Architect, deploy, and manage data solutions on Azure, including Azure Data Lake, Azure Databricks, Azure Synapse, and Azure Log Analytics.

  • Data Modelling & Architecture: Implement scalable data models and architectures, with a strong focus on lakehouse and medallion architectures to enable efficient and structured data analysis.

  • CI/CD & DevOps: Lead efforts to automate data pipeline deployments using Azure DevOps, CI/CD pipelines, and Terraform for infrastructure-as-code.

  • Networking: Design and configure secure networking for data solutions, ensuring data protection, compliance, and best practices across the stack.

  • Performance Tuning: Monitor and optimise system performance, ensuring high availability and efficient processing times for all data platforms.

  • Cross-functional Collaboration: Work closely with other team members, such as data analysts, analytics engineers, and FinOps analysts, to ensure the delivery of high-quality, reliable, and scalable data products, facilitating seamless access to clean and well-modelled data for reporting and advanced analytics.

    About You

    Behavioural competencies – Organisational and Behavioural Fit

  • You have a positive mindset: you're excited by unfamiliar challenges and learning new things

  • You’re an enthusiastic self-starter, eager to learn a wide variety of technologies and apply them to real-world customer problems

  • You are comfortable and experienced in talking to and working with senior stakeholders

  • You keep up to date on new technologies and trends

    Critical competencies – Technical Fit

  • Extensive experience with Power BI, with understanding of workspace administration, capacity management, and licensing

  • Strong knowledge of SQL and data warehouse and lakehouse design methodologies (i.e. STAR schema approach and medallion architecture); and experience applying this to real-world data modelling scenarios by developing raw data to create databases, tables, and views for analytics

  • Background working with programming (such as Python and/or R) and scripting languages (e.g. PowerShell) for statistical analysis, automation, and working with APIs

  • Comfortable accessing, transforming, analysing, and modelling data using big data frameworks such as Spark with e.g. PySpark or Sparklyr

  • Capable of researching and developing machine learning models and approaches for deploying models into production for real-world applications

  • Understanding of a variety of approaches to data platforms (e.g. Data Lake, Data Mesh, Data Warehouse, streaming, batch processing)

  • Knowledge of security, GDPR and PII data handling principles

    Experience with the following technologies is desirable:

  • 5+ years of experience in data engineering or a related role, with consulting experience preferred.

  • Expertise in SQL, Python, and Spark for data processing and analysis.

  • Strong experience with Azure Data services

  • Deep understanding of data warehousing, data lakes, data mesh, lakehouse architecture (with medallion architecture design principles), and streaming and batch processing.

  • Extensive experience with data modeling, ETL/ELT pipeline design, and pipeline optimisation.

  • Proficiency with CI/CD pipelines, Azure DevOps, and infrastructure automation tools like Terraform.

    The following would also be beneficial knowledge and experience:

  • Microsoft Fabric and Power BI, including: preparing data for use in semantic models and dataflows, configuring data source connections and dataflows, and deploying and managing data gateways.

  • Deploying and configuring Microsoft Purview and Profisee.

  • Azure Monitor for monitoring and securing data platforms.

  • Familiarity with containerization technologies (e.g., Docker, Kubernetes) in a cloud environment

Sign up for our newsletter

The latest news, articles, and resources, sent to your inbox weekly.

Similar Jobs

Senior Data Engineer

Senior Data Engineer 🌟Are you ready to take your career to the next level? We're looking for a Senior Data Engineer to lead the way in transforming data into actionable insights. If you’re passionate about innovation and thrive in a fast-paced, collaborative environment, this is the role for you!🔍 What...

Watford

Senior Data Engineer – Eligibe for SC – AWS or Azure - Belfast

Senior / Lead Data Engineer – Consultancy – Eligible for SC Clearance – AWS or Azure - BelfastI’m looking for an experienced Senior / Lead Data Engineer to join a successful, multinational Consultancy in their Belfast office working on high profile client projects.As a Senior Data Engineer, you'll design and...

Belfast

Senior Data Engineer (Python, PySpark) - Remote

Senior Data Engineer (Python, PySpark) - Remote(Data Engineer, Senior Data Engineer, Python, PySpark, SQL, C#, Big Data, Databricks, R, Machine Learning, AI, Agile, Scrum, TDD, BDD, CI / CD, SOLID principles, Github, Azure DevOps, Jenkins, Terraform, AWS CDK, AWS CloudFormation, Azure, Senior Data Engineer, Data Engineer)Our client is a global...

Noir Altrincham

Senior Data Engineer (Python, PySpark) - Remote

Senior Data Engineer (Python, PySpark) - Remote(Data Engineer, Senior Data Engineer, Python, PySpark, SQL, C#, Big Data, Databricks, R, Machine Learning, AI, Agile, Scrum, TDD, BDD, CI / CD, SOLID principles, Github, Azure DevOps, Jenkins, Terraform, AWS CDK, AWS CloudFormation, Azure, Senior Data Engineer, Data Engineer)Our client is a global...

Noir Manchester

Senior Data Engineer (Python, PySpark) - Remote

Senior Data Engineer (Python, PySpark) - Remote(Data Engineer, Senior Data Engineer, Python, PySpark, SQL, C#, Big Data, Databricks, R, Machine Learning, AI, Agile, Scrum, TDD, BDD, CI / CD, SOLID principles, Github, Azure DevOps, Jenkins, Terraform, AWS CDK, AWS CloudFormation, Azure, Senior Data Engineer, Data Engineer)Our client is a global...

Noir Bolton

Senior Data Engineer

Data Engineer - Bristol. Due to growing demand for my clients' Data and Analytics services, we are seeking to recruit a number of senior data engineersThe RoleAs part of the role you will:Work with developers, managers, and business stakeholders to understand and define components of the data landscape and how...

Sanderson Bristol