Lead Data Engineer

Canada Life
London, United Kingdom
3 weeks ago
Applications closed

Related Jobs

View all jobs
Spotlight

Forward Deployed Engineer

SolveAI London, United Kingdom
Hybrid

Data Science Lead - Fulfilment

Ocado United Kingdom
Hybrid

Senior Data Engineer, Core Experimentation

OpenAI United Kingdom
£293,000 – £325,000 pa Hybrid

Lead Data Scientist

Faculty AI London, United Kingdom
Hybrid

Lead Data Scientist - Customer Development

Faculty AI London, United Kingdom
Hybrid

Lead Data Scientist

Faculty AI London, United Kingdom
Hybrid
Job Type
Permanent
Work Pattern
Full-time
Work Location
Hybrid
Seniority
Lead
Education
Degree
Posted
30 Apr 2026 (3 weeks ago)

Location: London, Watford or Bristol (Hybrid working options available)

The Lead Data Engineer will provide hands-on technical leadership in Azure cloud and Databricks-based solutions within our Enterprise Data Platform. The role requires strong expertise in Azure cloud services, Databricks, data engineering, and DevOps, leading a cross-functional team to build, deploy, and support high-performance data-driven solutions.

The role involves:

  • Interpreting Outcomes and user stories and translating them into technical solutions.
  • Creating innovative solution designs for domain and enterprise data products.
  • Overseeing Data Analysts to support detailed data discovery.
  • Overseeing data modelling for Finance and Enterprise data products.
  • Leading Product Increment planning to break down solutions into Features and Epics for incremental delivery.
  • Designing and implementing scalable data solutions on Azure and Databricks within the assigned domain.
  • Ensuring appropriate engineering standards are applied to maintain data quality, performance and reliability.

Duties/Responsibilities

  • Work with Product Owners and Business Analysts to understand Outcomes, refine user stories.
  • Lead solution design for Finance and Enterprise data products, ensuring alignment with enterprise patterns and guardrails.
  • Direct and collaborate with Data Analysts on detailed data discovery, source understanding and requirements refinement.
  • Oversee logical and physical data modelling for Finance and Enterprise data products, working closely with architecture where required.
  • Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake).
  • Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established patterns.
  • Apply security, access control and compliance standards for Azure and Databricks in collaboration with platform and security teams.
  • Support monitoring, logging and basic cost optimisation for the team’s data products.
  • Support the development of DevOps practices within the team, including reducing technical debt and improving automation over time.

Skills, Knowledge and Experience

Lead Data Engineers are expected to have strong capability in at least three of the following areas of engineering practice.

Core skills

  • Automation including testing of data pipelines and data products.
  • Strong teamwork, communication and problem-solving skills to collaborate effectively with cross-functional teams.
  • Awareness of security principles and best practices to ensure secure data solutions.
  • Commitment to continuous learning and staying current with Azure, Databricks and data engineering trends.
  • Strong experience working within an agile development methodology, ideally Scaled Agile (SAFe or similar).
  • Excellent time and self-management through effective planning and prioritisation of tasks.
  • Proven and demonstrable data engineering capability.
  • Ability to influence within the team and communicate clearly with technical and non-technical stakeholders.

Data Engineer (New Technology / Microsoft)

  • Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous).
  • Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous).
  • Experience contributing to CI/CD pipelines (Azure DevOps, GitHub Actions, Terraform).
  • Scripting and programming skills (Python advantageous).
  • Good understanding of DevOps and automation concepts (e.g. YAML pipelines, IaC).
  • Solid understanding of cloud security, compliance and governance principles.
  • Experience working with Databricks and Azure in a product or Scaled Agile delivery environment.

Qualifications

  • Degree level IT or technical/scientific subject (or equivalent experience).
  • Microsoft Azure Data Engineer or Solutions Architect certification (desirable).
  • Databricks Certified Data Engineer or Machine Learning Associate (desirable).
  • Experience with streaming solutions (Kafka, Event Hubs, Spark Streaming) (desirable).
  • Knowledge of machine learning and AI on Databricks (desirable).

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Where to Advertise AI Jobs in the UK (2026 Guide)

Where to advertise AI jobs UK in 2026: the specialist boards and communities that reach AI engineers, ML scientists and applied research talent in the UK. The candidate pool is small, highly informed and in demand across multiple sectors simultaneously. General job boards reach a broad audience but lack the specificity that AI professionals expect — and the filtering mechanisms they rely on. Specialist platforms, direct outreach and academic channels each serve a different part of the market. This guide, published by ArtificialIntelligenceJobs.co.uk, covers where to advertise AI roles in the UK in 2026, how the main platforms compare, what employers should expect to pay, and what the data says about time-to-hire across different role types.

AI Jobs UK 2026: What to Expect Over the Next 3 Years

AI Jobs UK 2026: roles, salaries and the generative AI, machine learning and applied AI hiring trends shaping UK artificial intelligence careers. Artificial intelligence is creating jobs faster than the market can name them. New roles are appearing every quarter, existing titles are splitting into specialisms, and the technologies underpinning it all are evolving at a pace that makes even last year's job descriptions feel dated. For job seekers, this presents a genuinely unusual challenge. In most industries, career planning means understanding a relatively stable landscape and working out where you fit within it. In AI, the landscape itself is being redrawn in real time. The roles with the most hiring activity in 2028 may not yet have a widely agreed job title in 2026. That's not a reason to feel overwhelmed — it's a reason to get informed. The candidates who thrive in this market aren't necessarily those with the longest CVs or the most credentials. They're the ones who understand the direction of travel: which skills are gaining value, which technologies are driving employer decisions, and how the definition of an "AI job" is expanding well beyond the tech sector. This article breaks down what the UK AI jobs market is likely to look like over the next three years — covering emerging job titles, the technologies reshaping hiring, the skills employers are prioritising, and how to position yourself ahead of the curve rather than behind it.