National AI Awards 2025Discover AI's trailblazers! Join us to celebrate innovation and nominate industry leaders.

Nominate & Attend

Big Data Lead

Hirewand
London
4 months ago
Applications closed

Related Jobs

View all jobs

Senior Data Scientist

Lead Data Scientist - Applied Intelligence

Lead Data Scientist - Model Risk Management

Lead Data Scientist

Data Science Lead

Senior Data Scientist

Job Type:

Contract

Considering making an application for this job Check all the details in this job description, and then click on Apply.Job Location:

Wimbledon , UKJob Description :For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how to maintain and deliver world class data pipelines.Knowledge in the following areas essential:Databricks:

Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.AWS Cloud:

Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.IBM Skills:

DB2, Datastage, Tivoli Workload Scheduler, Urban CodeProgramming Languages:

Proficiency in Python, SQL.Data Warehousing & ETL:

Experience with modern ETL frameworks and data warehousing techniques.DevOps & CI/CD:

Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).Familiarity with big data technologies like Apache Spark, Hadoop, or similar.ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data storesLeadership & Strategy:

Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.Customer Data Platform Development:

Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.Data Governance & Best Practices:

Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.Pipeline Automation & Optimisation:

Drive the automation of data pipelines and workflows to improve efficiency and reliability.Team Management:

Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.Cross Company Collaboration:

Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.Cloud Management:

Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.Performance Monitoring: Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.

#J-18808-Ljbffr

National AI Awards 2025

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

AI Jobs Skills Radar 2026: Emerging Frameworks, Languages & Tools to Learn Now

As the UK’s AI sector accelerates towards a £1 trillion tech economy, the job landscape is rapidly evolving. Whether you’re an aspiring AI engineer, a machine learning specialist, or a data-driven software developer, staying ahead of the curve means more than just brushing up on Python. You’ll need to master a new generation of frameworks, languages, and tools shaping the future of artificial intelligence. Welcome to the AI Jobs Skills Radar 2026—your definitive guide to the emerging AI tech stack that employers will be looking for in the next 12–24 months. Updated annually for accuracy and relevance, this guide breaks down the top tools, frameworks, platforms, and programming languages powering the UK’s most in-demand AI careers.

How to Find Hidden AI Jobs in the UK Using Professional Bodies like BCS, IET & the Turing Society

When it comes to job hunting in artificial intelligence (AI), most candidates head straight to traditional job boards, LinkedIn, or recruitment agencies. But what if there was a better way to find roles that aren’t advertised publicly? What if you could access hidden job leads, gain inside knowledge, or get referred by people already in the field? That’s where professional bodies and specialist AI communities come in. In this article, we’ll explore how UK-based organisations like BCS (The Chartered Institute for IT), IET (The Institution of Engineering and Technology), and the Turing Society can help you uncover AI job opportunities you won’t find elsewhere. We'll show you how to strategically use their directories, special-interest groups (SIGs), and CPD (Continuing Professional Development) events to elevate your career and expand your AI job search in ways most job seekers overlook.

How to Get a Better AI Job After a Lay-Off or Redundancy

Being made redundant or laid off can feel like the rug has been pulled from under you. Whether part of a wider company restructuring, budget cuts, or market shifts in tech, many skilled professionals in the AI industry have recently found themselves unexpectedly jobless. But while redundancy brings immediate financial and emotional stress, it can also be a powerful catalyst for career growth. In the fast-evolving field of artificial intelligence, where new roles and specialisms emerge constantly, bouncing back stronger is not only possible—it’s likely. In this guide, we’ll walk you through a step-by-step action plan for turning redundancy into your next big opportunity. From managing the shock to targeting better AI jobs, updating your CV, and approaching recruiters the smart way, we’ll help you move from setback to comeback.