Big Data Lead

FalconSmartIT
London
1 month ago
Applications closed

Related Jobs

View all jobs

Data Scientist Lead - Employee Platforms

Senior Data Scientist

Junior Data Scientist

Senior Data Scientist

Senior Data Architect

Snr Data Engineer

Job TItle: Big Data Lead

Job Type: Contract

Job Location: Wimbledon , UK


Job Description:

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.

Knowledge in the following areas essential:

Data Engineering Experience:

  • Databricks:Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
  • AWS Cloud:Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
  • IBM Skills:DB2, Datastage, Tivoli Workload Scheduler, Urban Code
  • Programming Languages:Proficiency in Python, SQL.
  • Data Warehousing & ETL:Experience with modern ETL frameworks and data warehousing techniques.
  • DevOps & CI/CD:Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).
  • Familiarity with big data technologies like Apache Spark, Hadoop, or similar.
  • Test automation skills
  • ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores
  • Leadership & Strategy:Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.
  • Customer Data Platform Development:Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.
  • Data Governance & Best Practices:Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.
  • Pipeline Automation & Optimisation:Drive the automation of data pipelines and workflows to improve efficiency and reliability.
  • Team Management:Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.
  • Cross Company Collaboration:Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.
  • Cloud Management:Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.
  • Performance Monitoring:Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Top 10 Best UK Universities for AI Degrees (2025 Guide)

Discover the ten best UK universities for Artificial Intelligence degrees in 2025. Compare entry requirements, course content, research strength and industry links to choose the right AI programme for you. Artificial Intelligence continues to transform industries—from healthcare to finance to transportation. The UK leads the way in AI research and education, with several universities consistently ranked among the world’s best for Computer Science. Below, we spotlight ten UK institutions offering strong AI-focused programmes at undergraduate or postgraduate level. While league tables shift year to year, these universities have a track record of excellence in teaching, research, and industry collaboration.

How to Write a Winning Cover Letter for AI Jobs: Proven 4-Paragraph Structure

Learn how to craft the perfect cover letter for AI jobs with this proven 4-paragraph structure. Perfect for junior developers and career switchers. When applying for an AI job, your cover letter can make all the difference. For many, the process of writing a cover letter for an AI position can be daunting, especially when there are so few specific guides for tailoring it to the industry. However, a clear, effective structure combined with AI-specific language and examples can help you stand out from the competition. Whether you're a junior entering the field or a mid-career professional switching to AI, the following framework will make it easier for you to craft a compelling cover letter. In this article, we’ll take you through a proven four-paragraph structure that works and provide sample lines that you can adapt to your personal experience.

Veterans in Tech: A Military‑to‑Civilian Pathway into AI Jobs

Published on ArtificialIntelligenceJobs.co.uk – empowering the UK talent pipeline for artificial intelligence, data, and robotics. Introduction Leaving the Armed Forces is both a proud milestone and a daunting leap into the unknown. Whether you served with the Royal Navy, British Army, Royal Air Force or Royal Marines, one thing is certain: the skills you forged under pressure are in high demand—especially in the booming field of artificial intelligence (AI).  The UK’s AI market is expected to contribute £400 billion to the economy by 2030, with defence and security applications at its core. Employers from start‑ups to FTSE‑100 giants are crying out for disciplined professionals who understand mission‑critical environments. Ex‑service personnel fit the bill perfectly. This guide maps the military‑to‑civilian journey, signposts Ministry of Defence (MoD) transition programmes, and shows you exactly how to land your first AI role. Quick Win: Bookmark our live listings for Machine Learning Engineer roles to see which employers are hiring right now.