Be at the heart of actionFly remote-controlled drones into enemy territory to gather vital information.

Apply Now

Big Data Lead

FalconSmartIT
London
5 months ago
Applications closed

Related Jobs

View all jobs

Senior / Lead Data Scientist - AI Agents - Outside IR35

Lead Data Scientist - Remote

Lead Data Scientist

Lead Data Scientist - Remote

Lead R Data Scientist - Sustainability

Lead R Engineer / Data Scientist - Integrated Pest Management (IPM)

Job TItle: Big Data Lead

Job Type: Contract

Job Location: Wimbledon , UK


Job Description:

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.

Knowledge in the following areas essential:

Data Engineering Experience:

  • Databricks:Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
  • AWS Cloud:Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
  • IBM Skills:DB2, Datastage, Tivoli Workload Scheduler, Urban Code
  • Programming Languages:Proficiency in Python, SQL.
  • Data Warehousing & ETL:Experience with modern ETL frameworks and data warehousing techniques.
  • DevOps & CI/CD:Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).
  • Familiarity with big data technologies like Apache Spark, Hadoop, or similar.
  • Test automation skills
  • ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores
  • Leadership & Strategy:Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.
  • Customer Data Platform Development:Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.
  • Data Governance & Best Practices:Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.
  • Pipeline Automation & Optimisation:Drive the automation of data pipelines and workflows to improve efficiency and reliability.
  • Team Management:Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.
  • Cross Company Collaboration:Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.
  • Cloud Management:Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.
  • Performance Monitoring:Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Jobs

Why the UK Could Be the World’s Next AI Jobs Hub

Artificial Intelligence (AI) has rapidly moved from research labs into boardrooms, classrooms, hospitals, and homes. It is already reshaping economies and transforming industries at a scale comparable to the industrial revolution or the rise of the internet. Around the world, countries are competing fiercely to lead in AI innovation and reap its economic, social, and strategic benefits. The United Kingdom is uniquely positioned in this race. With a rich heritage in computing, world-class universities, forward-thinking government policy, and a growing ecosystem of startups and enterprises, the UK has many of the elements needed to become the world’s next AI hub. Yet competition is intense, particularly from the United States and China. Success will depend on how effectively the UK can scale its strengths, close its gaps, and seize opportunities in the years ahead. This article explores why the UK could be the world’s next global hub for artificial intelligence, what challenges it must overcome, and what this means for businesses, researchers, and job seekers.

Careers Education

The Best Free Tools & Platforms to Practise AI Skills in 2025/26

Artificial Intelligence (AI) is one of the fastest-growing career fields in the UK and worldwide. Whether you are a student exploring AI for the first time, a graduate looking to build your portfolio, or an experienced professional upskilling for career growth, having access to free tools and platforms to practise AI skills can make a huge difference. In this comprehensive guide, we’ll explore the best free resources available in 2025, covering AI coding platforms, datasets, cloud tools, no-code AI platforms, online communities, and learning hubs. These tools allow you to practise everything from machine learning models and natural language processing (NLP) to computer vision, reinforcement learning, and large language model (LLM) fine-tuning—without needing a huge budget. By the end of this article, you’ll have a clear roadmap of where to start practising your AI skills for free, how to build real-world projects, and which platforms can help you land your next AI job.

Jobs

Top 10 Skills in Artificial Intelligence According to LinkedIn & Indeed Job Postings

Artificial intelligence is no longer a niche field reserved for research labs or tech giants—it has become a cornerstone of business strategy across the UK. From finance and healthcare to manufacturing and retail, employers are rapidly expanding their AI teams and competing for talent. But here’s the challenge: AI is evolving so quickly that the skills in demand today may look different from those of just a few years ago. Whether you’re a graduate looking to enter the industry, a mid-career professional pivoting into AI, or an experienced engineer wanting to stay ahead, it’s essential to know what employers are actually asking for in their job ads. That’s where platforms like LinkedIn and Indeed provide valuable insight. By analysing thousands of job postings across the UK, they reveal the most frequently requested skills and emerging trends. This article distils those findings into the Top 10 AI skills employers are prioritising in 2025—and shows you how to present them effectively on your CV, in interviews, and in your portfolio.

🍪 We use cookies to enhance your browsing experience on our website. By continuing to use this site, you consent to the use of cookies as described in our Cookie Policy. You can review our Cookie Policy for more information.