Big Data Lead

FalconSmartIT
London
1 month ago
Applications closed

Related Jobs

View all jobs

Chief Data Scientist (London Area)

Data Engineer

Geospatial Data Engineer

Head of Data Science

Snr Data Engineer

Senior Data Science Developer

Job TItle: Big Data Lead

Job Type: Contract

Job Location: Wimbledon , UK


Job Description:

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.

Knowledge in the following areas essential:

Data Engineering Experience:

  • Databricks:Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
  • AWS Cloud:Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
  • IBM Skills:DB2, Datastage, Tivoli Workload Scheduler, Urban Code
  • Programming Languages:Proficiency in Python, SQL.
  • Data Warehousing & ETL:Experience with modern ETL frameworks and data warehousing techniques.
  • DevOps & CI/CD:Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).
  • Familiarity with big data technologies like Apache Spark, Hadoop, or similar.
  • Test automation skills
  • ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores
  • Leadership & Strategy:Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.
  • Customer Data Platform Development:Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.
  • Data Governance & Best Practices:Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.
  • Pipeline Automation & Optimisation:Drive the automation of data pipelines and workflows to improve efficiency and reliability.
  • Team Management:Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.
  • Cross Company Collaboration:Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.
  • Cloud Management:Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.
  • Performance Monitoring:Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Rural-Remote AI Jobs: A Breath of Fresh Air in the UK Tech Scene

A New Horizon for AI Professionals For years, conversations around tech careers in the UK have hinged on a central theme: to succeed in artificial intelligence (AI), you must be in or around London (or other big metropolitan areas like Manchester, Bristol, or Edinburgh). But times are changing. Technological leaps and the rise of flexible working are paving the way for AI professionals to live and work well beyond the capital. From the rugged coastlines of Cornwall and Pembrokeshire to the rolling hills of the Yorkshire Dales, we’re witnessing an exciting trend of AI remote countryside roles that allow you to work at the forefront of tech innovation—all while enjoying the tranquillity of rural or seaside living. At ArtificialIntelligenceJobs.co.uk, we’re seeing a marked increase in job postings and applications for these sorts of positions. A growing segment of job seekers is actively searching for “tech jobs by the sea” or “AI remote countryside,” driven by a desire for better work-life balance, lower living costs, and a healthier lifestyle. And it’s not just employees who stand to benefit; employers eager to attract top-tier AI talent are discovering that offering remote or flexible roles widens their candidate pool and enhances diversity. If you’re enticed by the idea of logging off from a day of coding neural networks and taking a stroll along a coastal path—or stepping outside your converted barn in Northumberland to soak in some fresh country air—this article is for you. Below, we’ll explore the benefits and challenges of rural-remote AI jobs, the specific roles best suited for remote work, and how to position yourself for success in this rapidly evolving sector.

When Qubits Fuel Neural Networks: The Emerging Frontier of Quantum-Enhanced AI

Artificial Intelligence (AI) has soared to unimaginable heights over the last few years, revolutionising sectors ranging from healthcare and finance to logistics and entertainment. But while deep learning models have grown more sophisticated and powerful, they remain tied to the limitations of classical computing systems. Enter quantum computing, a burgeoning field that leverages qubits—quantum bits—to achieve processing speeds that could leave even today’s most advanced supercomputers in the dust. What if we combined these two forces? Quantum-enhanced AI aims to integrate quantum hardware and algorithms into AI workflows, potentially unlocking efficiencies and capabilities that are currently out of reach. Although this domain is still in its infancy, experts predict it could reshape entire industries in the not-so-distant future. For professionals in AI, this is more than just an interesting development; it’s a pivotal shift that could spawn new roles, research areas, and opportunities. In this thought-leadership piece, we will: Outline the basics of quantum computing and why it matters to AI. Examine how quantum resources might supercharge neural networks. Highlight the career paths at the intersection of quantum and machine learning. Discuss the long-term outlook and what it means for AI professionals looking to stay ahead. Whether you’re already immersed in AI or just beginning to explore its potential, strap in—this new frontier promises a radical transformation.

AI Jobs at Newly Funded UK Start-ups: Q3 2025 Investment Tracker

Over the last decade, the United Kingdom has firmly established itself as one of Europe’s most significant technology hubs. Thanks to a vibrant ecosystem of venture capital, government-backed initiatives, and a wealth of academic talent, the UK has become especially fertile ground for artificial intelligence (AI) innovation. This growth is not just evident in established tech giants—new start-ups are emerging every quarter with fresh ideas, ground-breaking technologies, and a drive to solve real-world problems. In this Q3 2025 Investment Tracker, we take a comprehensive look at the latest AI start-ups in the UK that have successfully secured funding. Beyond celebrating these companies’ milestones, we’ll explore how these recent investments translate into exciting new job opportunities for AI professionals. Whether you’re an experienced machine learning engineer, a data scientist, or simply hoping to break into the AI sector, this roundup will give you insights into the most in-demand roles, the skills you need to stand out, and how you can capitalise on the current AI hiring boom.