National AI Awards 2025Discover AI's trailblazers! Join us to celebrate innovation and nominate industry leaders.

Nominate & Attend

Big Data Lead

FalconSmartIT
London
2 months ago
Applications closed

Related Jobs

View all jobs

Data Scientist Lead - Employee Platforms

Data Science Lead

Lead Data Scientist (Equity Only) - 1%

Lead Data Scientist (Equity Only) - 1%

Lead Data Scientist (Equity Only) - 1%

Lead Data Scientist (Equity Only) - 1%

Job TItle: Big Data Lead

Job Type: Contract

Job Location: Wimbledon , UK


Job Description:

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.

Knowledge in the following areas essential:

Data Engineering Experience:

  • Databricks:Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
  • AWS Cloud:Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
  • IBM Skills:DB2, Datastage, Tivoli Workload Scheduler, Urban Code
  • Programming Languages:Proficiency in Python, SQL.
  • Data Warehousing & ETL:Experience with modern ETL frameworks and data warehousing techniques.
  • DevOps & CI/CD:Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).
  • Familiarity with big data technologies like Apache Spark, Hadoop, or similar.
  • Test automation skills
  • ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores
  • Leadership & Strategy:Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.
  • Customer Data Platform Development:Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.
  • Data Governance & Best Practices:Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.
  • Pipeline Automation & Optimisation:Drive the automation of data pipelines and workflows to improve efficiency and reliability.
  • Team Management:Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.
  • Cross Company Collaboration:Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.
  • Cloud Management:Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.
  • Performance Monitoring:Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.
National AI Awards 2025

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Jobs Careers

Return-to-Work Pathways: Relaunch Your AI Career with Returnships, Flexible & Hybrid Roles

Stepping back into the workplace after a career break can feel like embarking on a whole new journey—especially in a cutting-edge field such as artificial intelligence (AI). For parents and carers, the challenge isn’t just refreshing your technical know-how but also securing a role that respects your family commitments. Fortunately, the UK’s tech sector now boasts a wealth of return-to-work programmes—from formal returnships to flexible and hybrid opportunities. These pathways are designed to bridge the gap, equipping you with refreshed skills, confidence and a supportive network. In this comprehensive guide, you’ll discover how to: Understand the booming demand for AI talent in the UK Leverage transferable skills honed during your break Overcome common re-entry challenges Build your AI skillset with targeted training Tap into returnship and re-entry programmes Find flexible, hybrid and full-time AI roles that suit your lifestyle Balance professional growth with caring responsibilities Master applications, interviews and networking Whether you’re returning after maternity leave, eldercare duties or another life chapter, this article will equip you with practical steps, resources and insider tips.

Jobs

LinkedIn Profile Checklist for AI Jobs: 10 Tweaks That Triple Recruiter Views

In today’s fiercely competitive AI job market, simply having a LinkedIn profile isn’t enough. Recruiters and hiring managers routinely scout for top talent in machine learning, data science, natural language processing, computer vision and beyond—sometimes before roles are even posted. With hundreds of applicants vying for each role, you need a profile that’s optimised for search, speaks directly to AI-specific skills, and showcases measurable impact. By following this step-by-step LinkedIn for AI jobs checklist, you’ll make ten strategic tweaks that can triple recruiter views and position you as a leading AI professional. Whether you’re a fresh graduate aiming for your first AI position or a seasoned expert targeting a senior role, these actionable changes will ensure your profile stands out in feeds, search results and recruiter queues. Let’s dive in.

Education

Part-Time Study Routes That Lead to AI Jobs: Evening Courses, Bootcamps & Online Masters

Artificial intelligence (AI) is reshaping industries at an unprecedented pace. From automating mundane tasks in finance to driving innovation in healthcare diagnostics, the demand for AI-skilled professionals is skyrocketing. In the United Kingdom alone, AI is forecast to deliver over £400 billion to the economy by 2030 and generate millions of new jobs across sectors. Yet, for many ambitious professionals, taking time away from work to upskill can feel like an impossible ask. Thankfully, part-time learning options have proliferated: evening courses, intensive bootcamps and flexible online master’s programmes empower you to learn AI while working. This comprehensive guide explores every route—from short tasters to deep-dive MScs—showcasing providers, course formats, funding options and practical tips. Whether you’re a career changer, a busy manager or a self-taught developer keen to go further, you’ll discover a pathway to fit your schedule, budget and goals.

🍪 We use cookies to enhance your browsing experience on our website. By continuing to use this site, you consent to the use of cookies as described in our Cookie Policy. You can review our Cookie Policy for more information.