Big Data Lead

FalconSmartIT
London
4 weeks ago
Applications closed

Related Jobs

View all jobs

Head of Data Science

Lead Data Scientist - UK 12 Month FTC

Senior Data Scientist

Chief Data Scientist

Data Scientist

Data Engineer

Job TItle: Big Data Lead

Job Type: Contract

Job Location: Wimbledon , UK


Job Description:

For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how D&G maintain and deliver world class data pipelines.

Knowledge in the following areas essential:

Data Engineering Experience:

  • Databricks:Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.
  • AWS Cloud:Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.
  • IBM Skills:DB2, Datastage, Tivoli Workload Scheduler, Urban Code
  • Programming Languages:Proficiency in Python, SQL.
  • Data Warehousing & ETL:Experience with modern ETL frameworks and data warehousing techniques.
  • DevOps & CI/CD:Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).
  • Familiarity with big data technologies like Apache Spark, Hadoop, or similar.
  • Test automation skills
  • ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data stores
  • Leadership & Strategy:Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.
  • Customer Data Platform Development:Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.
  • Data Governance & Best Practices:Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.
  • Pipeline Automation & Optimisation:Drive the automation of data pipelines and workflows to improve efficiency and reliability.
  • Team Management:Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.
  • Cross Company Collaboration:Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.
  • Cloud Management:Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.
  • Performance Monitoring:Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Jobs

Portfolio Projects That Get You Hired for AI Jobs (With Real GitHub Examples)

In the fast-evolving world of artificial intelligence (AI), an impressive portfolio of projects can act as your passport to landing a sought-after role. Even if you’ve aced interviews in the past, employers in AI and machine learning (ML) are increasingly asking candidates to demonstrate hands-on experience through the projects they’ve built and shared online. This is because practical ability often speaks volumes about your suitability for a role—far more than any exam or certification alone could. In this article, we’ll explore how to build an outstanding AI portfolio that catches the eye of recruiters and hiring managers, including: Why an AI portfolio is crucial for job seekers. How to choose AI projects that align with your target roles. Specific project ideas and real GitHub examples to help you stand out. Best practices for showcasing your work, from writing clear READMEs to using Jupyter notebooks effectively. Tips on structuring your GitHub so that employers can instantly see your value. Moreover, we’ll discuss how you can use your portfolio to connect with top employers in AI, with a handy link to our CV-upload page on Artificial Intelligence Jobs for when you’re ready to apply. By the end, you’ll have a clear roadmap to building a portfolio that will help secure interviews—and the AI job—of your dreams.

Jobs Careers

AI Job Interview Warm‑Up: 30 Real Coding & System‑Design Questions

In today's competitive AI job market, nailing a technical interview can be the difference between landing your dream role and getting lost in the crowd. Whether you're looking to break into machine learning, deep learning, NLP (Natural Language Processing), or data science, your problem-solving skills and system design expertise are certain to be put to the test. AI‑related job interviews typically involve a range of coding challenges, algorithmic puzzles, and system design questions. You’ll often be asked to delve into the principles of machine learning pipelines, discuss how to optimise large-scale systems, and demonstrate your coding proficiency in languages like Python, C++, or Java. Adequate preparation not only boosts your confidence but also reduces the likelihood of fumbling through unfamiliar territory. If you’re actively seeking positions at major tech companies or innovative AI start-ups, then check out www.artificialintelligencejobs.co.uk for some of the latest vacancies in the UK. Meanwhile, this blog post will guide you through 30 real coding & system-design questions you’re likely to encounter during your AI job interview. This list is designed to help you practise, anticipate typical question patterns, and stay ahead of the competition. By reading through each question and thinking about the possible approaches, you’ll sharpen your problem-solving skills, time management, and critical thinking. Each question covers fundamental concepts that employers regularly test, ensuring you’re well-equipped for success. Let’s dive right in.

Jobs

Negotiating Your AI Job Offer: Equity, Bonuses & Perks Explained

Artificial intelligence (AI) has proven itself to be one of the most transformative forces in today’s business world. From smart chatbots in customer service to predictive analytics in finance, AI technologies are reshaping how organisations operate and innovate. As the demand for AI professionals grows, so does the complexity of compensation packages. If you’re a mid‑senior AI professional, you’ve likely seen job offers that include far more than just a base salary—think equity, bonuses, and a range of perks designed to entice you into joining or staying with a company. For many, the focus remains squarely on salary. While that’s understandable—after all, your monthly take‑home pay is what covers day-to-day expenses—limiting your negotiations to salary alone can leave considerable value on the table. From stock options in ambitious startups to sign‑on bonuses that ‘buy you out’ of your current contract, modern AI job offers often include elements that can significantly boost your long-term wealth and job satisfaction. This article aims to shed light on the full scope of AI compensation—specifically focusing on how equity, bonuses, and perks can enhance (or sometimes detract from) the overall value of your package. We’ll delve into how these elements work in practice, what to watch out for, and how to navigate the negotiation process effectively. Our goal is to provide mid‑senior AI professionals with the insights and tools to land a holistic compensation deal that accurately reflects their technical expertise, leadership potential, and strategic importance in this fast-moving field. Whether you’re eyeing a leadership role in machine learning at an established tech giant, or you’re considering a pioneering position at a disruptive AI startup, the knowledge in this guide will help you weigh the merits of base salary alongside the potential riches—and risks—of equity, bonuses, and other benefits. By the end, you’ll have a clearer sense of how to align your compensation with both your immediate lifestyle needs and long-term career aspirations.

🍪 We use cookies to enhance your browsing experience on our website. By continuing to use this site, you consent to the use of cookies as described in our Cookie Policy. You can review our Cookie Policy for more information.