Big Data Lead

Hirewand
London
1 week ago
Applications closed

Related Jobs

View all jobs

Lead Data Engineer

Lead Data Scientist - UK 12 Month FTC

Data Architect

Data Scientist Team Leader - BIG DATA

Enterprise Data Governance & Architecture Lead

Senior Data Engineer

Job Type:

Contract

Considering making an application for this job Check all the details in this job description, and then click on Apply.Job Location:

Wimbledon , UKJob Description :For this role, senior experience of Data Engineering and building automated data pipelines on IBM Datastage & DB2, AWS and Databricks from source to operational databases through to curation layer is expected using the latest cloud modern technologies where experience of delivering complex pipelines will be significantly valuable to how to maintain and deliver world class data pipelines.Knowledge in the following areas essential:Databricks:

Expertise in managing and scaling Databricks environments for ETL, data science, and analytics use cases.AWS Cloud:

Extensive experience with AWS services such as S3, Glue, Lambda, RDS, and IAM.IBM Skills:

DB2, Datastage, Tivoli Workload Scheduler, Urban CodeProgramming Languages:

Proficiency in Python, SQL.Data Warehousing & ETL:

Experience with modern ETL frameworks and data warehousing techniques.DevOps & CI/CD:

Familiarity with DevOps practices for data engineering, including infrastructure-as-code (e.g., Terraform, CloudFormation), CI/CD pipelines, and monitoring (e.g., CloudWatch, Datadog).Familiarity with big data technologies like Apache Spark, Hadoop, or similar.ETL/ELT tools and creating common data sets across on-prem (IBMDatastage ETL) and cloud data storesLeadership & Strategy:

Lead Data Engineering team(s) in designing, developing, and maintaining highly scalable and performant data infrastructures.Customer Data Platform Development:

Architect and manage our data platforms using IBM (legacy platform) & Databricks on AWS technologies (e.g., S3, Lambda, Glacier, Glue, EventBridge, RDS) to support real-time and batch data processing needs.Data Governance & Best Practices:

Implement best practices for data governance, security, and data quality across our data platform. Ensure data is well-documented, accessible, and meets compliance standards.Pipeline Automation & Optimisation:

Drive the automation of data pipelines and workflows to improve efficiency and reliability.Team Management:

Mentor and grow a team of data engineers, ensuring alignment with business goals, delivery timelines, and technical standards.Cross Company Collaboration:

Work closely with all levels of business stakeholder including data scientists, finance analysts, MI and cross-functional teams to ensure seamless data access and integration with various tools and systems.Cloud Management:

Lead efforts to integrate and scale cloud data services on AWS, optimising costs and ensuring the resilience of the platform.Performance Monitoring: Establish monitoring and alerting solutions to ensure the high performance and availability of data pipelines and systems to ensure no impact to downstream consumers.

#J-18808-Ljbffr

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Careers

10 Ways AI Pros Stay Inspired: Boost Creativity with Side Projects, Hackathons & More

In the rapidly evolving world of Artificial Intelligence (AI), creativity and innovation are critical. AI professionals—whether data scientists, machine learning engineers, or research scientists—must constantly rejuvenate their thinking to solve complex challenges. But how exactly do these experts stay energised and creative in their work? The answer often lies in a combination of strategic habits, side projects, hackathons, Kaggle competitions, reading the latest research, and consciously stepping out of comfort zones. This article will explore why these activities are so valuable, as well as provide actionable tips for anyone looking to spark new ideas and enrich their AI career. Below, we’ll delve into tried-and-tested strategies that AI pros employ to drive innovation, foster creativity, and maintain an inspired outlook in an industry that can be both exhilarating and daunting. Whether you’re just starting your AI journey or you’re an experienced professional aiming to sharpen your skills, these insights will help you break out of ruts, discover fresh perspectives, and bring your boldest ideas to life.

Careers

Top 10 AI Career Myths Debunked: Key Facts for Aspiring Professionals

Artificial Intelligence (AI) is one of the most dynamic and rapidly growing sectors in technology today. The lure of AI-related roles continues to draw a diverse range of job seekers—from seasoned software engineers to recent graduates in fields such as mathematics, physics, or data science. Yet, despite AI’s growing prominence and accessibility, there remains a dizzying array of myths surrounding careers in this field. From ideas about requiring near-superhuman technical prowess to assumptions that machines themselves will replace these jobs, the stories we hear sometimes do more harm than good. In reality, the AI job market offers far more opportunities than the alarmist headlines and misconceptions might suggest. Here at ArtificialIntelligenceJobs.co.uk, we witness firsthand the myriad roles, backgrounds, and success stories that drive the industry forward. In this blog post, we aim to separate fact from fiction—taking the most pervasive myths about AI careers and debunking them with clear, evidence-based insights. Whether you are an established professional considering a career pivot into data science, or a student uncertain about whether AI is the right path, this article will help you gain a realistic perspective on what AI careers entail. Let’s uncover the truth behind the most common myths and discover the actual opportunities and realities you can expect in this vibrant sector.

Jobs

Global vs. Local: Comparing the UK AI Job Market to International Landscapes

How to navigate salaries, opportunities, and work culture in AI across the UK, the US, Europe, and Asia Artificial Intelligence (AI) has evolved from a niche field of research to an integral component of modern industries—powering everything from chatbots and driverless cars to sophisticated data analytics in finance and healthcare. The job market for AI professionals is consequently booming, with thousands of new positions posted each month worldwide. In this blog post, we will explore how the UK’s AI job market compares to that of the United States, Europe, and Asia, delving into differences in job demand, salaries, and workplace culture. Additionally, we will provide insights for candidates considering remote or international opportunities. Whether you are a freshly qualified graduate in data science, an experienced machine learning engineer, or a professional from a parallel domain looking to transition into AI, understanding the global vs. local landscape can help you make an informed decision about your career trajectory. As the demand for artificial intelligence skills grows—and borders become more porous with hybrid and remote work—the possibilities for ambitious job-seekers are expanding exponentially. This article will offer a comprehensive look at the various regional markets, exploring how the UK fares in comparison to other major AI hubs. We’ll also suggest factors to consider when choosing where in the world to work, whether physically or remotely. By the end, you’ll have a clearer picture of the AI employment landscape, and you’ll be better prepared to carve out your own path.

🍪 We use cookies to enhance your browsing experience on our website. By continuing to use this site, you consent to the use of cookies as described in our Cookie Policy. You can review our Cookie Policy for more information.