Senior Data Engineer

SR2 | Socially Responsible Recruitment | Certified B Corporation™
7 months ago
Applications closed

Related Jobs

View all jobs

Senior Data Engineer

Senior Data Engineer - Databricks

Senior Data Engineer_London_Hybrid

Senior Data Engineer - Snowflake - £100,000

Senior Data Scientist

Senior Data Scientist

The following role is a Fixed Term Contract, with the ideal candidate starting in October and working up until Christmas 2025.


The company:

Are an organisation dedicated to building a society where everyone has a secure place to live, supportive people around them, and a meaningful purpose. They achieve this through two key approaches: delivering high-quality public services to improve lives on the frontline, and using their knowledge to drive systemic change, addressing the complex web that can trap and disempower those it was created to help. With the heart of a charity and the mindset of a business, we are uniquely positioned to address this challenging agenda.


Where You Fit In

It’s an exciting time for the charity as they embark on their first large-scale data transformation project. We are seeking a forward-thinking, dynamic problem solver to be our go-to expert in data engineering. The Data Architecture Project will centralise their siloed data assets into one ecosystem, allowing them to dive deeper into insights from their projects and databases, and to measure impact more effectively. In the future, this valuable information will be used to predict needs on the frontline of service delivery.


In this project, you will work within the Evidence and Insight Team and collaborate across the organisation, including the wider Corporate Services Team and key stakeholders from various Delivery Teams. They are a rapidly growing, agile team that ensures we stay up-to-date with the latest standards by using key tools such as Azure Data Factory, Databricks, Terraform, DevOps, and Jinja. They believe trust and transparency are key drivers of best practice, and they offer an environment built on shared ownership, unity, and emotional intelligence.


Main Duties & Accountabilities

As the Senior Data Engineer, you will play a pivotal role in designing and building our data ecosystem. Key areas of responsibility will include:


  • Architecture Design:Lead the development and implementation of a dynamic, elastic, and resilient data architecture that supports current and future contracts. This role requires cross-departmental and external collaboration to identify needs and create data solutions, along with the ability to communicate and translate complex technical concepts effectively. You must understand the full data lifecycle and business needs to develop suitable solutions.


  • ETL Process:Collaborate with external and internal teams to build and implement Extract, Transform, Load (ETL) processes and comprehensive pipelines using tools like Azure Data Factory, Azure Logic/Functions Apps, and Databricks. You will extract data from diverse, complex sources across the organisation, transform it into a usable format, and load it into data storage systems. Reviewing systems and suggesting optimisation solutions for organisational efficiency and security will be required.


  • Data Pipeline Management:Build and manage data pipelines that are fault-tolerant, reliable, and dynamic. Automate data workflows, including data ingestion, aggregation, and ETL processing. Ensure data accuracy, integrity, security, and compliance in collaboration with the wider team and Information Security team.


  • Data Modelling:Ensure business readiness from the data we ingest, with a strong knowledge of data modelling, creating queryable, efficient analytics-ready models. Work alongside analysts to speed up analysis and make our organisation AI/ML-ready.


  • Cloud-Based Solutions:Serve as the expert on all things Azure, including the management of the data infrastructure, best practices, cost optimisation, and performance tuning.


  • Skill Development:As an expert in data engineering, you will play a fundamental role in the data maturity journey, upskilling and supporting data professionals across the organisation to sharpen and grow their skills while enhancing an environment of skill development.


About You

You will have at least 5+ years of experience in data engineering roles and be an Azure expert. You thrive in small, collaborative teams that work quickly and maintain a high level of quality. You should have enough business acumen to understand how our work ties into the bigger picture, along with a sharp eye for details and an ability to spot inconsistencies.


Your skill set should include:

  • Data Analysis and Synthesis:Ability to bring multiple data sources together in a conformed model for analysis.
  • Cloud Management:Expertise in managing and optimising cloud data management systems.
  • Data Modelling:Understanding of the concepts and principles of data modelling and the ability to produce relevant data models.
  • Programming and Building:Proficiency in leading on design, coding, testing, and corrections, with input from others on specifications.
  • Technical Understanding:Strong knowledge and understanding of technical concepts within the role and the wider landscape.
  • Skill Development:Capability to lead and coach others to improve team skills.
  • Collaboration:Ability to work across multiple teams and with external resources to achieve the best outcomes.
  • Strategy:Understanding of how work fits within the broader organisational landscape and potential future developments.


Key Success Outcomes for This Role:

  • Data is captured within one ecosystem, and systems are maintainable.
  • The organisation is well-positioned for future data insights, building predictive models, and data science opportunities by Summer 2025.
  • Clear and well-documented data pipelines are established and maintained.
  • A data maturity framework is co-created with the Data Transformation Programme Manager and other collaborative stakeholders.
  • Strong foundations of data quality, governance, and security are provided to future-proof data practices.

Get the latest insights and jobs direct. Sign up for our newsletter.

By subscribing you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Portfolio Projects That Get You Hired for AI Jobs (With Real GitHub Examples)

In the fast-evolving world of artificial intelligence (AI), an impressive portfolio of projects can act as your passport to landing a sought-after role. Even if you’ve aced interviews in the past, employers in AI and machine learning (ML) are increasingly asking candidates to demonstrate hands-on experience through the projects they’ve built and shared online. This is because practical ability often speaks volumes about your suitability for a role—far more than any exam or certification alone could. In this article, we’ll explore how to build an outstanding AI portfolio that catches the eye of recruiters and hiring managers, including: Why an AI portfolio is crucial for job seekers. How to choose AI projects that align with your target roles. Specific project ideas and real GitHub examples to help you stand out. Best practices for showcasing your work, from writing clear READMEs to using Jupyter notebooks effectively. Tips on structuring your GitHub so that employers can instantly see your value. Moreover, we’ll discuss how you can use your portfolio to connect with top employers in AI, with a handy link to our CV-upload page on Artificial Intelligence Jobs for when you’re ready to apply. By the end, you’ll have a clear roadmap to building a portfolio that will help secure interviews—and the AI job—of your dreams.

AI Job Interview Warm‑Up: 30 Real Coding & System‑Design Questions

In today's competitive AI job market, nailing a technical interview can be the difference between landing your dream role and getting lost in the crowd. Whether you're looking to break into machine learning, deep learning, NLP (Natural Language Processing), or data science, your problem-solving skills and system design expertise are certain to be put to the test. AI‑related job interviews typically involve a range of coding challenges, algorithmic puzzles, and system design questions. You’ll often be asked to delve into the principles of machine learning pipelines, discuss how to optimise large-scale systems, and demonstrate your coding proficiency in languages like Python, C++, or Java. Adequate preparation not only boosts your confidence but also reduces the likelihood of fumbling through unfamiliar territory. If you’re actively seeking positions at major tech companies or innovative AI start-ups, then check out www.artificialintelligencejobs.co.uk for some of the latest vacancies in the UK. Meanwhile, this blog post will guide you through 30 real coding & system-design questions you’re likely to encounter during your AI job interview. This list is designed to help you practise, anticipate typical question patterns, and stay ahead of the competition. By reading through each question and thinking about the possible approaches, you’ll sharpen your problem-solving skills, time management, and critical thinking. Each question covers fundamental concepts that employers regularly test, ensuring you’re well-equipped for success. Let’s dive right in.

Negotiating Your AI Job Offer: Equity, Bonuses & Perks Explained

Artificial intelligence (AI) has proven itself to be one of the most transformative forces in today’s business world. From smart chatbots in customer service to predictive analytics in finance, AI technologies are reshaping how organisations operate and innovate. As the demand for AI professionals grows, so does the complexity of compensation packages. If you’re a mid‑senior AI professional, you’ve likely seen job offers that include far more than just a base salary—think equity, bonuses, and a range of perks designed to entice you into joining or staying with a company. For many, the focus remains squarely on salary. While that’s understandable—after all, your monthly take‑home pay is what covers day-to-day expenses—limiting your negotiations to salary alone can leave considerable value on the table. From stock options in ambitious startups to sign‑on bonuses that ‘buy you out’ of your current contract, modern AI job offers often include elements that can significantly boost your long-term wealth and job satisfaction. This article aims to shed light on the full scope of AI compensation—specifically focusing on how equity, bonuses, and perks can enhance (or sometimes detract from) the overall value of your package. We’ll delve into how these elements work in practice, what to watch out for, and how to navigate the negotiation process effectively. Our goal is to provide mid‑senior AI professionals with the insights and tools to land a holistic compensation deal that accurately reflects their technical expertise, leadership potential, and strategic importance in this fast-moving field. Whether you’re eyeing a leadership role in machine learning at an established tech giant, or you’re considering a pioneering position at a disruptive AI startup, the knowledge in this guide will help you weigh the merits of base salary alongside the potential riches—and risks—of equity, bonuses, and other benefits. By the end, you’ll have a clearer sense of how to align your compensation with both your immediate lifestyle needs and long-term career aspirations.