Senior Data Engineer

Are you looking to work for an organisation with a real purpose one where there is no limit to fresh ideas and discovering whats next? As the charity for everyone in retail, our cause is to create, hope and health and happiness for everyone in retail. Yes, thats right, everyone. Thats more than three million retail colleagues and their families. And you can help to achieve that mission. We're revolutionising the way data drives business insights and innovation to support more people to be happy at work. Our cutting-edge data platform, built on Databricks and Azure, is the backbone of our data-driven decision-making. This platform already leverages AI to generate insights for retailers and is built to be carbon-efficient, supporting our commitment to sustainability. Join us and be part of a team that's pushing the boundaries of technology to deliver real-world impact and make the retail industry a happy and healthy place to work. £70,000 - £80,000 per annum Full-Time, Permanent 35 hours per week Monday - Friday Hybrid - remote working with 1-2 days in the Covent Garden office As a Senior Data Engineer, you'll be at the forefront of our Databricks-based data platform. You'll manage automated deployments across multiple workspaces, pipelines, and workflows using Databricks SQL, Python notebooks, and Delta Live Tables (DLT). Your expertise will ensure smooth data operations by tracking down and resolving issues within Databricks notebooks and codebase. You'll streamline the ingestion of new data sources and build pipelines to integrate additional data into existing models. This will likely include delta live sharing with other organisations and leading a platform migration to Unity Catalogue. Comprehensive documentation of data pipelines, workflows, and processes will be maintained by you, ensuring clarity and continuity. You'll implement top-notch data governance and security measures, including data quality checks, lineage tracking, and compliance with relevant standards. Monitoring, managing, and optimising data storage and compute resources for peak performance, cost-efficiency, and carbon impact will be a key part of your role. You will carry out day-to-day activities to run the operational data platform, including performing daily checks to ensure the platform is running smoothly, onboarding new clients and offboarding existing clients, fixing bugs and addressing issues as they arise and over time, automate manual tasks for business efficiency. You'll work closely with stakeholders to understand data requirements and deliver solutions that meet business needs. Collaboration with the Databricks SME and the Retail Trust product team will ensure our Databricks platform is always up to date with the latest features and best practices. You'll also work with the product team to extend our data platform with additional AI services, leveraging new capabilities to enhance data processing and analytics in a responsible and ethical way. Providing guidance and mentorship to junior data engineers is a vital part of this role. You'll foster a collaborative and innovative environment, helping team members grow and succeed. Requirements We are looking for individuals with strong problem-solving skills, attention to detail, and excellent communication and collaboration abilities. An analytical mindset is essential, along with the ability to adapt to changing requirements and work in a small team in a fast-paced environment. Proven experience as a Senior Data Engineer, with a focus on Databricks and Azure, is essential. Proficiency in Python and SQL, strong knowledge of Spark, and experience with automated deployment tools like Azure DevOps and Terraform are required. Familiarity with Azure Data Lake Storage is crucial. We are looking for a T-shaped data engineer who is happy to work on all aspects of the data platform. A passion for using technology and data to impact others is highly valued. You will work with Databricks, including Workflows/Jobs, DLT Pipelines, Delta Live Tables (DLT), automated deployment, architecture, and security. Programming languages and tools such as Python, PySpark, Pandas, Pytest, and SQL will be used. You will be involved in data modelling concepts like schema design and ETL/ELT processes. The role involves working with cloud technologies, specifically Azure, and using version control systems like Git. CI/CD pipelines and collaboration tools such as Azure ADO, Slack, and Teams. Experience with Dynamic CRM along with advanced analytics and machine learning in Databricks are all advantageous. Other information At Retail Trust, we look after our colleagues by offering market-leading benefits, including: A pension scheme of up to 9% employer contributions. Life assurance of four times your salary. Private Medical Insurance on completion of 12 months service. A health cash plan, allowing you to claim back on the cost of a number of services such as optical, dental, and medical. Access to a free counselling and support phoneline and a second medical advice service. 25 days annual leave, plus bank holidays. Additional days off including a duvet day, self-development/volunteering day, and a day to celebrate your birthday . Enhanced family-friendly benefits including up to ten days compassionate leave per year, three months full maternity pay, and two weeks full paternity pay. Ready to take your career to the next level? Apply now and join us on this exciting journey ADZN1_UKTJ