My client is seeking a skilled Senior Data Engineer to join their team as they develop a modern, scalable, and automated data architecture. The successful candidate will be at the forefront of automating the collection and standardisation of diverse datasets, enabling data-driven decision-making across the organisation. This is an exciting opportunity for someone with a strong background in Azure services and a keen interest in leveraging Large Language Models (LLMs) to streamline manual data processes.

Key Responsibilities:

• Data Engineering & Architecture:Design, build, and maintain scalable data pipelines using Azure Data Factory, Databricks, and Synapse Analytics.
• Data Collection & Standardisation:Develop tools to collect and standardise data from various sources, ensuring consistent and accurate representation of values.
• Automation & Optimisation:Utilise LLMs to classify and align data sets, automating manual processes and improving operational efficiency.
• Model Deployment:Support the training and deployment of machine learning models, ensuring they’re integrated into data pipelines and accessible through APIs.
• Cross-functional Collaboration:Partner with product and engineering teams to understand their data needs and provide solutions that enhance their capabilities.
• Innovation & Proof of Concept (PoC):Design and implement PoCs to demonstrate the value of new data solutions and technologies to internal stakeholders.

Required Skills & Experience:

• Strong experience with Azure-managed services (Azure Data Lake, Data Factory, Databricks, Event Hubs, Synapse Analytics).
• Proficient in Python and CI/CD pipeline implementation.
• Hands-on experience with LLMs or classification models, particularly in using supervised learning methods.
• Demonstrated ability to build and deploy APIs or interfaces that integrate with other systems.
• Proven track record of creating scalable data solutions that drive business outcomes.
• Experience in working with large, bespoke data sets and resolving issues related to data consistency.

Preferred Qualifications:

• Understanding of how to incorporate ML models into data architectures and enhance pipelines using advanced analytics.
• Ability to build and refine automated systems to eliminate manual data processing tasks.

Benefits:

• Competitive salary and benefits package.
• Flexible working hours with a hybrid setup.
• Opportunity to work on cutting-edge data solutions that drive innovation.