National AI Awards 2025Discover AI's trailblazers! Join us to celebrate innovation and nominate industry leaders.

Nominate & Attend

Principal Software Engineer - Data Platform

Rapid7
Belfast
8 months ago
Applications closed

Related Jobs

View all jobs

Principal Software Engineer

Principal Software Engineer

Principal Software Developer

Head of Engineering

Principal C++ Engineer

Principal MLOps/GenAI Infrastructure Engineer

As a Principal Engineer, you’ll get the opportunity to be a hands-on engineer, learning best practice engineering processes and approaches whilst receiving ongoing development through coaching, mentoring and pairing with other engineers on your team. From problem-solving to challenging old ways of thinking, you will have the opportunity to unleash your full potential and creativity whilst working with cutting edge technologies in a dynamic and collaborative team.

About the Team

The Data Platform team is responsible for building ETL Pipelines that fuel the Data Platform at Rapid7. Moving Product Data into our Data Platform for product teams to develop new features, enhance existing features and build shared experiences to create value for customers across the world.

We have a cutting edge data stack including Kafka, K8s, Spark and Iceberg.

About the Role


The Principal Engineer role is a part of our Data Platform Engineering team. In this role you will be focussed on helping our product teams move data into our Data Platform for in product experiences and product analytics.

As a Principal Engineer on the Data Platform Engineering team, you will be responsible for architecting and scaling streaming and batch data pipelines, while also designing the CI/CD infrastructure that ensures efficient development and deployment of data services. You will play a key role in shaping the architecture of our data platform, collaborating with cross-functional teams to deliver highly available, performant, and scalable solutions for both real-time and large-scale data processing.

In this role, you will:

Architect and implement a highly scalable Data Platform that supports Change Data Capture (CDC) using Debezium and Kafka for data replication across different databases and services.

Design and maintain large-scale data lakes using Apache Iceberg, ensuring efficient data partitioning, versioning, and schema evolution to support real-time analytics and historical data access.

Build and optimize CI/CD pipelines for the deployment and automation of data platform services using tools like Jenkins.

Lead the integration of Apache Spark for large-scale data processing and ensure that both batch and streaming workloads are handled efficiently.

Collaborate with our Platform Delivery teams to ensure high availability and performance of the data platform, implementing monitoring, disaster recovery, and automated testing frameworks.

Provide technical leadership and mentoring to junior engineers, promoting best practices in CDC architecture, distributed systems, and CI/CD automation.

Ensure that the platform adheres to data governance principles, including data lineage tracking, auditing, and compliance with regulatory requirements.

Stay informed about the latest advancements in CDC, data engineering, and infrastructure automation to guide future platform improvements.

Work closely with product and data science teams to understand business requirements and translate them into scalable and efficient data platform solutions.

Stay current with the latest trends in data engineering and infrastructure, making recommendations for improvements and introducing new technologies as appropriate.

The skills you’ll bring include:

10+ years of experience in software engineering with a focus on data platform engineering, data infrastructure, or distributed systems.

Expertise in building data pipelines using Apache Kafka or similar for ingesting, processing, and distributing high-throughput data.

Strong experience designing and managing CI/CD pipelines for data platform services using tools such as Jenkins.

Experience with Apache Iceberg (or similar Delta Lake/Apache Hudi) for managing versioned, partitioned datasets in data lakes with an understanding of Apache Spark for both batch and streaming data processing, including optimization strategies for distributed data workloads.

Expertise in designing distributed systems and managing high-throughput, fault-tolerant, and low-latency data architectures.

Strong programming skills in Java, Scala, or Python.

Experience with cloud-based environments (AWS, GCP, Azure) and containerized infrastructure using Kubernetes and Docker.

The attitude and ability to thrive in a high-growth, evolving environment

Collaborative team player who has the ability to partner with others and drive toward solutions

Strong creative problem solving skills

Solid communicator with excellent written and verbal communications skills both within the team and cross functionally

Passionate about delighting customers, puts the customer needs at the forefront of all decision making

Excellent attention to detail


We know that the best ideas and solutions come from multi-dimensional teams. That’s because these teams reflect a variety of backgrounds and professional experiences. If you are excited about this role and feel your experience can make an impact, please don’t be shy - apply today.

#LI_FB1

National AI Awards 2025

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

AI Jobs UK 2025: 50 Companies Hiring Now

Bookmark this guide – we refresh it every quarter so you always know who’s really scaling their artificial‑intelligence teams. Artificial intelligence hiring has roared back in 2025. The UK’s boosted National AI Strategy funding, record‑breaking private investment (£18.1 billion so far) & a fresh wave of generative‑AI product launches mean employers are jockeying for data scientists, ML engineers, MLOps specialists, AI product managers, prompt engineers & applied researchers. Below are 50 organisations that have advertised UK‑based AI vacancies in the past eight weeks or formally announced growth plans. They’re grouped into five easy‑scan categories so you can jump straight to the kind of employer – & culture – that suits you. For each company you’ll find: Main UK hub Example live or recent vacancy Why it’s worth a look (tech stack, culture, mission) Use the internal links to browse current vacancies on ArtificialIntelligenceJobs.co.uk – or set up a free job alert so fresh roles land in your inbox.

Return-to-Work Pathways: Relaunch Your AI Career with Returnships, Flexible & Hybrid Roles

Stepping back into the workplace after a career break can feel like embarking on a whole new journey—especially in a cutting-edge field such as artificial intelligence (AI). For parents and carers, the challenge isn’t just refreshing your technical know-how but also securing a role that respects your family commitments. Fortunately, the UK’s tech sector now boasts a wealth of return-to-work programmes—from formal returnships to flexible and hybrid opportunities. These pathways are designed to bridge the gap, equipping you with refreshed skills, confidence and a supportive network. In this comprehensive guide, you’ll discover how to: Understand the booming demand for AI talent in the UK Leverage transferable skills honed during your break Overcome common re-entry challenges Build your AI skillset with targeted training Tap into returnship and re-entry programmes Find flexible, hybrid and full-time AI roles that suit your lifestyle Balance professional growth with caring responsibilities Master applications, interviews and networking Whether you’re returning after maternity leave, eldercare duties or another life chapter, this article will equip you with practical steps, resources and insider tips.

LinkedIn Profile Checklist for AI Jobs: 10 Tweaks That Triple Recruiter Views

In today’s fiercely competitive AI job market, simply having a LinkedIn profile isn’t enough. Recruiters and hiring managers routinely scout for top talent in machine learning, data science, natural language processing, computer vision and beyond—sometimes before roles are even posted. With hundreds of applicants vying for each role, you need a profile that’s optimised for search, speaks directly to AI-specific skills, and showcases measurable impact. By following this step-by-step LinkedIn for AI jobs checklist, you’ll make ten strategic tweaks that can triple recruiter views and position you as a leading AI professional. Whether you’re a fresh graduate aiming for your first AI position or a seasoned expert targeting a senior role, these actionable changes will ensure your profile stands out in feeds, search results and recruiter queues. Let’s dive in.