Job Title: Senior Data Engineer
About Luupli:
Luupli is a social media app that has equity, diversity, and equality at its heart. We believe that social media can be a force for good, and we are committed to creating a platform that maximises the value that creators and businesses can gain from it, while making a positive impact on society and the planet. Our app is still in development, but we are excited about the possibilities it presents. Our team is made up of passionate and dedicated individuals who are committed to making Luupli a success.
About the Role:
We’re seeking a skilled Senior Data Engineer with experience in recommendation systems to join our team. This role is pivotal in enhancing our backend systems, managing databases, and optimising recommendation algorithms. You’ll play a critical role in the architecture and performance of our recommendation infrastructure, ensuring accurate, relevant, and efficient recommendations for our users.
Responsibilities:
Data and Database Management:
- Design, optimise, and maintain tables and data structures to support recommendation and trending content data.
- Work with structured data storage solutions, including PostgreSQL and JSONB, to manage recommendation and interaction data.
Recommendation Systems:
- Implement and refine recommendation algorithms (e.g., collaborative filtering, content-based, and hybrid approaches) to enhance relevancy.
- Use similarity search libraries like Annoy or Faiss to optimise recommendation speed and accuracy.
- Continuously evaluate recommendation logic to better serve user preferences, ensuring real-time delivery.
Data Aggregation and Analysis:
- Aggregate, analyses, and process user interaction data to support recommendations and trending content.
- Design efficient queries and implement aggregation methods to capture relevant data and insights for recommendations.
Debugging and Optimization:
- Identify, troubleshoot, and resolve data handling issues to ensure accurate recommendation delivery.
- Optimize queries, processing workflows, and containerized services for high performance and scalability within AWS ECS.
Requirements:
- Proven experience as a Data Engineer or Backend Engineer, with a focus on recommendation systems.
- Proficient in SQL and database management, especially with PostgreSQL and JSONB for structured data handling.
- Solid understanding of recommendation algorithms (collaborative filtering, content-based, hybrid approaches).
- Experience with similarity search libraries such as Annoy or Faiss.
- Strong programming skills in Python, with experience in building backend logic for data-intensive applications in a containerized environment.
- Familiarity with AWS ECS for container management, including task scheduling and scaling.
- Experience using AWS Event Bridge to trigger workflows or automate tasks in response to application events.
- Analytical skills for data aggregation, querying, and insights generation.
- Strong debugging and optimization skills for handling large-scale data processing in cloud-based environments.
Preferred Qualifications:
- Knowledge of data aggregation pipelines, ETL processes, and data handling at scale.
- Familiarity with additional AWS services (e.g., S3, Lambda) for data storage and event-driven architectures.
- Experience with machine learning libraries or tools used in recommendation systems.
Compensation:
This is an equity-only position, offering a unique opportunity to gain a stake in a rapidly growing company and contribute directly to its success.
