Data lakes, Hadoop Developer

N Consulting Ltd
London
1 year ago
Applications closed

Related Jobs

View all jobs

MLOps Tech Lead

Data Scientist (Public sector)

Data Scientist (Public sector)

Data & Analytics Data Scientist (Public sector) Professional Multiple Cities

Senior Clinical Data Science Programmer

Data Scientist | Machine Learning | Azure | £54k + 10% Bonus


Job Title: Data lakes, Hadoop Developer
Location: London
Work model: Hybrid

Key Responsibilities
Design, build, and manage scalable Data Lakes to support large-scale data processing and analytics.
Develop and maintain Big Data solutions using the Hadoop ecosystem (HDFS, Hive, HBase, Spark, Pig, MapReduce, etc.).
Implement data ingestion pipelines and workflows for structured, semi-structured, and unstructured data.
Optimize data processing and storage to ensure high performance and low latency.
Collaborate with data engineers, analysts, and scientists to provide robust and efficient data access solutions.
Monitor and troubleshoot data pipelines and applications to ensure reliability and accuracy.
Implement data security, governance, and compliance practices across the data lake and Hadoop systems.
Stay updated with emerging Big Data technologies and recommend tools or approaches to enhance the data platform.

Required Skills and Qualifications
Proven experience with Hadoop ecosystems, including HDFS, YARN, Hive, HBase, MapReduce, and Spark.
Expertise in Data Lake architectures and principles.
Proficiency in programming languages such as Python, Java, or Scala for Big Data processing.
Hands-on experience with ETL tools, data ingestion frameworks, and workflow schedulers (e.g., Apache Nifi, Airflow).
Strong knowledge of cloud platforms such as AWS (S3, EMR, Glue), Azure (Data Lake Storage, Synapse), or Google Cloud (BigQuery, Dataflow).
Familiarity with query languages like SQL, HiveQL, or Presto.
Understanding of data governance, security, and compliance (e.g., GDPR, HIPAA).
Excellent problem-solving skills and the ability to debug and resolve issues in distributed systems.

Preferred Qualifications
Experience with Kubernetes, Docker, or other containerization technologies for Big Data deployments.
Knowledge of streaming frameworks like Kafka, Flume, or Spark Streaming.
Hands-on experience in implementing machine learning workflows in a Big Data environment.
Certifications in Big Data technologies or cloud platforms (e.g., AWS Big Data Specialty, Cloudera Certified Professional).
Familiarity with tools like Databricks, Delta Lake, or Snowflake.
 

Subscribe to Future Tech Insights for the latest jobs & insights, direct to your inbox.

By subscribing, you agree to our privacy policy and terms of service.

Industry Insights

Discover insightful articles, industry insights, expert tips, and curated resources.

Jobs Careers

AI Jobs for Career Switchers in Their 30s, 40s & 50s (UK Reality Check)

Changing career into artificial intelligence in your 30s, 40s or 50s is no longer unusual in the UK. It is happening quietly every day across fintech, healthcare, retail, manufacturing, government & professional services. But it is also surrounded by hype, fear & misinformation. This article is a realistic, UK-specific guide for career switchers who want the truth about AI jobs: what roles genuinely exist, what skills employers actually hire for, how long retraining really takes & whether age is a barrier (spoiler: not in the way people think). If you are considering a move into AI but want facts rather than Silicon Valley fantasy, this is for you.

Jobs

How to Write an AI Job Ad That Attracts the Right People

Artificial intelligence is now embedded across almost every sector of the UK economy. From fintech and healthcare to retail, defence and climate tech, organisations are competing for AI talent at an unprecedented pace. Yet despite the volume of AI job adverts online, many employers struggle to attract the right candidates. Roles are flooded with unsuitable applications, while highly capable AI professionals scroll past adverts that feel vague, inflated or disconnected from reality. In most cases, the issue isn’t a shortage of AI talent — it’s the quality of the job advert. Writing an effective AI job ad requires more care than traditional tech hiring. AI professionals are analytical, sceptical of hype and highly selective about where they apply. A poorly written advert doesn’t just fail to convert — it actively damages your credibility. This guide explains how to write an AI job ad that attracts the right people, filters out mismatches and positions your organisation as a serious employer in the AI space.

Education Jobs

Maths for AI Jobs: The Only Topics You Actually Need (& How to Learn Them)

If you are a software engineer, data scientist or analyst looking to move into AI or you are a UK undergraduate or postgraduate in computer science, maths, engineering or a related subject applying for AI roles, the maths can feel like the biggest barrier. Job descriptions say “strong maths” or “solid fundamentals” but rarely spell out what that means day to day. The good news is you do not need a full maths degree worth of theory to start applying. For most UK roles like Machine Learning Engineer, AI Engineer, Data Scientist, Applied Scientist, NLP Engineer or Computer Vision Engineer, the maths you actually use again & again is concentrated in a handful of topics: Linear algebra essentials Probability & statistics for uncertainty & evaluation Calculus essentials for gradients & backprop Optimisation basics for training & tuning A small amount of discrete maths for practical reasoning This guide turns vague requirements into a clear checklist, a 6-week learning plan & portfolio projects that prove you can translate maths into working code.

🍪 We use cookies to enhance your browsing experience on our website. By continuing to use this site, you consent to the use of cookies as described in our Cookie Policy. You can review our Cookie Policy for more information.