eDV Senior Big Data Engineer

We are working closely with a major National Security & Defence Consultancy in their search for an experienced Big Data Engineer to join the growing team on a permanent basis. This is a hybrid role with 4-day a week onsite in London.

Due to the nature of their projects, you must already hold an active UK*C DV Clearance.

In this role, you will be responsible for designing, implementing, and maintaining scalable big data systems that support their data science, machine learning, and AI workloads. You will work closely with cross-functional teams to ensure seamless data integration, security, and compliance with GDPR and privacy regulations. Your expertise in scripting, troubleshooting, and integration testing will be essential in optimizing our data pipelines and orchestration processes.

Key Requirements:

Proven experience in scripting languages such as Python, Bash, or similar.

Strong troubleshooting skills and the ability to diagnose and resolve complex technical issues.

Experience with integration testing and ensuring seamless tool integration.

In-depth knowledge of Linux internals and system administration.

Understanding of TCP/IP and OSI models.

Hands-on experience with data pipeline tools like NiFi and Airflow.

Proven ability to design scalable big data systems with a focus on security and GDPR compliance.

Hybrid cloud experience, specifically with on-premise and AWS cloud environments.

Familiarity with data science, machine learning, and AI tools such as Jupyter, Spacy, Transformers, and NLTK.

Experience with big data NoSQL engines/platforms such as Hive, Impala, and Elasticsearch.

Proficiency with business intelligence and visualization tools like Tableau, Kibana, and PowerBI.

Excellent communication and collaboration skills.

Preferred Experience:

Certification in AWS or other cloud platforms.

Experience with additional data orchestration tools.

Familiarity with other big data tools and technologies. (Hadoop, Cloudera)

Previous experience in a similar role within a dynamic and fast-paced environment.

Don't miss out on this opportunty to be at the forefront of ensuring the capabilities supply to the national security and defence customers is of the best quality you would expect from a world leading organisation. Get in touch now if you are interested.