Jobs

Spark Scala Engineer


Job details
  • Axiom Software Solutions Limited
  • Leeds
  • 1 month ago
Applications closed

JOB ADVERT FOR IJP

Exciting LongTerm Opportunity to Work on CuttingEdge Technology in One of Wipro's Top Fastestgrowing Accounts which has over 1900 associates working across India UK China Hong Kong and Mexico. HSBC is one of the biggest financial services organizations in the world with operations in more than 38 countries. It has an IT infrastructure of 200000 servers 20000 database instances and over 150 PB of data. As a Spark Scala Engineer you will have the responsibility to refactor Legacy ETL code for example DataStage into PySpark using Prophecy lowcode nocode and available converters. Converted code is causing failures/performance issues.
The HSBC Account is looking for an enthusiastic Spark Scala Engineer who will be responsible for designing building and maintaining data pipelines using Apache Spark and Scala. This includes tasks like:
• Extracting data from various sources (databases APIs files)
• Transforming and cleaning the data
• Loading the data into data warehouses or data lakes (e.g. BigQuery Amazon Redshift)
Automating the data pipeline execution using scheduling tools (e.g. Airflow)
• Work with Big Data technologies: You'll likely work with various Big Data technologies alongside Spark including:
o Hadoop Distributed File System (HDFS) for storing large datasets
o Apache Kafka for realtime data streaming
o Apache Hive for data warehousing on top of HDFS
o Cloud platforms like AWS Azure or GCP for deploying and managing your data pipelines
• Data analysis and modeling: While the primary focus might be on data engineering some JDs might require basic data analysis skills: Writing analytical queries using SQL or Spark SQL to analyze processed data and Building simple data models to understand data relationships
Your benefits
As the Spark Scala Engineer you will have the opportunity to work with one of the biggest IT landscapes in the world. You can also look forward to being mentored and groomed in your career journey by some of the finest in the business.

Your responsibilities
As a Spark Scala Engineer you will be working for HSBC – GDT (Global Data Technology) Team you will be responsible for:
• designing building and maintaining data pipelines using Apache Spark and Scala
• Working on an Enterprise scale Cloud infrastructure and Cloud Services in one of the Clouds (GCP).

Mandatory Skills
You need to have the below skills.
• At least 8 Years of IT Experience with designing building and maintaining data pipelines.
• At least 4 Years of experience with designing building and maintaining data pipelines using Apache Spark and Scala
• Programming languages: Proficiency in Scala and Spark is essential. Familiarity with Python and SQL is often a plus.
• Big Data technologies: Understanding of HDFS Kafka Hive and cloud platforms is valuable.
• Data engineering concepts: Knowledge of data warehousing data pipelines data modeling and data cleansing techniques is crucial.
• Problemsolving and analytical skills: You should be able to analyze complex data problems design efficient solutions and troubleshoot issues.
• Communication and collaboration: The ability to communicate effectively with data scientists analysts and business stakeholders is essential.
• Ready to work at least three days from HSBC Leeds (UK) office and accept changes as per customer/Wipro policies.
• To be able to traverse and explain the system designs and file format usages you have been a part of and why any tool/technology was used.

Good to have skills.
Ideally you should be familiar with
• Machine learning libraries: Familiarity with Spark ML or other machine learning libraries in Scala can be advantageous.
• Cloud computing experience: Experience with cloud platforms like AWS Azure or GCP for data pipelines deployment is a plus.
• DevOps tools: Knowledge of DevOps tools like Git CI/CD pipelines and containerization tools (Docker Kubernetes) can be beneficial.

Sign up for our newsletter

The latest news, articles, and resources, sent to your inbox weekly.

Similar Jobs

Software Dev Engineer II, Global Transportation Technology Services

GTTS (Global Transportation Technology Services) builds products that help Amazon run the world's largest transportation network, using cutting-edge technologies and machine learning, all running on AWS. We are looking for someone who is passionate about technology, loves solving customer problems, and delivers the high quality work that we expect for...

Amazon London

Senior Data Scientist - London- Spark | AWS | Python | SQL | Scala | Java

Summary:A globally leading technology firm are looking for a hands-on, engineering and data-focussed Senior Data Scientist to join their engineering team in London. Working in a heavily data-driven role, with platforms that can handle over 15 million queries/ second and multiple petabytes of data, the successful Senior Data Scientist will...

Oxford Knight London

Data Engineer - Synthetic Data Team

Data Engineer - Synthetic DataTeam Who We AreIpsos is one of the world’s largest research companies and currently the only one primarily managed by researchers, ranking as a #1 full-service research organization for four consecutive years. With over 75 different data-driven solutions, and presence in 90 markets, Ipsos brings together...

Ipsos

Lead Data Engineer

Lead Data EngineerLocation: CorkSalary: €(phone number removed) (DOE)HybridReperio are working with a major software company who are looking to add to their Data team ahead of a projected busy year in 2025. To facilitate this, they are looking for a Lead Data Engineer to play a crucial role in building...

Cork

Principal Software Engineer - Data Platform

As a Principal Engineer, you’ll get the opportunity to be a hands-on engineer, learning best practice engineering processes and approaches whilst receiving ongoing development through coaching, mentoring and pairing with other engineers on your team. From problem-solving to challenging old ways of thinking, you will have the opportunity to unleash...

Rapid7 International Limited Belfast

Principal Data Engineer

Principal Data Engineerat Capco UK - LondonPrincipal Data EngineerWhy Join Capco?Capco is a global technology and business consultancy, focused on the financial services sector. We are passionate about helping our clients succeed in an ever-changing industry.You will work on engaging projects with some of the largest banks in the world,...

CAPCO London