Senior Data Engineer - Scala / Spark / Hadoop

We are looking for an experienced Data Engineer to join an international team working on a strategic regulatory initiative within the financial sector. You will develop and maintain a data platform and reporting processes operating in a complex Big Data environment.

Responsibilities

• Design, develop and maintain Data Engineering solutions using Hadoop and Apache Spark.
• Build scalable solutions for processing large volumes of data.
• Analyse complex business requirements and translate them into effective technical solutions.
• Develop and optimise SQL queries and data-processing workflows.
• Build and enhance data pipelines and reporting processes.
• Create and maintain workflows using Apache Oozie and Apache Airflow.
• Develop data solutions using Scala.
• Work with event-driven architecture based on Apache Kafka.
• Maintain and improve CI/CD processes using Git, pull requests and Jenkins.
• Provide L3 support for production environments.
• Investigate production incidents and technical problems in cooperation with relevant operational teams.
• Collaborate with business stakeholders to define requirements and analyse issues.
• Ensure the quality, performance, security and reliability of data solutions.

Requirements

• Strong hands-on experience with Scala.
• Commercial experience with Apache Hadoop.
• Practical knowledge of Apache Spark and large-scale data processing.
• Experience building and maintaining Data Engineering pipelines.
• Experience with Apache Kafka and event-driven architecture.
• Practical knowledge of CI/CD processes, particularly Jenkins.
• Very good knowledge of Git, code review and pull request processes.
• Strong SQL development and optimisation skills.
• Experience supporting production environments.
• Ability to independently investigate complex technical issues.
• English proficiency sufficient for daily work in an international team.
• Availability to work from the Warsaw or Gdańsk office twice per week.
• Ability to perform the work from Poland.

Nice to have

• Kubernetes experience.
• Knowledge of Apache Flink.
• Experience with Apache Airflow or Apache Oozie.
• Previous involvement in regulatory or reporting projects.
• Experience in the financial sector.

What we offer

• Participation in a strategic regulatory initiative with a tangible impact on the financial sector.
• Work within a large and advanced Big Data environment.
• A modern technology stack including Scala, Hadoop, Spark, Kafka, Airflow and Jenkins.
• A high level of independence and influence over technical solutions.
• Cooperation with an experienced international team.
• Stable, long-term cooperation.
• A professional environment focused on quality, security and expert development.
• An opportunity to gain experience in financial regulations and regulatory reporting.