B2B Project: Data Engineer (Python, Azure)

Hays Contracting Services operates under a B2B service delivery model. We are currently looking for a B2B contractor to provide services in the area of Data Engineering and Azure cloud solutions as part of a greenfield data platform initiative focused on regulatory intelligence and data processing.

Location: remote Cooperation type: B2B contract Start date: ASAP Project duration: 2-3 months (part-time) Rate: 145-155 PLN/h (net + VAT) Experience: min. 3 years

Project description

The project involves building a new data warehouse in the client's Azure environment for a regulatory monitoring platform. As a Mid Data Engineer, you will contribute to the implementation of Python ETL/ELT pipelines, data integrations, and data quality mechanisms, cooperating with a Senior Data Engineer on the target data model and quality standards.

Technology environment

• Python, SQL, ETL/ELT pipelines
• Azure: Data Lake Storage, Data Factory, Functions, Azure SQL
• Databricks or Microsoft Fabric
• REST APIs, JSON/XML, Pydantic, Pandera, Great Expectations
• Git, Azure DevOps, GitHub Actions, Medallion Architecture, Star Schema

Scope of services

• Developing and maintaining production-ready ETL/ELT pipelines in Python
• Building Python-based data pipelines feeding the data warehouse with data from 3-6 regulatory and parliamentary sources across the US, UK, Canada, France, Australia and New Zealand
• Integrating external data sources through REST APIs
• Supporting integrations with non-API data sources, including PDF-based content and web scraping solutions
• Ensuring data quality, validation, monitoring, logging, error handling and reprocessing mechanisms
• Handling schema drift and changes in source data structures
• Working within the client's Azure environment and repositories
• Cooperating with a Senior Data Engineer on implementing the target data model and data quality standards

Required competencies

• Minimum 3 years of commercial experience as a Data Engineer
• Hands-on experience building production ETL/ELT pipelines in Python
• Experience with Azure Data Lake Storage, Azure Data Factory, Azure Functions and Azure SQL
• Practical knowledge of Databricks or Microsoft Fabric
• Strong SQL skills
• Experience working with REST APIs, including authentication, pagination, rate limiting, retry mechanisms and JSON/XML processing
• Experience with data validation frameworks such as Pydantic, Pandera or Great Expectations
• Knowledge of Git and CI/CD practices using Azure DevOps or GitHub Actions
• Understanding of Medallion Architecture and Star Schema concepts
• English at B2+ level, used daily in communication with the international project team
• Ability to work independently, proactively clarify requirements and take ownership of assigned tasks

Nice to have

• Experience with Terraform or Bicep
• Experience with web scraping solutions
• Experience processing PDF-based data sources
• Experience using AI/LLM capabilities within data pipelines
• Experience designing data models for Power BI reporting environments

Cooperation process

• Profile and experience review by the Hays team
• A conversation with a Hays consultant about the project and scope of services
• A technical meeting with project representatives (up to 2 stages)
• Feedback and cooperation proposal
• Welcome to the project!

The cooperation is carried out between independent businesses (B2B model) and covers the performance of services within the agreed project scope. The Contractor independently organizes and manages the manner in which the services are performed, taking into account the project requirements, security obligations, and the necessary communication with the client. Any information regarding location, availability, or project meetings is provided solely for organizational purposes and shall not be construed as creating an employment relationship, employer-like supervision, or performance of work under the direction or control of the client. #1202141 - Katarzyna Kendzior