Data Engineer - Databricks

B2B Project: Data Engineer - Databricks Hays Contracting Services operates under a B2B service delivery model. We are currently looking for a B2B contractor to provide Data Engineering services on the Azure Databricks platform, with responsibility for designing and building scalable data solutions.

Location, remote within Poland, with one visit per month in Warsaw Cooperation type: B2B contract with Hays Poland Project duration: initial contract for 12 months Start date, ASAP / 1 month Rate:160-200 PLN/hour, net + VAT

Project description The contractor will be responsible for designing and building data solutions on the Azure Databricks platform.

Technology environment

• Azure Databricks: Workflows, Jobs, Notebooks and Spark Declarative Pipelines (SDP)
• Python, PySpark, Spark, Delta Lake and Lakehouse architecture
• Medallion Architecture: Bronze, Silver and Gold layers
• Unity Catalog, service principals, permissions and data governance policies
• Azure Data Lake Storage Gen2, Azure Key Vault and Azure SQL / T-SQL
• Databricks Asset / Declarative Automation Bundles (DAB)
• GitHub Actions or Azure DevOps, source control, pull requests and code reviews
• Terraform or Pulumi, automated testing and testing frameworks

Scope of services

• Design and build scalable Databricks-based solutions using Workflows, Jobs, Notebooks, Spark Declarative Pipelines, Delta Lake and Medallion Architecture.
• Implement CDC / CDF patterns, data ingestion frameworks and orchestration mechanisms, including scheduled, file-arrival, continuous and API-driven triggers.
• Manage and govern data assets through Unity Catalog, including access management, permissions, service principals and data governance policies.
• Deliver solutions through CI/CD pipelines using DAB, GitHub Actions or Azure DevOps, in line with engineering best practices and clean branching strategies.
• Collaborate with business stakeholders and Business Analysts to translate requirements into scalable technical solutions.
• Define and promote engineering standards covering code quality, testing, documentation, reusability and production support.
• Work closely with QA throughout the delivery lifecycle, ensuring acceptance criteria, data quality validation and testing are built into the solution from design through production release.

Required competencies

• Minimum 3-5 years of experience in a similar position.
• English proficiency at B2+ level.
• Strong hands-on experience with Databricks Workflows, Jobs, PySpark, Python and Spark Declarative Pipelines.
• Practical knowledge of CDC / CDF implementation, Delta Lake, Unity Catalog and Lakehouse architecture.
• Experience configuring pipeline triggers, orchestrating workloads and managing production-grade data pipelines.
• Experience with Databricks Asset / Declarative Automation Bundles, source control, pull requests and code review processes.
• Good knowledge of Azure Data Lake Storage Gen2, Azure Key Vault and Azure SQL / T-SQL.
• Strong understanding of Medallion Architecture, data platform design principles and modern data engineering tooling.
• Knowledge of the Python data engineering ecosystem, including PySpark, Delta and testing frameworks.
• Experience with CI/CD using GitHub Actions or Azure DevOps.
• Experience with Infrastructure as Code using Terraform or Pulumi.
• Knowledge of Test-Driven Development, automated testing and software design best practices, including SOLID principles.
• Experience working in Agile delivery methodologies.
• Experience using AI tools throughout analysis, implementation, testing, code review and documentation activities.
• Experience with spec-driven development, including agreed specifications, testable acceptance criteria and traceability between requirements, implementation and delivered outcomes.

Nice to have

• Degree in Computer Science, Mathematics or a related field.
• Databricks Associate or Professional certification.
• Experience with Structured Streaming, Kafka or Event Hub integrations.
• Experience with Data Warehousing and Kimball dimensional modelling.
• Knowledge of Power BI, including semantic models and DirectLake connectivity.
• Financial services domain knowledge, including investment, fund accounting or regulatory reporting.
• Experience with Logic Apps, Power Automate, MongoDB or other NoSQL technologies.
• Knowledge of Model Context Protocol, LLM API integrations, including OpenAI or Anthropic, and AI-powered data engineering solutions.
• Experience building Agentic AI workflows and production-grade AI-enabled automation.

Cooperation process

• profile and experience review by the Hays team;
• a 15-minute phone conversation with a Hays recruiter about the project and your experience;
• a technical conversation with the client (2-3 stages);
• agreement on cooperation terms and offer;
• commencement of service delivery.

The cooperation is carried out between independent businesses (B2B model) and covers the performance of services within the agreed project scope. The Contractor independently organizes and manages the manner in which the services are performed, taking into account the project requirements, security obligations, and the necessary communication with the client. Any information regarding location, availability, or project meetings is provided solely for organizational purposes and shall not be construed as creating an employment relationship, employer-like supervision, or performance of work under the direction or control of the client.

#li-dni

#1201189 - Mateusz Malachowski