Senior Data Engineer / Data Architect - Azure Data Explorer, IoT streaming (Freelance)

We connect talented freelancers with cutting-edge projects, offering a platform for you to showcase your expertise and thrive in a collaborative environment.

Currently we are looking for a Senior Data Engineer with architect skills to join an environmental monitoring platform for industry. You'll take over a working proof of concept of a streaming IoT measurement pipeline on Azure and bring it to production - from edge devices, through the raw layer, to read-optimised data and rollups in Azure Data Explorer.

Location: Remote/Warsaw Office Salary (PLN): 160-200 net + VAT (B2B) Start: November

What will you do?

• Take over the PoC and bring it to production - the data semantics are already defined, so you can focus on the pipeline.
• Build the stream processor from Event Hubs to Azure Data Explorer.
• Design and schedule the cascade of rollups and time-window aggregations.
• Implement reprocessing of data after configuration changes.
• Model and optimise ADX tables, update policies, materialized views, retention policies and queries for large data volumes.
• Work closely with our PM, our ADX specialist, the client's engineering team and our DevOps.

What we're looking for

Must-have

• At least 2-3 data pipeline projects taken from zero to production in the role of architect or lead data engineer, with Azure Data Explorer in production in at least one of them.

Experience

• Senior experience in data engineering (around 5+ years), including IoT streams and time-series data
• Broad, hands-on experience with Azure data services

Tech skills

• Azure Data Explorer (Kusto) and KQL: table modelling, update policies, materialized views, retention policies, query optimisation for large volumes
• Streaming ingestion on Azure: IoT Hub, Event Hubs, parsing functions (e.g. Azure Functions), Blob Storage
• Layered data architecture (raw/Bronze Silver aggregates), designing rollups and time-window aggregations
• Python
• Terraform

Soft skills

• Independence - you can own a production pipeline end to end without hand-holding
• At least B2 in English - we work internationally, so that's a must
• Strong collaboration and communication skills - you'll work with several teams
• Attention to detail

Nice-to-haves

• Environmental data (spectra, multi-parameter data, interval and event data)
• Preparing data for ML (e.g. anomaly detection) and near-real-time queries
• PostgreSQL/PostGIS and spatial data
• Edge devices / edge computing
• TypeScript/NestJS

How to apply Along with your CV, please send a short description of 2-3 data pipelines you took to production. For each one: your role, scope, stack, data scale, what went wrong and what you would do differently - plus a link or reference if possible.

Note: Please keep in mind that this is a part-time, freelance opening.