AI Evaluation Engineer (Testing oraz Development)

Why us?

We are part of the ALTEN Group - a leading European provider of engineering and technology consulting servics.

Join ALTEN Polska and let’s grow together!

• Location: Kraków- hybird work (6 x month)
• In this role, you will:

We are seeking an AI Evaluation & Development Engineer who will play a key role in testing, evaluating, and improving AI agents and GenAI applications while contributing to the development of supporting services and integrations.

Tasks associated with this role include

• Design, build, and maintain evaluation frameworks for AI agents and Generative AI solutions.
• Define testing methodologies, benchmarks, and quality metrics for agent performance.
• Evaluate AI outputs for accuracy, relevance, consistency, safety, and adherence to business requirements.
• Identify and analyze hallucinations, failure cases, and context-related issues.
• Create automated testing pipelines and regression testing strategies for GenAI applications.
• Validate Retrieval Augmented Generation (RAG) systems, memory stores, and context-engineering mechanisms.
• Collaborate with AI Engineers, Data Engineers, and Product teams to improve agent behavior and user experience.
• Develop Python-based services and tools supporting evaluation and monitoring processes.
• Analyze prompts and agent orchestration workflows to optimize performance and quality.
• Support production monitoring and continuous improvement of AI solutions.
• Key Skills and Qualifications:
• Strong experience with Python development.
• Hands-on experience testing AI, Machine Learning, or Generative AI systems.
• Understanding of Large Language Models (LLMs), AI agents, and RAG architectures.
• Experience creating test scenarios, evaluation datasets, and automated validation frameworks.
• Knowledge of API testing and automation tools.
• Experience working with cloud environments such as Azure or Google Cloud Platform.
• Strong analytical mindset and ability to investigate complex AI behaviors.
• We offer

We offer a full-time employment contract

• Stable and long-term cooperation
• Well-defined career path at the European leader in engineering & IT consulting
• Opportunity to relocate and work in different ALTEN Polska branches
• After completion of the project, opportunity to engage in a subsequent one within the company
• Introduction and cooperation with dedicated Competences Manager
• Benefits:
• Medicover medical care
• Medicover dental care
• Medicover Benefits platform
• Medicover Sport card
• Referral Program
• Layette for a newborn employee’s child
• Group life insurance
• Pension scheme
• We also offer cooperation based on a B2B or Contract of mandate (UZ)
• We also guarantee:
• Access to professional certifications with our technology partners (e.g. Google, ISTQB)
• Access to relevant industry events
• Access to the Knowledge Sharing Program
• Access to Referral program

Do not hesitate and join our team!

NOTICE: We kindly inform you that we will contact the selected people.

Do not hesitate and join our team!