AI Test Automation Engineer

Description

Job Summary: We are seeking an AI Test Automation Engineer to ensure the quality and reliability of AI applications and agents through test automation and language model evaluation. Key Highlights: 1. Ensure the quality of AI-based innovative solutions. 2. Design and maintain AI test automation frameworks. 3. Evaluate the performance of large language models (LLMs). Accenture is a global leader in professional services, delivering solutions in strategy and consulting, technology, operations, and digital capabilities to help our clients achieve their full potential. Within our \#Technology team, we support organizations across industries in adopting cutting-edge AI-driven innovations, automation, and next-generation digital platforms. We are currently seeking an AI Test Automation Engineer for AI Solutions. The selected candidate will be responsible for ensuring the quality and reliability of AI applications and agents through test automation, defining validation scenarios, and continuously evaluating large language models (LLMs), thereby contributing to the delivery of robust, scalable, and quality-standard-aligned solutions. **Role Responsibilities:** * Design, develop, and maintain test automation frameworks for AI-based solutions. * Execute functional, integration, and AI agent validation tests. * Define and manage test scenarios using YAML and JSON formats. * Evaluate the performance of large language models (LLMs), including metrics such as hallucination, groundedness, relevance, and response quality. * Integrate automated testing processes into Azure DevOps pipelines and CI/CD environments. * Collaborate with development, data, and AI engineering teams to ensure the quality of implemented solutions. * Analyze results, document findings, and propose continuous improvements to testing processes. **✔️ Job Requirements:** * More than 1 year of experience in testing AI solutions or related environments. * Experience in test automation and quality assurance of applications. * Knowledge and experience in validating and testing AI agents. * Experience working with Azure DevOps and CI/CD processes. * Knowledge of LLM evaluation, including concepts such as hallucination and groundedness. * Experience designing and executing test scenarios using YAML and JSON. * Strong analytical ability to identify defects, assess risks, and ensure solution quality. * Collaborative work skills and effective communication with multidisciplinary teams. We seek professionals passionate about innovation, software quality, and artificial intelligence — with an analytical mindset, attention to detail, and motivation to drive high-impact technological solutions. All Accenture decisions are made without discrimination on the basis of race, religion, color, sex, age, disability, nationality, sexual orientation, gender identity or expression, or any other characteristic protected by local law.