AI QA Automation Engineer

Description

Job Summary: We are seeking an AI QA Automation Engineer to ensure the quality of applications and AI agents through test automation and LLM evaluation. Key Highlights: 1. Ensure the quality and reliability of applications and AI agents 2. Design, develop, and maintain test automation frameworks 3. Collaborate with development, data, and AI engineering teams Accenture is a global leader in professional services, offering solutions in strategy and consulting, technology, operations, and digital capabilities that empower our clients to achieve their full potential. Within our \#Technology team, we support organizations across industries in adopting innovative solutions powered by Artificial Intelligence, automation, and next-generation digital platforms. We are currently seeking an AI QA Automation Engineer. The selected candidate will be responsible for ensuring the quality and reliability of applications and AI agents through test automation, definition of validation scenarios, and continuous evaluation of large language models (LLMs), thereby contributing to the delivery of robust, scalable, and quality-standard-aligned solutions. **Role Responsibilities:** * Design, develop, and maintain test automation frameworks for AI-based solutions. * Execute functional, integration, and AI agent validation tests. * Define and manage test scenarios using YAML and JSON formats. * Evaluate large language model (LLM) performance, including metrics such as hallucination, groundedness, relevance, and response quality. * Integrate automated testing processes into Azure DevOps pipelines and CI/CD environments. * Collaborate with development, data, and AI engineering teams to ensure the quality of implemented solutions. * Analyze results, document findings, and propose continuous improvements to testing processes. **✔️ Job Requirements:** * Over 1 year of experience in testing AI solutions or related environments. * Experience in test automation and quality assurance of applications. * Knowledge and experience in validating and testing AI agents. * Experience working with Azure DevOps and CI/CD processes. * Familiarity with LLM evaluation, including concepts such as hallucination and groundedness. * Experience designing and executing test scenarios using YAML and JSON. * Strong analytical ability to identify defects, assess risks, and ensure solution quality. * Collaborative work skills and effective communication with multidisciplinary teams. We seek professionals passionate about innovation, software quality, and Artificial Intelligence—possessing an analytical mindset, attention to detail, and motivation to drive high-impact technological solutions. All Accenture decisions are made without discrimination based on race, religion, color, sex, age, disability, nationality, sexual orientation, gender identity or expression, or any other reason covered by local legislation.