AI QA Engineer

Moody’s Analytics is a global leader in ratings and integrated risk assessment that is advancing artificial intelligence to help clients navigate uncertainty. The AI QA Engineer will support quality engineering for software and AI-enabled products by executing manual and automated testing, evaluating AI and large language model outputs, documenting defects, and improving QA processes.

What you'll do:

• Support quality engineering efforts across an Agile team by executing software and AI product testing, documenting findings, and contributing to continuous improvement of QA processes
• Perform manual, exploratory, regression, API, and integration testing within an Agile team environment under close guidance
• Test software features that include AI or large language model components, reviewing outputs for accuracy, relevance, completeness, consistency, and instruction-following
• Create basic test scenarios, prompts, and test data for evaluating AI behavior and identifying issues such as hallucinations, incomplete responses, and inconsistent results
• Apply basic prompt engineering and context engineering techniques during testing activities
• Use AI-assisted tools such as Codex, Claude Code, and similar AI command-line interface tools under the guidance of senior team members
• Assist with existing AI evaluation harnesses, test scripts, and regression checks to support quality validation efforts
• Document defects with clear steps to reproduce, expected results, and actual results, and collaborate with engineers, developers, and product owners to investigate and resolve issues
• Contribute to test documentation, reusable test cases, and the continuous improvement of quality assurance processes and standards

What they're looking for:

• Basic understanding of software testing concepts, including manual, exploratory, regression, API, and integration testing methodologies
• Basic programming or scripting knowledge, preferably in Python, with familiarity in APIs, databases, JSON, version control, and defect tracking tools
• Basic understanding of AI concepts, large language models, and common AI limitations such as hallucinations, incomplete responses, and inconsistent outputs
• Interest in prompt engineering, context engineering, and the evaluation of AI-enabled products
• Ability to document defects clearly with steps to reproduce, expected results, and actual results
• Strong analytical, communication, documentation, and problem-solving skills with the ability to collaborate across engineering and product teams
• Basic understanding of artificial intelligence concepts, with curiosity and enthusiasm for learning how AI tools can be used to improve processes and drive efficiency. Interest in exploring AI systems and a willingness to develop awareness of responsible AI practices, including risk management and ethical use
• Bachelor's degree in Computer Science, Software Engineering, or a related field

Benefits:

• Incentive compensation
• Medical insurance
• Dental insurance
• Vision insurance
• Parental leave
• Paid time off
• A 401(k) plan with employee and company contribution opportunities
• Life insurance
• Disability insurance
• Accident insurance