Python Automation (10324)

Job Description: Senior Quality Assurance Engineer - AI & GenAI Applications

We are seeking a Senior Quality Assurance Engineer to drive quality assurance for AI-powered chat applications, RAG-based knowledge retrieval systems, Agentic AI workflows, and intelligent support automation solutions. The ideal candidate will ensure the quality, reliability, accuracy, and performance of AI-driven products through a combination of manual and automated testing practices.

Key Responsibilities

• Define and execute QA strategies for AI Assistant and Generative AI applications.
• Design and perform functional, integration, regression, API, and end-to-end testing.
• Validate chatbot responses for accuracy, relevance, context retention, and overall user experience.
• Test RAG pipelines, knowledge retrieval, citation accuracy, and response grounding.
• Validate Agentic AI workflows, tool integrations, task orchestration, and multi-step reasoning processes.
• Perform API testing and automation using Postman and REST services.
• Analyze application logs, traces, and production issues using Kibana, Elasticsearch, and observability tools.
• Monitor AI behavior, prompts, token usage, and response quality using Langfuse or similar LLM observability platforms.
• Develop and maintain AI test automation frameworks.
• Conduct performance, load, and reliability testing for AI applications.
• Identify defects, perform root cause analysis, and collaborate closely with developers for timely resolution.
• Ensure compliance with quality standards, release readiness criteria, and Definition of Done (DoD).
• Actively participate in Agile/Scrum ceremonies and quality planning activities.

Required Skills & Experience

• 6+ years of experience in Software Quality Assurance and Test Automation.
• Hands-on experience testing AI Chatbots, Conversational AI, Generative AI, or RAG-based applications.
• Strong expertise in API testing, Postman, and test automation frameworks.
• Strong proficiency in Python and PyTest for test automation, API validation, data processing, and AI/LLM testing workflows.
• Experience with Langfuse, Kibana, Elasticsearch, and AI observability/monitoring tools.
• Good understanding of LLMs, Prompt Engineering, RAG architectures, Vector Databases, and Agentic AI concepts.
• Experience with Selenium, Playwright, Cucumber, or similar automation tools.
• Strong analytical, debugging, problem-solving, and communication skills.
• Experience working in Agile/Scrum-based enterprise product development environments.

Good to Have

• Knowledge of Fitness testing framework.

Education Bachelor’s or Master’s degree in Computer Science, Engineering, Information Technology, or a related field, with proven experience in enterprise-scale QA delivery.