Hexaware Technologies Logo

Hexaware Technologies

Test Analyst  - AI Assurance

Posted 6 Days Ago
Be an Early Applicant
Remote
Hiring Remotely in India
Senior level
Remote
Hiring Remotely in India
Senior level
Develop AI assurance testing strategies for generative AI, LLM, RAG, and agent-based systems. Design functional, non-functional, safety, reliability, and responsible AI evaluations; curate golden datasets; generate synthetic data; implement Python evaluation frameworks; establish observability and tracing with OpenTelemetry; integrate evaluations into CI/CD; report responsible AI metrics; and perform monitoring, diagnostics, root cause analysis, and regulatory validation.
The summary above was generated by AI

Test Analyst - AI Assurance
Role Overview
Strong understanding of GenAI system architecture, including Retrieval-Augmented Generation (RAG) and Agent-based systems, with hands-on experience in implementing Python-based AI evaluation frameworks such as Ragas, DeepEval, and similar tools. Proven ability to perform Model and Agent quality evaluations using platforms like Arize, LangFuse, and Confident AI, while driving Responsible AI (RAI) validation and reporting key RAI metrics. Experienced in golden dataset curation, synthetic dataset generation, and establishing AI observability and tracing through OpenTelemetry (OTEL) instrumentation to enable effective monitoring, diagnostics, and continuous quality improvement of AI systems.
Key Responsibilities
•  Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
•  Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
•  Experience in Golden dataset curation & Synthetic dataset generation
•  Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc. 
•  Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
•  Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
•  Experience in Integrating AI evaluation into CI/CD pipeline
•  Experience in Responsible AI validation 

 Required Skills and Capabilities
•  Strong understanding of GenAI System Architecture of RAG & Agents
•  Proficiency in Python & AI evaluation SDKs
•  AI Evaluation Frameworks (Ragas, DeepEval) 
•  Model Evaluation 
•  Agent Evaluation 
•  Responsible AI (RAI) Validation 
•  RAI Metrics Reporting 
•  Golden Dataset Curation 
•  Synthetic Dataset Generation 
•  AI Observability 
•  AI Tracing 
•  OpenTelemetry (OTEL) 
•  LLM Evaluation 
•  AI Quality Engineering 
•  AI Governance 
•  AI Monitoring 
•  AI Testing & Assurance 
•  Root Cause Analysis (AI Systems)
•  Knowledge of regulatory considerations for AI systems
Role-Specific Requirements
•  Proficiency in Python & AI evaluation SDKs
•  Strong understanding of RAG & agentic system architecture
•  Ability to develop AI Assurance test strategy to assess AI systems for quality, safety, reliability and responsible AI behavior
•  Design AI evaluation strategy for LLMs, RAG and agent evaluation across functional and non-functional dimensions
•  Experience in Golden dataset curation & Synthetic dataset generation
•  Experience in implementing AI evaluation frameworks like Ragas, DeepEval, etc. 
•  Experience in using AI Evaluation tools like LangSmith, Arize, Langfuse, Galileo, Evidently, etc.
•  Experience in instrumentation to establish AI Observability / tracing using OTEL libraries
•  Experience in Integrating AI evaluation into CI/CD pipeline
•  Experience in Responsible AI validation
•  Knowledge of regulatory considerations for AI systems
Experience Requirements
More than 4 years of experience in AI Assurance Testing
 


Hexaware Technologies Pune, Mahārāshtra, IND Office

North Block, Plot No. 19, Rajiv Gandhi InfoTech Park, MIDC - SEZ, Phase 3, Hinjawadi, Pune, Maharastra, India, 411057

Similar Jobs

6 Minutes Ago
Remote or Hybrid
India
Entry level
Entry level
Big Data • Food • Hardware • Machine Learning • Retail • Automation • Manufacturing
Early-career sales trainee role supporting Mondelez India's sales organization and learning within a fast-paced consumer packaged goods environment. The position is part of the 2026 onboarding batch and is based at the North Branch, with no relocation support available.
6 Minutes Ago
Remote or Hybrid
India
Entry level
Entry level
Big Data • Food • Hardware • Machine Learning • Retail • Automation • Manufacturing
Early-career sales trainee role supporting Mondelez India’s commercial operations and distribution of snack and beverage brands. The position is designed to develop early-career talent through practical experience in a fast-paced sales environment. It is a fixed-term internship based at the North Branch, with no relocation support available.
6 Minutes Ago
Remote or Hybrid
India
Entry level
Entry level
Big Data • Food • Hardware • Machine Learning • Retail • Automation • Manufacturing
Early-career sales trainee position with Mondelez International's North Branch. The role is designed to develop sales professionals through hands-on experience in a fast-paced consumer goods environment. It is an internship-style, fixed-term opportunity for the 2026 onboarding batch. No relocation support is provided.

What you need to know about the Pune Tech Scene

Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account