Own production operations for AI agents, including monitoring accuracy, latency, SLAs, drift, guardrails, and reliability. Provide L1/L2 support, investigate traces and root causes, evaluate reasoning and retrieval quality, manage API and authentication issues, and partner with AI Engineering on prompt, routing, and context improvements. Embed governance, approval workflows, and audit logging into the agent learning lifecycle.
Our Purpose
Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we're helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.
Title and Summary
Senior AI Product Ops
AI Ops
Overview:
Mastercard's Operational Intelligence team is building the next generation of data products and AI solutions that help customers access and use operational insights in new ways. As AI agents move from POC to production, their behavior keeps changing in real time: the context they retrieve, the skills they select, how they resolve exceptions. These agents are never really "finished." They keep learning from production outcomes, so their reliability, safety, and trustworthiness need to be actively operated, not just monitored. That is the job of AI Ops: making sure each agent's reasoning stays correct, its context stays current, and its behavior stays governed as it learns.
Role:• Own end to end production monitoring for one or more AI agents, tracking key metrics, KPIs, accuracy, latency, SLA breaches, guardrail triggers, and out of scope rates through dedicated observability dashboards• Serve as L1/L2 production support, providing first response monitoring, triage, and escalation to tech teams as new agentic services go live• Identify and diagnose issues by monitoring eval scores and drift alerts, inspecting failing traces and patterns, and classifying root causes such as intent, tool, parameter, or hallucination errors• Evaluate whether the agent's reasoning remains correct in production, checking whether context retrieval stays current, whether resolutions match ground truth, and whether the learning loop is drifting• Monitor and manage post production reliability, including API and authentication failures and issues originating from external AI endpoints, to protect revenue generating, client facing agents• Partner with AI Engineering to prioritize fixes, run experiments on prompts, context, and routing in Dev, and validate improvements via evals before handing off proven changes for deployment• Embed governance within the agent's learning loop so that approval workflows and audit logging travel with every production change, rather than being added after the fact
All About You:• Experience monitoring or operating production AI/ML or agentic systems, including observability, evaluation, and drift detection practices• Strong analytical skills to diagnose root causes across intent classification, tool selection, parameter errors, and hallucination patterns• Understanding of AI/LLM observability concepts such as tracing, telemetry, guardrails, and evaluation engineering including benchmarks, automated evals, human review, and regression testing• Familiarity with governance, compliance, and responsible AI practices such as PII detection, access control, audit logging, and policy enforcement is a plus• Ability to work independently alongside existing Tech teams while owning a dedicated agent's post launch health• Excellent communication skills to translate production signals into clear, prioritized asks for AI Product and AI Engineering teams• Comfort working in a fast evolving environment where the process itself, not just the throughput, is what is being continuously operated on and improved
Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we're helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.
Title and Summary
Senior AI Product Ops
AI Ops
Overview:
Mastercard's Operational Intelligence team is building the next generation of data products and AI solutions that help customers access and use operational insights in new ways. As AI agents move from POC to production, their behavior keeps changing in real time: the context they retrieve, the skills they select, how they resolve exceptions. These agents are never really "finished." They keep learning from production outcomes, so their reliability, safety, and trustworthiness need to be actively operated, not just monitored. That is the job of AI Ops: making sure each agent's reasoning stays correct, its context stays current, and its behavior stays governed as it learns.
Role:• Own end to end production monitoring for one or more AI agents, tracking key metrics, KPIs, accuracy, latency, SLA breaches, guardrail triggers, and out of scope rates through dedicated observability dashboards• Serve as L1/L2 production support, providing first response monitoring, triage, and escalation to tech teams as new agentic services go live• Identify and diagnose issues by monitoring eval scores and drift alerts, inspecting failing traces and patterns, and classifying root causes such as intent, tool, parameter, or hallucination errors• Evaluate whether the agent's reasoning remains correct in production, checking whether context retrieval stays current, whether resolutions match ground truth, and whether the learning loop is drifting• Monitor and manage post production reliability, including API and authentication failures and issues originating from external AI endpoints, to protect revenue generating, client facing agents• Partner with AI Engineering to prioritize fixes, run experiments on prompts, context, and routing in Dev, and validate improvements via evals before handing off proven changes for deployment• Embed governance within the agent's learning loop so that approval workflows and audit logging travel with every production change, rather than being added after the fact
All About You:• Experience monitoring or operating production AI/ML or agentic systems, including observability, evaluation, and drift detection practices• Strong analytical skills to diagnose root causes across intent classification, tool selection, parameter errors, and hallucination patterns• Understanding of AI/LLM observability concepts such as tracing, telemetry, guardrails, and evaluation engineering including benchmarks, automated evals, human review, and regression testing• Familiarity with governance, compliance, and responsible AI practices such as PII detection, access control, audit logging, and policy enforcement is a plus• Ability to work independently alongside existing Tech teams while owning a dedicated agent's post launch health• Excellent communication skills to translate production signals into clear, prioritized asks for AI Product and AI Engineering teams• Comfort working in a fast evolving environment where the process itself, not just the throughput, is what is being continuously operated on and improved
Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:
- Abide by Mastercard's security policies and practices;
- Ensure the confidentiality and integrity of the information being accessed;
- Report any suspected information security violation or breach, and
- Complete all periodic mandatory security trainings in accordance with Mastercard's guidelines.
Mastercard Pune, Mahārāshtra, IND Office
Mastercard Pune Tech Hub Office



Poona Club Road, Pune, Maharashtra, India, 411001
Similar Jobs at Mastercard
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Develop and maintain scalable full-stack applications using Java, Spring Boot, microservices, event-driven architecture, and cloud technologies. Build reusable solutions, implement coding and quality standards, contribute in CI/CD and Agile environments, conduct proofs of concept, collaborate across teams, and provide technical mentoring. The role emphasizes customer experience, secure development, online systems, and continuous innovation.
Top Skills:
Ci/CdEvent-Driven ArchitectureJava 8+MicroservicesNoSQLPivotal Cloud FoundryRdbmsSpring Boot
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Develop and optimize Mastercard payment applications through performance testing, Java and database tuning, bottleneck analysis, automation, and APM tooling. Design load tests, measure latency and throughput, resolve system performance issues, improve testing environments, coach junior engineers, and present results to stakeholders. Requires Java Spring Boot, PostgreSQL, Oracle, JMeter, BlazeMeter, performance analysis tools, and payments industry knowledge.
Top Skills:
Apache JmeterBlazemeterDynatraceIso 8583JavaJava Microbenchmark Harness (Jmh)JprofilerOraclePostgresSplunkSpring Boot
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Develop, test, deliver, and support scalable software solutions for Mastercard’s Cardholder Services platform. Build Java-based applications, REST APIs, automation frameworks, and CI/CD solutions while collaborating in Agile teams. Lead automation initiatives, troubleshoot production issues, conduct code reviews and technical evaluations, mentor engineers, and promote secure, high-quality engineering practices.
Top Skills:
Api AutomationCi/CdCloud-Native Application DevelopmentGitJ2EeJavaMicroservicesPlaywrightRest ApisSeleniumSpringSpring BootWeb Services
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.




