Accrete AI Logo

Accrete AI

AI Platform Senior Engineer, QA

Posted One Month Ago
Be an Early Applicant
In-Office
Mumbai, Maharashtra
Senior level
In-Office
Mumbai, Maharashtra
Senior level
Lead quality assurance for agentic AI platforms, models, and cross-product workflows. Design Python and JavaScript/TypeScript automation frameworks, validate non-deterministic AI outputs, test APIs and agent pipelines, perform performance testing, and build AI evaluation and regression capabilities. Collaborate with Engineering, Product, and Data Science teams to define test strategies, monitor quality metrics, identify scalability risks, and support release readiness.
The summary above was generated by AI

About Accrete

Accrete is an agentic managed services company: an AI workforce that does high-stakes, judgement-heavy work for government and enterprise clients — at the economics of software, not labor.

At the core is Accrete’s Knowledge Engine, a dynamic context graph that captures an organization’s tacit knowledge, resolves data across silos, and builds a living, auditable ground truth. Expert agents reason against that ground truth to act on complex, high-stakes decisions — not just answer questions about them.

From national security to commercial operations, Accrete delivers the work on one platform, with unlimited expert agents and expert judgement.

About the Role: 

We are seeking an AI Platform Senior Engineer, QA – Agentic AI & Automation to lead quality assurance for our agent-driven platforms and AI models. This role focuses on validating intelligent agents, non-deterministic AI outputs, and cross-product workflows using advanced automation and AI evaluation techniques.

You will play a key role in building our Enterprise AI Trust & Validation Framework, ensuring reliability, transparency, and scalability of AI systems.


Key Responsibilities
Agentic AI & System Testing
  • Test multi-agent workflows, orchestration layers, and cross-domain intelligence systems
  • Validate AI-generated narratives, themes, and analytical outputs.
  • Perform groundedness, coherence, hallucination, and stability testing.
  • Test conversational AI and chat-based agent systems.
  • Evaluate consistency across multiple runs and detect output drift.

Automation & Framework Development
  • Design and maintain automation frameworks using Python, JavaScript/TypeScript, Pytest, and Playwright.
  • Build E2E automation for UI, APIs, and agent pipelines.
  • Develop reusable test utilities for AI validation and regression testing.
  • Integrate automation with CI/CD pipelines.

AI Model & Evaluation Testing
  • Validate in-house AI/ML models and inference pipelines.
  • Implement LLM-as-Judge and semantic similarity-based validation.
  • Perform statistical and comparative testing for model outputs.
  • Support testing of models deployed on AWS.

API & Performance Engineering
  • Perform API testing using Postman and automated scripts.
  • Design and execute load and stress testing.
  • Monitor system performance under AI-heavy workloads.
  • Identify bottlenecks and scalability risks.

Quality Strategy & Collaboration
  • Analyze business requirements, user stories, and AI specifications.
  • Design test strategies for agentic and AI-driven systems.
  • Work closely with Engineering, Product, and Data Science teams.
  • Track defects, quality metrics, and release readiness.
  • Drive continuous improvement in QA processes

Required Qualifications
  • 6–8+ years of experience in QA, Automation, or Quality Engineering.
  • Strong proficiency in Python, JavaScript/TypeScript, Pytest, and Playwright.
  • Experience testing AI systems, chatbots, or LLM-based platforms.
  • Strong understanding of non-deterministic system testing.
  • Experience with API testing (Postman, REST frameworks).
  • Hands-on experience with performance testing. (K6 familiarity is a plus).
  • Experience working in Agile/Scrum environments.
  • Strong analytical and debugging skills.
  • Excellent communication and documentation abilities.


Preferred Qualifications

  • Experience with agent frameworks (LangChain, AutoGen, CrewAI, etc.).
  • Knowledge of RAG systems and vector databases.
  • Exposure to MLOps and AI observability tools.
  • Experience with cloud platforms (AWS preferred).
  • Familiarity with statistical validation and NLP evaluation.
  • CI/CD experience (GitHub Actions, Jenkins, GitLab).
  • Familiarity with containerization technologies such as Docker/Kubernetes


Core Competencies

  • AI Quality Engineering
  • Agent Workflow Validation
  • Automation Architecture
  • Model Evaluation & Monitoring
  • Performance Engineering
  • Risk Assessment
  • Cross-team Collaboration
  • Ownership Mindset


Career Growth

This role offers growth into positions such as:

  • Principal AI QA Engineer
  • AI Systems Quality Architect
  • QA Engineering Lead
  • AI QA Engineering Manager

Cultural Fit & Values

At Accrete, success goes beyond technical expertise. We value:

  • Ownership & Accountability: Drive initiatives to completion with high standards and proactive problem-solving.
  • Transparency & Communication: Share information openly and give and receive constructive feedback.
  • Strategic Thinking & Problem-Solving: Analyze challenges critically and develop effective, scalable solutions.
  • Collaboration & Professionalism: Work effectively across teams and contribute to a culture of mutual respect.
  • Passion for AI & Innovation: Stay curious, continuously learn, and push the boundaries of AI.
  • Going Beyond the Job Description: Look for opportunities to create impact beyond your immediate responsibilities.
    These principles shape our culture, performance expectations, and growth opportunities.


Company Benefits

  • Competitive Compensation: Aligned with experience and market standards.
  • Mediclaim: ₹10 lakh coverage for you and your family.
  • Flexible PTO & Hybrid Work: Take time off when needed and enjoy remote flexibility in accordance with company guidelines.
  • Growth & Development: Access to professional learning opportunities and career advancement support.
  • Onsite Perks: Catered lunches and snacks.
  • Team Bonding: Company-sponsored social events and team activities.

Accrete is an equal opportunity/affirmative action employer. All qualified applicants will receive consideration for employment without regard to sex, gender identity, sexual orientation, race, color, religion, national origin, disability, protected Veteran status, age, or any other characteristic protected by law.

Similar Jobs

4 Minutes Ago
In-Office
Mid level
Mid level
Fintech • Information Technology • Financial Services
Build scalable software and AI-enabled solutions using Python, Java, LLMs, generative AI, agentic workflows, and RAG. Translate business needs into production-ready prototypes and applications, integrate enterprise tools and APIs, and contribute across development, testing, and delivery in an Agile environment. Collaborate with users, designers, developers, product managers, and clients to improve operational scalability and outcomes.
Top Skills: Agentic AiAladdinCi/CdCloud TechnologiesContainer TechnologiesEmbeddingsFastapiGenerative AiGoogle AdkJavaLangchainLanggraphLlmsModel Context Protocol (Mcp)PythonRest ApisRetrieval-Augmented Generation (Rag)Source ControlVector Search
7 Minutes Ago
Hybrid
Senior level
Senior level
Artificial Intelligence • Big Data • Enterprise Web • Fintech • Software • Financial Services
Develop investment analytics, portfolio monitoring, performance measurement, risk reporting, benchmark analytics, and business intelligence solutions. Build scalable dashboards and reporting tools using Power BI, Tableau, or Qlik Sense. Analyze multi-asset financial datasets with Python, PySpark, and SQL; automate reporting; integrate APIs and financial data providers; and collaborate with investment and technology teams to support portfolio management and research decisions.
Top Skills: APIsAWSGitGithub CopilotMatplotlibMicrosoft CopilotNumpyPandasPlotlyPower BIPysparkPythonQlik SenseSeabornSQLTableau
21 Minutes Ago
Hybrid
Pune, Maharashtra, IND
Mid level
Mid level
Artificial Intelligence • Cloud • Sales • Security • Software • Cybersecurity • Data Privacy
Manage office facilities, access security, maintenance, travel and visa coordination, telecom services, vendors, inventory, billing, helpdesk tickets, and event logistics. The role supports workplace infrastructure, utilities, office supplies, security personnel, corporate travel, and facility-related vendor relationships while coordinating with Accounts and global teams. It requires hands-on troubleshooting, office walk-throughs, schedule flexibility, and occasional Saturday support for maintenance activities.
Top Skills: Access Control SystemsCctv SurveillanceDg Power BackupFire Alarm SystemsMS OfficeServicenowTelecom SystemsUps

What you need to know about the Pune Tech Scene

Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account