AI Platform Senior Engineer, QA

About Accrete

Accrete is an agentic managed services company: an AI workforce that does high-stakes, judgement-heavy work for government and enterprise clients — at the economics of software, not labor.

At the core is Accrete’s Knowledge Engine, a dynamic context graph that captures an organization’s tacit knowledge, resolves data across silos, and builds a living, auditable ground truth. Expert agents reason against that ground truth to act on complex, high-stakes decisions — not just answer questions about them.

From national security to commercial operations, Accrete delivers the work on one platform, with unlimited expert agents and expert judgement.

About the Role: 

We are seeking an AI Platform Senior Engineer, QA – Agentic AI & Automation to lead quality assurance for our agent-driven platforms and AI models. This role focuses on validating intelligent agents, non-deterministic AI outputs, and cross-product workflows using advanced automation and AI evaluation techniques.

You will play a key role in building our Enterprise AI Trust & Validation Framework, ensuring reliability, transparency, and scalability of AI systems.

• Test multi-agent workflows, orchestration layers, and cross-domain intelligence systems
• Validate AI-generated narratives, themes, and analytical outputs.
• Perform groundedness, coherence, hallucination, and stability testing.
  
• Test conversational AI and chat-based agent systems.
• Evaluate consistency across multiple runs and detect output drift.

• Design and maintain automation frameworks using Python, JavaScript/TypeScript, Pytest, and Playwright.
• Build E2E automation for UI, APIs, and agent pipelines.
• Develop reusable test utilities for AI validation and regression testing.
• Integrate automation with CI/CD pipelines.

• Validate in-house AI/ML models and inference pipelines.
• Implement LLM-as-Judge and semantic similarity-based validation.
• Perform statistical and comparative testing for model outputs.
• Support testing of models deployed on AWS.

• Perform API testing using Postman and automated scripts.
• Design and execute load and stress testing.
• Monitor system performance under AI-heavy workloads.
• Identify bottlenecks and scalability risks.

• Analyze business requirements, user stories, and AI specifications.
• Design test strategies for agentic and AI-driven systems.
• Work closely with Engineering, Product, and Data Science teams.
• Track defects, quality metrics, and release readiness.
• Drive continuous improvement in QA processes

• 6–8+ years of experience in QA, Automation, or Quality Engineering.
• Strong proficiency in Python, JavaScript/TypeScript, Pytest, and Playwright.
• Experience testing AI systems, chatbots, or LLM-based platforms.
• Strong understanding of non-deterministic system testing.
• Experience with API testing (Postman, REST frameworks).
• Hands-on experience with performance testing. (K6 familiarity is a plus).
• Experience working in Agile/Scrum environments.
• Strong analytical and debugging skills.
• Excellent communication and documentation abilities.

Preferred Qualifications

• Experience with agent frameworks (LangChain, AutoGen, CrewAI, etc.).
• Knowledge of RAG systems and vector databases.
• Exposure to MLOps and AI observability tools.
• Experience with cloud platforms (AWS preferred).
• Familiarity with statistical validation and NLP evaluation.
• CI/CD experience (GitHub Actions, Jenkins, GitLab).
• Familiarity with containerization technologies such as Docker/Kubernetes

Core Competencies

• AI Quality Engineering
• Agent Workflow Validation
• Automation Architecture
• Model Evaluation & Monitoring
• Performance Engineering
• Risk Assessment
• Cross-team Collaboration
• Ownership Mindset

Career Growth

This role offers growth into positions such as:

• Principal AI QA Engineer
• AI Systems Quality Architect
• QA Engineering Lead
• AI QA Engineering Manager

Cultural Fit & Values

At Accrete, success goes beyond technical expertise. We value:

• Ownership & Accountability: Drive initiatives to completion with high standards and proactive problem-solving.
• Transparency & Communication: Share information openly and give and receive constructive feedback.
• Strategic Thinking & Problem-Solving: Analyze challenges critically and develop effective, scalable solutions.
• Collaboration & Professionalism: Work effectively across teams and contribute to a culture of mutual respect.
• Passion for AI & Innovation: Stay curious, continuously learn, and push the boundaries of AI.
• Going Beyond the Job Description: Look for opportunities to create impact beyond your immediate responsibilities.
  These principles shape our culture, performance expectations, and growth opportunities.

Company Benefits

• Competitive Compensation: Aligned with experience and market standards.
• Mediclaim: ₹10 lakh coverage for you and your family.
• Flexible PTO & Hybrid Work: Take time off when needed and enjoy remote flexibility in accordance with company guidelines.
• Growth & Development: Access to professional learning opportunities and career advancement support.
• Onsite Perks: Catered lunches and snacks.
• Team Bonding: Company-sponsored social events and team activities.

Accrete is an equal opportunity/affirmative action employer. All qualified applicants will receive consideration for employment without regard to sex, gender identity, sexual orientation, race, color, religion, national origin, disability, protected Veteran status, age, or any other characteristic protected by law.