Orion Innovation Logo

Orion Innovation

AI QA Test Engineer

Sorry, this job was removed at 07:40 p.m. (IST) on Wednesday, Dec 10, 2025
Be an Early Applicant
In-Office
Pune, Mahārāshtra, IND
In-Office
Pune, Mahārāshtra, IND

Similar Jobs

An Hour Ago
Remote or Hybrid
India
Senior level
Senior level
Aerospace • Artificial Intelligence • Cloud • Machine Learning • Software • Cybersecurity • Defense
Leads verification and validation for aerospace displays, graphics, touch-screen controllers, and electronic display systems under DO-178B/C objectives. Performs requirement- and scenario-based testing, develops test automation, resolves technical issues, drives process improvements, and collaborates with global engineering teams. Requires deep avionics and embedded systems expertise, proficiency in Python and C/C++, and extensive displays and graphics testing experience.
Top Skills: Avionics Display SystemsCC++Do-178BDo-178CEmbedded SystemsPythonTest Automation
6 Hours Ago
Remote or Hybrid
India
Entry level
Entry level
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Supports Pfizer CentreOne’s inside sales and marketing activities by reviewing and routing marketing-qualified leads, executing outreach sequences, documenting activity in Salesforce, and assisting with campaigns, content, meetings, reporting, and marketing projects. The role develops commercial expertise across marketing, inside sales, and business development while progressing leads toward initial meetings and BD handoff.
Top Skills: HubspotExcelMicrosoft OutlookMicrosoft PowerpointMicrosoft SharepointMicrosoft TeamsMicrosoft WordSalesforce
8 Hours Ago
Remote or Hybrid
India
Senior level
Senior level
Fintech • Legal Tech • Software • Financial Services • Cybersecurity • Data Privacy
Process loan settlement documents and lifecycle events including drawdowns, rollovers, rate sets, and paydowns. Coordinate with global teams, banks, and brokers to record activities accurately, manage loan trades, resolve client queries, and support loan operations. The role requires loan syndication or loan operations experience, capital markets knowledge, strong Excel skills, excellent communication, problem-solving abilities, and flexibility to work APAC, EMEA, and Americas shifts, including nights.
Top Skills: ClearparMarkitExcelWso

Orion Innovation is a premier, award-winning, global business and technology services firm.  Orion delivers game-changing business transformation and product development rooted in digital strategy, experience design, and engineering, with a unique combination of agility, scale, and maturity.  We work with a wide range of clients across many industries including financial services, professional services, telecommunications and media, consumer products, automotive, industrial automation, professional sports and entertainment, life sciences, ecommerce, and education.

About the Role

Experienced AI QA Test Engineer to ensure the quality, reliability, and ethical performance of AI-driven systems and applications. This role requires a deep understanding of AI/ML workflows, data validation, model testing techniques, and automation frameworks. The ideal candidate is highly analytical, detail-oriented, and passionate about building trusted AI products.

Participates in QA activities on project assignments and ensures timely delivery on QA commitments in a fast-paced product development environment.

Review of functional requirements and technical design documentation and creation of test plans, test scenarios and test cases. Execution of test plans and certification of application releases

Key Responsibilities:

AI/ML Testing

  • Validate ML models, data pipelines, and inference APIs for accuracy, fairness, and robustness
  • Evaluate model behavior across diverse data sets, edge cases, and unexpected inputs
  • Monitor model drift, performance degradation, and bias over time
  • Conduct prompt testing, hallucination detection, and safety validation for LLM-based systems
  • Experience of testing implementation of AI/ML into application
  • Manual Testing with automation exposure
  • API testing with SOAP UI/ Postman
  • Agile methodology experience
  • Test & Defect management with JIRA/ Azure DevOps (ADO)
  • DB Testing Experience  - Good to Have                                                                              
  • Tools - Deepeval/ Langfuse/ Pytest

Quality Engineering & Automation

  • Build and maintain automated test suites for AI features
  • Develop test scripts for ML workflows using Python, Playwright, and CI/CD tools
  • Design and execute manual and automated tests for functional, performance, and security scenarios
  • Identify defects, establish test metrics, reporting dashboards, and test traceability

Data Quality & Responsible AI

  • Validate training and inference data quality, labeling consistency, and data governance
  • Work with data science and product teams to implement responsible AI testing guidelines
  • Document and escalate risks related to bias, privacy, explainability, and compliance

Cross-Functional Collaboration

  • Partner with Data Scientists, ML Engineers, and Product Managers to define acceptance criteria
  • Participate in design reviews, model evaluations, and release readiness checkpoints
  • Contribute to QA strategy and Test Plans

Required Qualifications

  • Bachelor’s degree in computer science, Engineering, or related field (or equivalent experience)
  • 4+ years of experience in software QA engineering
  • 2+ years working with AI/ML or data-centric systems
  • Strong Python programming skills
  • Proficiency in automation frameworks (e.g., Playwright, WebdriverIO, Selenium)
  • Experience with ML platforms (TensorFlow, PyTorch, Hugging Face, MLflow, etc.)
  • Hands-on experience testing APIs, data pipelines, and distributed systems
  • Exposure to RAG pipelines and agent frameworks
  • Familiarity with prompt testing and LLM evaluation techniques
  • Experience with model evaluation metrics (precision, recall, F1, ROC-AUC, BLEU, perplexity, etc.)
  • Knowledge of MLOps, CI/CD, and cloud platforms (AWS, Azure, or GCP)
  • Familiarity with Responsible AI frameworks (NIST AI RMF, ISO/IEC standards, fairness toolkits)
  • Experience testing vector databases and AI search (Pinecone, Elasticsearch, Redis Vector)

Key Skills

  • AI/ML testing & analysis
  • Automation & scripting
  • Data validation & pipelines
  • Risk, ethics, & compliance mindset
  • Strong problem-solving & debugging skills
  • Excellent documentation & communication skills
  • Working with distributed teams

Orion is an equal opportunity employer, and all qualified applicants will receive consideration for employment without regard to race, color, creed, religion, sex, sexual orientation, gender identity or expression, pregnancy, age, national origin, citizenship status, disability status, genetic information, protected veteran status, or any other characteristic protected by law.

Candidate Privacy Policy

Orion Systems Integrators, LLC and its subsidiaries and its affiliates (collectively, “Orion,” “we” or “us”) are committed to protecting your privacy. This Candidate Privacy Policy (orioninc.com) (“Notice”) explains:

  • What information we collect during our application and recruitment process and why we collect it;
  • How we handle that information; and
  • How to access and update that information.

Your use of Orion services is governed by any applicable terms in this notice and our general Privacy Policy.


What you need to know about the Pune Tech Scene

Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account