Senior QA Automation Engineer - AI / ML

We are seeking a detail-oriented Senior QA Automation Engineer with 5 – 10 years of hands-on experience to join our team. The ideal candidate will have strong expertise in test automation using tools like Selenium, API validation/testing, and experience with various architectural frameworks This role involves creating, maintaining, and executing robust automated testing frameworks to ensure the delivery of high-quality software solutions.
In addition to the traditional QA automation tasks, the role also includes test data creation, validation of applications integrated with AI / ML / GenAI workflows.

Automation Framework and Testing:

• Design, develop, and maintain modular, extensible, and reusable test automation frameworks and scripts, while maintaining high stability.
• Set up, configure, and maintain the automation environment and data to ensure smooth testing processes.
• Develop and execute efficient automated test scripts to validate stability and responsiveness.
• Create and maintain a comprehensive test automation library with scenarios to ensure thorough requirements and regression coverage.

Testing and Validation:

• Assist in functional, regression, and integration testing for new or modified software programs using both manual and automated approaches.
• Collaborate with developers and product teams to identify test requirements and ensure coverage of various workflows.
• Perform validation and create test data for various testing scenarios for the applications / APIs / Agents integrated with AI / ML / GenAI workflows and models as necessary.
• Implement Risk Gating processes to validate outputs for accuracy, bias, toxicity, correctness, and traceability.
• Develop and maintain automation frameworks for continuous validation of applications and AI-driven features.
• Identify, report, and track product defects, providing clear and concise documentation of issues and potential improvements.

Collaboration and Process Improvement:

• Participate in product design reviews to provide input on requirements, automation feasibility, and testing implications.
• Write, review, and verify test plans and procedures for product evaluations and defect identification.
• Accurately estimate testing timelines and work toward targeted on-time delivery by proactively removing roadblocks.
• Collaborate with cross-functional teams to align product goals and ensure quality standards are met.
• Operational Excellence:
• Provide feedback and insights to improve the software quality assurance process and methodology.
• Stay up to date with the latest automation tools and best practices involving AI/ML to optimize the automation framework.

Qualifications

• At least 5 - 10 years of experience in developing and executing automated test cases and test plans scripts.
• Proficiency with tools such as Selenium/WebDriver, Playwright or Postman (API automation).
• Strong programming skills in one of Python, Java, JavaScript/TypeScript, or C#.
• Strong understanding of software quality assurance standards, practices, and methodologies.
• Hands-on experience validating end-to-end workflows, edge cases, and non-functional aspects.
• Ability to design checks for accuracy, hallucinations, bias, correctness, and traceability of model outputs.
• Familiarity with model evaluation metrics (precision/recall, BLEU/ROUGE, toxicity/bias scores) and guardrail patterns.
• Strong skills in synthetic test data generation, data masking, and scenario design using SQL or CSVs.
• Ability to balance individual and team effort in collaborative processes while meeting set deadlines
• Proficiency in API and application / functional validation and experience with API testing tools.
• Extensive understanding and continued learnings around QA methodology, tools, and approaches
• Strong written and verbal communication skills, with the ability to collaborate effectively across technical and non-technical teams.

Preferred

• Experience testing AI-driven products and LLM-integrated workflows.
• Exposure to ML pipelines and data validation (e.g., feature store checks, dataset drift detection).
• Knowledge of data visualization tools (e.g., Matplotlib, Seaborn).
• Knowledge of cloud platforms (AWS, Azure, or GCP)

Netsmart is proud to be an equal opportunity workplace and is an affirmative action employer, providing equal employment and advancement opportunities to all individuals. We celebrate diversity and are committed to creating an inclusive environment for all associates. All employment decisions at Netsmart, including but not limited to recruiting, hiring, promotion and transfer, are based on performance, qualifications, abilities, education and experience. Netsmart does not discriminate in employment opportunities or practices based on race, color, religion, sex (including pregnancy), sexual orientation, gender identity or expression, national origin, age, physical or mental disability, past or present military service, or any other status protected by the laws or regulations in the locations where we operate.

Netsmart desires to provide a healthy and safe workplace and, as a government contractor, Netsmart is committed to maintaining a drug-free workplace in accordance with applicable federal law. Pursuant to Netsmart policy, all post-offer candidates are required to successfully complete a pre-employment background check, which is provided at Netsmart’s sole expense.