The AI Automation Engineer will create automated systems for testing and enhancing conversational AI models using Python, integrating LLMs and developing pipelines for quality evaluation, monitoring, and compliance.
About the Company:
Netomi is the leading agentic AI platform for enterprise customer experience. We work with the largest global brands like Delta Airlines, MetLife, MGM, United, and others to enable agentic automation at scale across the entire customer journey. Our no-code platform delivers the fastest time to market, lowest total cost of ownership, and simple, scalable management of AI agents for any CX use case. Backed by WndrCo, Y Combinator, and Index Ventures, we help enterprises drive efficiency, lower costs, and deliver higher quality customer experiences.
Want to be part of the AI revolution and transform how the world’s largest global brands do business? Join us!
About the Role
An AI Automation Engineer focuses on building automated systems to evaluate, test, and enhance the performance of conversational AI models (LLMs, chatbots, voice assistants). They sit at the intersection of machine learning, software engineering, and QA automation, ensuring that AI systems produce high-quality, safe, and reliable responses at scale.
Responsibilities
- Automation Pipelines: Build Python-based pipelines for automated quality testing of AI responses.
- Integrate LLMs into automated evaluation frameworks (e.g., using GPT-based evaluators, embeddings, or custom scoring).
- Automate regression and stress testing for conversational AI flows.
- Quality & Evaluation: Define evaluation metrics (relevance, factuality, coherence, safety, empathy).
- Implement both rule-based and AI-driven quality checks.
- Monitor model drift, bias, and hallucinations using automated workflows.
- Integration & Deployment: Work with APIs, SDKs, and CI/CD pipelines to embed automated AI evaluation in production.
- Develop monitoring dashboards to visualize conversation quality.
- Collaborate with ML engineers, product managers, and QA teams to close the feedback loop.
- Optimization & Innovation: Experiment with prompt engineering and automated prompt-testing frameworks.
- Explore reinforcement learning, self-critique models, or human-in-the-loop automation for continuous improvement.
- Automate compliance and policy adherence checks for enterprise AI systems.
Requirements
- Strong 2-5 Years of experience in Python development (automation, scripting, data handling).
- Experience with LLMs/NLP frameworks.
- Understanding of MLOps / AI deployment pipelines.
Netomi is an equal opportunity employer committed to diversity in the workplace. We evaluate qualified applicants without regard to race, color, religion, sex, sexual orientation, disability, veteran status, and other protected characteristics.
Top Skills
APIs
Ci/Cd
Llms
Nlp
Python
Sdks
Similar Jobs
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
Develop and maintain high performance Pega Applications, implement integration solutions, troubleshoot issues, and collaborate with stakeholders. Ensure security and efficiency of applications using Pega 8.6+ and cloud technologies.
Top Skills:
Azure DevopsCosmos DbDockerJavaKubernetesOraclePega PlatformPrpcRestful ApiSpring BootSQL Server
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
The Software Engineer II will develop and maintain Pega integration solutions, design high-performance applications, manage user authentication, and implement technology best practices. Responsibilities include collaboration with stakeholders, troubleshooting, and resolving incidents, alongside working with REST APIs and CI/CD pipelines.
Top Skills:
Azure DevopsCosmos DbDockerJavaKubernetesOraclePegaPega ConstellationRestful ApiSpring BootSQL Server
Artificial Intelligence • Fintech • Hardware • Information Technology • Sales • Software • Transportation
The Salesforce Developer will design, develop, and maintain Salesforce applications, ensuring scalability and adherence to best practices while collaborating with various teams.
Top Skills:
ApexBulk ApiGearsetGitJavaScriptLwcRest ApiSalesforceSalesforce Soap ApiSfdxSOQLSoslVisualforce
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
