Gramian Consulting Group Logo

Gramian Consulting Group

Cybersecurity Engineer - AI Benchmarking

Posted 2 Days Ago
Be an Early Applicant
Remote
Hiring Remotely in India
Senior level
Remote
Hiring Remotely in India
Senior level
Develop security benchmarks for AI coding models by researching and reproducing real-world vulnerabilities, creating Harbor-format audit-and-patch tasks, building Docker environments and automated verifiers, evaluating AI-generated patches, and documenting security findings and remediation requirements. The role requires strong software engineering, application security, vulnerability analysis, testing, and technical writing skills across multiple programming languages and ecosystems.
The summary above was generated by AI

About Gramian

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

About the Role

We are seeking an experienced Senior Software Security Engineer to develop high-quality training and evaluation tasks for advanced AI coding models. You will transform real-world software vulnerabilities, including CVEs, security advisories, and open-source bug reports, into realistic Harbor-format audit-and-patch tasks across multiple programming languages.

This role combines software security research, vulnerability reproduction, benchmark development, automated testing, and technical review. You will create tasks that assess whether AI systems can identify vulnerabilities, understand their underlying causes, reproduce security issues, and implement reliable fixes rather than simply recognizing familiar patterns or copying existing patches.

The ideal candidate has strong professional software engineering experience, deep practical knowledge of application security, and the ability to independently investigate vulnerabilities and build reproducible testing environments.

Engagement Type: Contractor assignment
Duration: 3 months
Start Date: Expected to start next week
Commitment: 40 hours per week
Work Arrangement: Fully remote
Project Focus: AI coding model training and evaluation, software vulnerability research, and secure patch development.

Key Responsibilities

  • Vulnerability Research: Source and investigate real-world vulnerabilities from CVEs, security advisories, bug reports, and open-source repositories.
  • Security Classification: Identify and classify vulnerabilities using the most specific applicable CWE and map them to relevant OWASP Top 10 categories where appropriate.
  • Vulnerability Reproduction: Reproduce security issues and develop reliable proof-of-vulnerability tests that demonstrate the underlying weakness.
  • Benchmark Development: Build Harbor-format audit-and-patch tasks across multiple programming languages and software ecosystems.
  • Task Authoring: Write realistic task instructions that describe the security issue without revealing vulnerable code, exploit details, or the expected remediation.
  • Environment Setup: Create reproducible Docker-based environments with pinned dependencies, clear setup instructions, and reliable execution procedures.
  • Verifier Development: Design and harden automated verifiers to prevent gaming, test manipulation, hardcoded solutions, and overfitting.
  • Patch Evaluation: Review and score AI-generated patches and technical reasoning using structured security and engineering rubrics.
  • Quality Assurance: Identify tasks that can be solved through superficial pattern matching or copying known upstream patches without understanding the vulnerability.
  • Technical Documentation: Document vulnerability context, reproduction steps, expected behavior, patch requirements, test methodology, and review decisions.

Requirements
  • 5+ years of professional software engineering experience.

  • Expert-level proficiency in at least one of the following programming languages:

    • Python

    • JavaScript / TypeScript

    • Java

    • C / C++

    • C#

    • Go

    • Rust

    • Ruby

    • PHP

    • Kotlin

    • Swift

  • Strong understanding of software security principles and common vulnerability classes.

  • Practical knowledge of CWE, CVE, and OWASP Top 10.

  • Ability to independently reproduce vulnerabilities and develop proof-of-vulnerability tests.

  • Hands-on experience with security testing and analysis tools, including:

    • Sanitizers

    • Fuzzers

    • Static-analysis tools such as CodeQL or Semgrep

  • Strong proficiency with Git, Linux, Bash, and Docker.

  • Ability to create automated tests and reproducible technical evaluation environments.

  • Excellent technical writing skills, with the ability to explain complex security issues and remediation requirements clearly.

  • Ability to work independently in a fully remote, research-oriented environment.

Similar Jobs

2 Days Ago
Remote
India
Senior level
Senior level
Artificial Intelligence • Information Technology • Professional Services • Consulting
Lead a team developing AI coding-model security benchmarks from real-world vulnerabilities. Manage delivery, staffing, quality assurance, client reporting, calibration, and documentation while contributing hands-on vulnerability research, reproduction, proof-of-vulnerability testing, and audit-and-patch task development.
Top Skills: BashCC#C++CodeqlCveCweDockerFuzzersGitGoJavaJavaScriptKotlinLinuxOwasp Top 10PHPPythonRubyRustSanitizersSemgrepSwiftTypescript
5 Days Ago
Remote
Entry level
Entry level
Events • Food • Travel • Hospitality • Hotels & Resorts
Provides administrative and guest-service support to the General Manager and department managers. Responsibilities include word processing, data entry, filing, copying, answering phones, managing email and correspondence, responding to guests, coordinating travel bookings, and assisting with special projects and departmental initiatives.
10 Days Ago
Remote
Entry level
Entry level
Events • Food • Travel • Hospitality • Hotels & Resorts
Provides friendly, efficient food and beverage service to hotel guests. Responsibilities include learning menu items and promotions, taking and entering orders into the POS system, delivering food and drinks, maintaining a stocked and organized service station, and following health, safety, sanitation, and alcohol-awareness standards.
Top Skills: Point-Of-Sale (Pos) System

What you need to know about the Pune Tech Scene

Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account