Design, build, and operate high-throughput data platform components and pipelines using Spark and Kafka. Debug, tune, and optimize jobs for massive transaction volumes, collaborate on architecture and standards, and mentor junior engineers.
Come work at a place where innovation and teamwork come together to support the most exciting missions in the world!
Senior Big Data EngineerJob Description: We are seeking a Senior Big Data Engineer to design, build, and operate core components of the data platform and pipeline architecture behind Qualys' Enterprise TruRisk Platform, processing billions of daily transactions.
Responsibilities:
- Design, build, and maintain high-throughput data platform and pipeline components using Spark and Kafka.
- Debug, tune, and optimize Spark jobs and Kafka pipelines to manage growing data volumes.
- Collaborate with product, professional services, and sales engineering teams on architecture design and technical trade-offs.
- Establish and refine technical standards, system documentation, and best practices across the engineering team.
- Mentor junior and mid-level engineers on complex technical issues and pipeline performance.
Required Qualifications:
- Minimum 5 years of experience in data engineering or big data platform development.
- Bachelor's degree in Computer Science, Computer Engineering, or a related technical field.
- Demonstrated experience designing, building, and optimizing distributed pipelines using Apache Spark and Apache Kafka.
- Hands-on experience scaling data platform infrastructure to process high-volume/high-throughput transaction workloads.
Preferred Qualifications:
- Experience working with high-volume, multi-tenant enterprise platform environments.
- Proficiency in Scala, Java, or Python for data processing applications.
- Proven experience mentoring junior or mid-level engineering team members.
Qualys Pune, Mahārāshtra, IND Office
Survey No. 20, 10th to 16th Floor, Tower B Panchshil Business Park, Balewadi, Pune, Maharashtra , India, 411045
Qualys Shivaji Nagar, Maharashtra, IND Office
Survey No. 20, 10th to 16th Floor, Tower B Panchshil Business Park,, Shivaji Nagar, 411005, India
Similar Jobs
Information Technology • Professional Services • Software • App development
Lead the design, implementation, maintenance, and monitoring of scalable Big Data pipelines. Build Java/Scala-based systems using distributed processing frameworks, ETL tools, and cloud-based infrastructure. Review and optimize existing pipelines, troubleshoot production issues, lead technical teams, coordinate with US and India engineers, and deliver new modules and features using Agile practices.
Top Skills:
AkkaApache AirflowApache NifiSparkApache StormGitGradleHadoopJavaJIRAMavenSbtScala
Information Technology • Security • Cybersecurity
Lead design, development, and support of a scalable, distributed SaaS security risk platform. Provide technical leadership on data platform, pipelines, and architecture; prototype and implement big data solutions; ensure performance, scalability, security and compliance; mentor engineers; collaborate with product, services and sales; perform benchmarking, troubleshooting, and guide cross-team delivery.
Top Skills:
AirflowApache FlinkSparkCassandraData LakeElasticsearchHadoopJavaKafkaKappa ArchitectureLambda ArchitectureMessage QueueMicroservicesNoSQLOraclePrestoRdbmsRedisRule EngineScalaSolrUnix
Big Data • Fintech • Information Technology • Business Intelligence • Financial Services • Cybersecurity • Big Data Analytics
Leads TransUnion’s Record to Report accounting team in Pune, overseeing US GAAP month-end and quarter-end close, journal entries, reconciliations, audits, controls, KPI reporting, risk management, budgeting, and continuous improvement. The role manages multiple accounting teams, partners with controllers and global stakeholders, ensures service-level delivery, develops employees, and uses Tableau and Power BI dashboards to improve process performance.
Top Skills:
BlacklineExcelOnestreamOracle PeoplesoftPower BITableau
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
