Top Data Engineer Jobs in Pune

Designs and operates scalable data and machine learning platforms using Databricks, Python, SQL, Spark, and PySpark. Builds batch, streaming, and production ML pipelines; manages Delta Lake, Unity Catalog, Workflows, MLflow, governance, data quality, observability, and model deployment. Partners with data scientists and business teams, optimizes performance and costs, contributes to platform architecture, mentors engineers, and documents operational practices.

The Senior Data Engineer will build and optimize ETL/ELT data pipelines using Azure Databricks and Apache Spark, ensure data governance, and collaborate with stakeholders to translate business needs into data engineering solutions.

Design, develop, test, and maintain scalable Big Data systems and microservices. Own features, collaborate cross-functionally, leverage AI-assisted development tools, and improve engineering processes. Work with Spark, Hadoop, Kafka, containerization (Docker, Kubernetes, OpenShift) to solve complex technical challenges and deliver secure, high-quality solutions.

Design, build and maintain cloud-oriented big data solutions for ingestion, processing, cleaning and exposition. Improve data quality, implement ETL and data-warehouse architectures, and collaborate with DDS/Data Foundation teams to ensure platform reliability.

Design, implement, and optimize Apache DolphinScheduler workflows and ETL pipelines. Integrate with Hadoop, Spark, Flink, Hive, and Kafka. Develop custom plugins, troubleshoot performance and scalability issues, monitor job execution and resource allocation, and maintain CI/CD, version control, and infrastructure automation while collaborating with data engineers and DevOps teams.

Design, build, and optimize PySpark applications and ETL pipelines to process large-scale datasets from relational, NoSQL, file, and streaming sources. Ensure data quality, implement error handling, and collaborate with analysts, scientists, and architects to deliver performant data solutions.

Design, configure, and support backend software and APIs; manage configuration files and environments; test and document API endpoints; collaborate on system data flow; participate in large-scale live data migrations (billions of records) into FinCloud; produce documentation and troubleshooting guides; work in agile teams and support integration and observability efforts.

Design, develop, test, and deploy high-performance Spark/Scala data processing applications and ETL pipelines on Cloudera (CDH). Optimize Spark and Cloudera platform performance, integrate HDFS/Hive/Impala/HBase/Kafka, ensure data integrity/security, and implement version control and CI/CD for production pipelines.

Build and scale an AI-powered stock research platform: design data pipelines ingesting SEC and market data, develop backend APIs, implement semantic search with embeddings and vector DBs, improve production reliability and performance, and own features end-to-end from design through deployment and monitoring.

As a Senior Engineer for Strategic Data Solutions, you will manage data warehousing, reporting, and analytics for internal operations and external partners. Responsibilities include collaborating with business units, providing insights, and ensuring high-quality data reporting.

Responsible for guiding teams in utilizing technology effectively, mentoring colleagues and devising advanced solutions to meet client challenges.

As a Senior Engineer in Data Analytics, you'll work on challenging problems using cutting-edge technology alongside talented colleagues to drive innovative solutions.

The Senior Staff Engineer, Data Science role involves guiding team direction, solving complex challenges for clients, and leveraging technical expertise to innovate solutions.

The Principal Engineer, Data Science role involves leading data science projects and providing strategic insights using extensive IT experience.

Design and own end-to-end data platform architecture, build ingestion and transformation pipelines, model multi-tenant datasets, optimize Postgres and ClickHouse performance/cost, implement orchestration and observability, define data quality and reconciliation standards, and mentor data/product engineers.

Support electrical design, engineering, commissioning, and delivery of liquid-cooled data centre solutions: prepare layouts, SLDs, wiring diagrams, BOMs, perform load calculations, assist testing/FAT/SAT, troubleshoot commissioning issues, coordinate cross-functional teams, and ensure compliance with IEC/NEC/IEEE and company HSE standards.

Design, implement, and lead large-scale generative AI and ML systems (LLM fine-tuning, RAG, agents, Graph ML). Build REST APIs, deploy with Docker/Kubernetes, optimize big-data workflows, mentor engineers, and operate ML services on cloud platforms.

Own and automate end-to-end incentive calculation pipelines, translate business rules into auditable data workflows, build monitoring and reconciliation dashboards, ensure accuracy and auditability across multiple products and countries, and partner with Sales, Finance, and field teams to resolve discrepancies and support escalations.

Build and maintain end-to-end data pipelines using PySpark and SQL on Databricks/Delta. Establish data modeling and pipeline best practices, create AI-ready analytical datasets, troubleshoot pipeline issues, and collaborate with stakeholders to embed business logic and optimize reliability, efficiency, and performance.

Lead insider risk and data loss investigations from triage to closure, operate and tune DLP across environments, build detections and workflows, partner with HR/Legal/Compliance/IT, mature insider risk program, support audits, and mentor others.

As a Data Engineer, you'll build and monitor large-scale data pipelines, develop the DBT setup for data transformation and solve customer problems using your SQL skills.

Manage and support AKS-based enterprise data platform infrastructure, administer Apache NiFi, Azure Event Hub and Functions, implement CI/CD with Azure DevOps, automate infrastructure with Terraform/Bicep/ARM, monitor with Azure Monitor and Log Analytics, troubleshoot production incidents, ensure security, capacity planning, and collaborate with data engineering teams.

Lead the architecture and implementation of a distributed data platform for microscopy, ensuring scalability and high-performance in data management.

As a key member of the Data Infrastructure team, you will develop and optimize software features for Apache Hudi's transactional engine, enhance database performance, and solve complex data processing challenges.

Design, build, and optimize high-volume data pipelines using Python, Databricks and Spark on Azure. Create and maintain automation test suites (Azure DevOps), perform manual/load/exploratory testing, collaborate with BAs and developers to meet sprint goals, and assist with sprint estimation and delivery.