Top Data Engineer Jobs in Pune

Designs, develops, supports, and optimizes scalable data pipelines and distributed processing workloads. Writes production-grade Python, PySpark, and SQL; tunes Spark and SQL performance; troubleshoots Linux and AWS infrastructure; improves monitoring, reliability, security, and cost efficiency; performs root cause analysis; and collaborates with engineering, product, architecture, and operations teams. The role also supports AI-enabled data workflows using LLMs, Amazon Bedrock, Agentic AI, and MCP.

Develop and maintain batch and real-time data pipelines using Java, Python, Spark, SQL, and Kafka. Build ETL/ELT solutions across data lakes and warehouses, perform data validation, optimize processing, and support cloud deployments, CI/CD, monitoring, and production troubleshooting. Collaborate with engineering and platform teams while developing reusable components and data pipelines supporting analytics, machine learning, and generative AI initiatives.

Designs and develops high-performance API platforms for transaction data using Java, microservices, and distributed systems. Leads architecture, technical roadmaps, system development, maintenance, troubleshooting, performance optimization, and delivery against SLAs. Guides engineers, conducts technical interviews, coaches team members, and collaborates globally. Requires expertise in Spring, SQL and NoSQL databases, cloud hosting, automated testing, and CI/CD practices.

Develop secure, scalable data pipelines and data engineering solutions using Spark, Scala, Python, Java, Hadoop, and object storage. Build and maintain data warehouse projects, automate data ingestion and movement workflows, enforce data standards, and support production incidents. The role requires strong SQL and database expertise, analytical debugging skills, collaboration across distributed teams, and continuous process improvement.

Lead design and build cloud-native batch and real-time data platforms. Develop scalable ETL/ELT and streaming pipelines (Spark, Kafka, Flink), ensure data quality, governance, observability, security, cost optimization, and drive platform engineering excellence and team mentorship.

Leads enterprise data quality strategy, architecture, platforms, frameworks, observability, governance, and automation. Designs reusable quality capabilities for batch, streaming, cloud, on-premises, and SaaS environments; establishes rules, metrics, alerts, and remediation workflows; integrates quality with metadata, lineage, catalogs, MDM, and data contracts; mentors engineering teams; evaluates tools; and applies AI and machine learning to anomaly detection, root-cause analysis, and trusted AI-ready data platforms.

Designs, builds, and maintains scalable batch and real-time data pipelines across Azure and on-premises environments. Develops data lakes, warehouses, marts, and Medallion architectures using Spark and big data technologies. Ensures pipeline reliability, performance, security, governance, privacy, and data quality while supporting platform modernization. Collaborates with business, technology, operations, data governance, data science, DevOps, and BI teams.

Build and maintain large-scale backend data systems, batch and streaming pipelines, datamarts, and cloud data workloads. Develop AI-powered agents for anomaly detection and workflow automation, using Scala, Python, Java, Spark, Cloudera, Databricks, and AWS. Support platform reliability, data integrity, downstream analytics, documentation, Agile collaboration, and team mentoring.

Designs and develops scalable data management solutions, including data acquisition, integration, transformation pipelines, data structures, and cloud or edge infrastructure. Uses Azure Data Factory, ADLS, Databricks, SAP Data Services, DevOps pipelines, and containerization. Collaborates with analysts, architects, product owners, and technical leads to define requirements, deliver high-quality code, and support application enhancements.

Build and manage scalable data pipelines and Celonis data models for process mining and business intelligence. Responsibilities include integrating enterprise data through connectors, APIs, databases, and extracts; developing SQL transformations; validating data quality; troubleshooting loads and performance; and translating banking processes into reliable models. The role also involves Agile delivery, stakeholder collaboration, production support, governance, risk controls, and creating reusable engineering patterns across financial services processes.

Build, test, release, monitor, and support platform services, APIs, connectors, and automation for customer data exchange and pipeline orchestration. Deliver components independently, write tests and documentation, replace manual processes with code, investigate production issues, and contribute to design, estimation, code reviews, and root cause analysis. The role uses cloud technologies, data platforms, secure coding practices, and CI/CD.

Execute large-scale data platform migrations to Snowflake, adapting approximately 1,000 datasets and pipelines. Build, test, monitor, and troubleshoot Airflow workflows while supporting migrations from Airflow 2 to 3, Confluent region changes, and Iceberg adoption. Ensure data quality and reliability using Python, advanced SQL, Spark, PySpark, Kafka, Snowflake, and AWS S3. Collaborate with data platform teams on modernization initiatives and meet time-sensitive delivery targets.

Design, develop, and optimize scalable ETL/ELT pipelines using Databricks, PySpark, SQL, GCP, BigQuery, and Delta Lake. Build agentic workflows that automate data validation, troubleshooting, and operational processes. Develop reusable frameworks, implement data quality and monitoring controls, optimize workloads, and support testing, deployment, productionization, and ongoing enhancements across data and AI-enabled solutions.

Designs and maintains scalable Azure data pipelines, ETL processes, integrations, and data models supporting Procurement modernization. Uses Azure Data Factory, Data Lake, Azure SQL, Functions, Terraform, SQL, and Python. Collaborates with technical and business stakeholders, optimizes performance and cost, implements security controls, troubleshoots production issues, documents architecture, and contributes to testing, deployments, and code reviews.

Build and maintain data pipelines, warehouses and lakes; design processing and analysis algorithms; ensure data accuracy, security and performance; collaborate with data scientists to deploy ML models; lead or support team members and embed risk controls and performance engineering throughout the SDLC.

Lead the architecture and development of scalable batch, streaming, and event-driven data platforms supporting analytics, machine learning, generative AI, and agentic AI. Build governed lakehouse architectures, reusable data products, ingestion and orchestration pipelines, and AI lifecycle capabilities. Drive security, governance, observability, reliability, performance, and cost optimization while translating R&D concepts into production systems. Provide hands-on technical leadership, architecture reviews, mentorship, and cross-functional collaboration.

Lead the architecture and hands-on development of scalable batch, streaming, and event-driven data platforms supporting analytics, machine learning, generative AI, and agentic AI. Build governed lakehouse architectures, reusable data products, ingestion and orchestration pipelines, and ML lifecycle capabilities. Establish standards for security, governance, testing, CI/CD, observability, reliability, and cost optimization while providing technical leadership, mentoring, architecture reviews, and cross-functional collaboration.

Build and maintain scalable data platforms, pipelines, data warehouses, data lakes, and real-time processing systems. Develop Kafka, Flink, and Spring-based applications; parse financial message formats; transform data into Avro schemas; integrate source systems; and implement testing, logging, security, and governance. Collaborate with data scientists and cross-functional teams, participate in Scrum and code reviews, troubleshoot production issues, and guide teams on complex data engineering assignments.

Design, build, and maintain data pipelines, data warehouses and lakes; implement data processing and analysis algorithms; collaborate with data scientists to deploy ML models; lead performance engineering, system stability and scalability efforts; partner with cross-functional teams and ensure compliance with controls and service-level expectations.

Lead technical design and engineering standards for scalable data platforms using Databricks, Spark, Python, PySpark, and dbt. Own architecture reviews, CI/CD, DataOps, observability, platform administration, deployment governance, migrations, and upgrades. Resolve complex platform challenges, mentor engineers, and guide code quality. Partner with stakeholders on roadmaps, prioritization, and technical trade-offs while supporting Mastercard’s cloud strategy.

Maintain Python-based web scraping configurations, monitor errors and data blockages, coordinate scraping requirements with stakeholders, and report on scraping activities. Build and optimize pipelines for data lake ingestion and transformation using ETL, data modeling, cloud, database, and infrastructure tools. Collaborate with data scientists and analysts while ensuring data security, compliance, testing, and maintainability.

Build and maintain governed finance data products on Microsoft Fabric and Databricks. Responsibilities include developing PySpark and Python ingestion pipelines, implementing incremental loading and CDC, optimizing Delta Lake tables, managing Azure Data Factory pipelines and Analysis Services cubes, designing Power BI semantic models, and ensuring version control, testing, monitoring, governance, and data contract compliance. The role also involves stakeholder collaboration, mentoring junior engineers, and applying enterprise data architecture standards.

Build and support large-scale, real-time kdb+ analytics platforms processing billions of daily messages. Develop kdb Insights or KDB-X modules, packages, pipelines, performant q queries, and functions for trading desks. Create SRE tools to detect performance bottlenecks across q, Python, and Java processes running continuously in AWS. Support ingestion stability, Kubernetes deployments, PyKX data processing, and cloud-native analytics infrastructure.

Build and operate production data platforms, tooling, and infrastructure supporting autonomous pipeline delivery. Own monitoring, alerting, DataOps, Kafka, Kubernetes, Terraform, AWS, Databricks/Spark, dbt, and Airflow environments. Improve platform reliability, troubleshoot production issues, and partner with product, analytics, and infrastructure teams. Champion AI adoption and automation while enabling high-throughput, low-latency data processing and strong developer experience.

Build production-grade agentic AI systems that reason, search, and execute workflows. Develop RAG pipelines, AI agents, semantic search layers, data processing solutions, dimensional data models, and scalable AI workloads. Work with structured and unstructured data, LLMs, vector databases, cloud infrastructure, CI/CD, and ML observability. Responsibilities include fine-tuning models, prompt engineering, optimizing context windows, and deploying reliable AI applications.