Build, optimize, and monitor large-scale data pipelines using Databricks, Spark, and Delta Lake. Implement ingestion, transformation, and integration workflows, ensure data quality and performance, enable analytics with stakeholders, apply lakehouse and cloud best practices, and work in an Agile delivery model.
- Design, develop, and optimize data pipelines using Databricks, Apache Spark, and Delta Lake.
- Implement data ingestion, transformation, and integration workflows for large-scale datasets.
- Ensure data quality, reliability, and performance across all pipelines.
- Collaborate with data architects and analysts to enable analytics and reporting capabilities.
- Apply best practices for Lakehouse architecture and cloud-based data engineering.
- Monitor and troubleshoot pipeline performance, ensuring scalability and efficiency.
- Work within an Agile framework for sprint planning, backlog management, and continuous delivery.
- Design, develop, and optimize data pipelines using Databricks, Apache Spark, and Delta Lake.
- Implement data ingestion, transformation, and integration workflows for large-scale datasets.
- Ensure data quality, reliability, and performance across all pipelines.
- Collaborate with data architects and analysts to enable analytics and reporting capabilities.
- Apply best practices for Lakehouse architecture and cloud-based data engineering.
- Monitor and troubleshoot pipeline performance, ensuring scalability and efficiency.
- Work within an Agile framework for sprint planning, backlog management, and continuous delivery.
- 5+ years of experience in data engineering with strong focus on Databricks.
- Hands-on expertise in Apache Spark, Delta Lake, and Databricks Lakehouse architecture.
- Strong knowledge of Python, PySpark, and SQL.
- Experience with cloud platforms (Azure preferred) and data integration tools.
- Strong SQL skills (joins, window functions, optimization)
- Experience with cloud platforms (Azure / AWS / GCP)
- Knowledge of ETL/ELT, CDC, SCD (Type 1 & 2)
- Familiarity with CI/CD pipelines and DevOps practices.
- Understanding of data governance, metadata management, and performance tuning.
- Excellent problem-solving and communication skills.
EXL Pune, Mahārāshtra, IND Office
Pune, India
Similar Jobs
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
Design, build, and maintain cloud-native data pipelines and ETL/ELT workflows for high-volume trade lifecycle data. Integrate DTCC sources (RTTM, Settlement Web) into a Unified Trade Record, process ISO 15022 messages, manage trade exception databases, optimize SQL Server performance, and ensure data quality, lineage, governance, and observability while collaborating with business stakeholders.
Top Skills:
AWSDockerDtcc RttmDtcc Settlement WebEksEltETLFixFxmlIbm MqIndexingIso 15022KafkaKubernetesRdsRest ApiS3SQLSQL ServerStored ProceduresUnified Trade Record (Utr)
HR Tech • Legal Tech • Software • Consulting
Lead technical direction for data platforms, design and maintain data pipelines for reporting and AI (RAG, embeddings, vector stores), reduce architectural debt, mentor engineers, partner with product leadership, and represent Mitratech externally.
Top Skills:
AirbyteAnalytics PlatformsAWSBi ToolsCi/CdClaude CodeCloud StorageCopilotCursorDbtDbt Semantic LayerEmbeddingsEtl/EltFivetranGitInfrastructure-As-CodeMonitoringPostgresRag IngestionReactRuby On RailsSnowflakeSQLTerraformVector Stores
Cloud • Security • Software • Cybersecurity • Automation
Lead design and implementation of in-product insights built on the Data Insights Platform and Knowledge Graph. Architect data ingestion, modeling, and synchronization; integrate third-party systems; design a Data Marketplace for Snowflake/Databricks/BigQuery access; publish observability via OpenTelemetry; define operational readiness across deployment types; coordinate cross-team delivery; and mentor senior/intermediate engineers.
Top Skills:
BigQueryData Insights PlatformDatabricksGoJIRAKnowledge GraphNode.jsOpentelemetryRubyServicenowSnowflakeZendesk
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
