Build and operate Microsoft Fabric data pipelines for six heterogeneous sources. Responsibilities include ingestion, mirroring and CDC, Delta Lake raw-layer management, PySpark standardization and transformation, data quality checks, monitoring, troubleshooting, performance tuning, and source-to-target documentation. The role supports entity resolution through reliable, observable pipelines and identifier-spine construction.
Build and operate the data pipelines that feed the Entity Hub. This role lands all six in-scope sources into Fabric, implements standardization and transformation logic, and maintains the data quality checks and monitoring that the entity resolution engine depends on. Reliable, observable ingestion is the foundation the entire programme rests on.
Responsibilities- Ingestion development — build and maintain pipelines to land the six in-scope sources (Secretary of State, D&B, ARROW, E1, hCue, DocCentral) into the Fabric Bronze/raw layer.
- Mirroring & CDC — implement Fabric Mirroring for supported structured sources and establish change-data-capture patterns; implement watermark/incremental load logic where mirroring is unavailable.
- Raw layer management — maintain one Delta table per source on an append-only basis, retaining evidence records and full source provenance.
- Standardization & transformation — implement name normalization, address parsing and attribute standardization logic in Spark notebooks; support identifier-spine construction.
- Data quality — implement data quality checks, validation rules, threshold alerts and exception handling; support reconciliation against source.
- Pipeline operations — schedule, monitor and troubleshoot pipeline runs; investigate failures and performance issues; maintain run documentation.
- Performance tuning — optimise Spark jobs, Delta file sizes, partitioning and pipeline efficiency to manage Fabric capacity consumption.
- Documentation — produce and maintain source-to-target mappings, transformation logic documentation and lineage records.
- + years hands-on data engineering with strong PySpark and SQL
- Production experience building ingestion pipelines from multiple heterogeneous sources
- Working knowledge of Delta Lake and medallion/lakehouse architecture
- Experience implementing incremental loads and CDC-style processing
- Experience implementing data quality checks and troubleshooting pipeline failures
Nice-to-Have
- Microsoft Fabric hands-on experience (Mirroring, Copy Jobs, Environments)
- Exposure to entity/master data standardization (name and address parsing)
- Familiarity with libraries such as Great Expectations for data quality
- Experience optimising for Fabric capacity/CU consumption
Key Deliverables Owned
- Operational ingestion pipelines for all agreed sources
- Bronze/raw layer with one Delta table per source and CDC retained
- Standardization and parsing transformation logic
- Data quality checks, monitoring and exception handling
- Source-to-target mapping and run documentation
EXL Pune, Mahārāshtra, IND Office
Pune, India
Similar Jobs
Information Technology • Database • Consulting
Build and deploy machine learning models, predictive analytics, segmentation frameworks, recommendation signals, and data-driven insights. Responsibilities include data preparation, feature engineering, model training and evaluation, quality checks, client deliverable ownership, requirements translation, and collaboration with data engineering, BI, and AI/ML teams. The role also involves analyzing complex datasets and communicating actionable business recommendations.
Top Skills:
AzureMachine LearningExcelNlpPythonSQL
Professional Services • Real Estate • Consulting
Lead delivery of public‑sector infrastructure projects from planning through completion, managing scope, schedule, cost, quality, risks, contractors, procurement and stakeholder relations while ensuring regulatory and safety compliance in accordance with Singapore authority requirements.
AdTech
The Associate Project Manager oversees client projects, ensuring timely communication, adherence to budgets, and high-quality standards while managing resources and risks.
Top Skills:
Microsoft Office SuiteProject Management Software
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
