Design, build, and maintain scalable ETL/ELT pipelines and lakehouse architectures on Azure Databricks. Implement batch and streaming workflows, enforce data modeling and governance, optimize Spark transformations, deploy using CI/CD and IaC, mentor junior engineers, and collaborate with data scientists to productionize ML models.
JOB DESCRIPTION
Senior Data Engineer
Data & Analytics Engineering
You will collaborate closely with data scientists, analysts, and product teams to deliver high-quality, reliable, and performant data solutions in a fast-paced environment.
Senior Data Engineer
Data & Analytics Engineering
About the Role
We are looking for a Senior Data Engineer to join our growing Data & Analytics team. In this role, you will design, build, and maintain scalable data pipelines and platforms that power critical business decisions. You will be a key contributor in driving our cloud-first data strategy, leveraging Azure Databricks as a core technology and Azure Data Factory (ADF) as Good to have feature.You will collaborate closely with data scientists, analysts, and product teams to deliver high-quality, reliable, and performant data solutions in a fast-paced environment.
Key Responsibilities
Data Pipeline & Architecture- Design, develop, and maintain robust ETL/ELT pipelines using Azure Databricks and Azure Data Factory (ADF).
- Architect and implement scalable data lakehouse solutions on Azure using Delta Lake.
- Build and optimize data workflows across batch and streaming workloads (Spark Structured Streaming, Event Hubs).
- Define and enforce data modeling best practices (star schema, data vault, medallion architecture).
- Develop and optimize Spark-based data transformations using PySpark and Spark SQL in Databricks.
- Manage Databricks clusters, jobs, and workspace configurations for performance and cost efficiency.
- Implement Delta Live Tables (DLT) pipelines for declarative, auto-scaling data transformations.
- Leverage Unity Catalog for data governance, lineage tracking, and access control.
- Utilize Databricks Asset Bundles (DABs) and CI/CD practices for deployment automation.
- Build, schedule, and monitor complex ADF pipelines with parameterized templates.
- Integrate ADF with Azure Key Vault, Linked Services, and Integration Runtimes (SHIR/Azure IR).
- Implement incremental load patterns, watermarking, and change data capture (CDC) strategies.
- Troubleshoot pipeline failures and optimize ADF pipeline performance and cost.
- Implement data quality frameworks and validation checks across pipelines.
- Enforce data cataloging, lineage, and metadata management practices.
- Collaborate with data governance teams to ensure compliance with data policies and regulations (GDPR, HIPAA).
- Mentor junior data engineers and conduct code reviews.
- Work closely with data scientists and ML engineers to productionize machine learning models.
- Partner with DevOps/Cloud teams on infrastructure-as-code (Terraform/Bicep) for data platform provisioning.
- Document architecture decisions, pipeline designs, and operational runbooks.
Required Qualifications
- 7+ years of hands-on experience with Azure Databricks (Spark SQL, Delta Lake, PySpark).
- Strong proficiency in SQL preferred and Python.
- Deep understanding of distributed computing principles and the Apache Spark ecosystem.
- Experience with Azure data services: ADLS Gen2, Azure Synapse, Azure SQL, Event Hubs / Kafka.
- Solid understanding of data warehousing concepts and dimensional modeling.
- Experience with version control (Git) and CI/CD tools (Azure DevOps, GitHub Actions).
- Familiarity with infrastructure-as-code tools (Terraform or ARM/Bicep).
Preferred Qualifications
- Databricks Certified Data Engineer Associate or Professional certification.
- Microsoft Certified: Azure Data Engineer Associate (DP-203).
- Experience with Delta Live Tables (DLT) and Databricks Workflows.
- Familiarity with Power BI or other BI/reporting tools.
- Experience with Scala or Java for Spark development.
Technology Stack
Core PlatformsAzure Databricks (Preferred), Azure Data Factory (ADF) (Good to have)Data StorageAzure Data Lake Storage Gen2, Delta Lake, Azure SQL, Cosmos DBLanguagesPython (PySpark), SQLOrchestrationADF Triggers, Databricks Workflows, Apache AirflowMonitoringAzure Monitor, Log Analytics, Databricks Cluster PoliciesGovernanceUnity Catalog, Azure Purview, Azure Key Vault – (Good to have)BI / ReportingPower BI, Tableau Key Competencies
- Strong analytical and problem-solving skills with attention to detail.
- Excellent communication skills — ability to translate complex technical concepts to non-technical stakeholders.
- Self-motivated and able to manage competing priorities in an agile environment.
- Collaborative team player with a growth mindset and eagerness to mentor others.
- Proactive in identifying performance bottlenecks and proposing architectural improvements.
Concord USA Pune, Mahārāshtra, IND Office
Baner Pashan Link Road,, Pune, India, 411 045
Similar Jobs
Fintech • Legal Tech • Software • Financial Services • Cybersecurity • Data Privacy
Lead design, development, and deployment of PEGA-based solutions; oversee delivery team technical direction; collaborate with distributed developers, QA, PM, and product teams; work in Agile/Scrum; perform BPM modeling and JAD workshops; troubleshoot production incidents; enforce PEGA guardrails and maintain quality with automated and manual unit tests; mentor PEGA developers and provide technical leadership.
Top Skills:
AgileBpmJadPegaPega CsaPega CssaPega GuardrailsPega LsaPega PrpcScrum
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Lead application development and technical direction, mentor junior engineers, and integrate AI-assisted engineering (code generation, reviews, research). Design and implement scalable microservices using .NET/C#, TypeScript, SQL and Azure/Terraform. Ensure coding standards, security, quality, and participate in full lifecycle delivery, testing, deployment, support, and cross-functional collaboration.
Top Skills:
.NetAzureC#GitGithub ActionsGithub CopilotLangchainLanggraphMcp-Based DevelopmentMicroservicesMicrosoft 365 CopilotSQLTerraformTypescript
Greentech • Social Impact
Design, build, and scale backend services and APIs; own AWS infrastructure and CI/CD; optimize codebase and databases; ensure security, reliability, and observability; collaborate cross-functionally and evaluate AI tooling.
Top Skills:
Ai/Agentic WorkflowsAWSCdkCi/CdCSSDockerGraphQLHTMLJavaScriptKubernetesNode.jsNoSQLObservability ToolingReactRestSQLTerraformTypescriptVue
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
