Design and optimize scalable ETL/ELT pipelines and modern data platforms using Databricks, PySpark, Azure Data Factory, ADLS, Delta Lake, and Synapse Analytics. Develop data ingestion and transformation workflows, optimize Spark performance, implement CI/CD automation, and uphold data quality, security, and governance standards. Collaborate with architects, analysts, and business stakeholders on cloud-based data engineering solutions.
Job
Description – Lead/Senior Databricks Engineer
Experience: 8+ Years
Location: Pune / Hyderabad (Hybrid)
Employment Type: Full-Time
Role Summary
We are looking
for a skilled Data Engineer with strong experience in Databricks and Azure Cloud to design, develop, and optimize scalable data pipelines
and modern data platforms. The ideal candidate should have expertise in big
data technologies, ETL development, and cloud-based data engineering.
Key
Responsibilities
- Design and develop scalable ETL/ELT pipelines using Databricks and PySpark.
- Build and optimize data ingestion frameworks from
multiple data sources.
- Develop data pipelines using Azure Data Factory
(ADF).
- Work with Azure Data Lake Storage (ADLS Gen2) for storing and processing large datasets.
- Implement Delta Lake and Lakehouse
architecture concepts.
- Optimize Spark jobs for performance and scalability.
- Develop data transformations using SQL and PySpark.
- Collaborate with data architects, analysts, and
business stakeholders.
- Implement CI/CD pipelines and deployment automation.
- Ensure data quality, security, and governance
standards.
Required
Skills
- Strong hands-on experience with Databricks
- Proficiency in PySpark and Spark SQL
- SSIS, Essbase, SQL Server, Data Warehouse, Stored
Procedure, Views Triggers
- Experience with Microsoft Azure
- Good knowledge of:
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Delta Lake
- Azure Synapse Analytics
- Strong SQL and data modeling skills
- Experience with Git and CI/CD processes
- Knowledge of performance tuning and optimization
Preferred
Skills
- Experience with:
- Streaming using Spark Structured
Streaming
- Kafka/Event Hub
- Data Warehousing concepts
- DevOps and Infrastructure as Code
- Knowledge of:
- Data Governance
- Data Quality frameworks
- Medallion Architecture (Bronze,
Silver, Gold)
Educational
Qualification
- Bachelor's degree in Computer Science, Information
Technology, or related field.
- Relevant Azure or Databricks certifications are
preferred.
Nice to Have
Certifications
- Databricks Certified Data Engineer Associate
- Microsoft Certified Azure Data Engineer Associate
Requirements
Required
Skills
- Strong hands-on experience with Databricks
- Proficiency in PySpark and Spark SQL
- SSIS, Essbase, SQL Server, Data Warehouse, Stored
Procedure, Views Triggers
- Experience with Microsoft Azure
- Good knowledge of:
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Delta Lake
- Azure Synapse Analytics
- Strong SQL and data modeling skills
- Experience with Git and CI/CD processes
- Knowledge of performance tuning and optimization
DATAECONOMY Pune, Mahārāshtra, IND Office
1001, 10th Floor, AP81, North Main Road, Koregaon Park Annex, Mundhwa,, Pune, India, 411006
Similar Jobs
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
Develop and maintain Azure Databricks data pipelines using PySpark, Spark SQL, SQL, and Delta Lake. Support Informatica-to-Databricks modernization and HR data integrations, build source-to-target mappings and dimensional models, perform profiling, testing, reconciliation, and defect resolution, and create technical documentation. Apply data governance and access controls through Unity Catalog, participate in agile ceremonies and code reviews, and escalate risks while working under senior direction.
Top Skills:
SparkAzure Data FactoryAzure Data Lake StorageAzure DatabricksAzure DevopsAzure SqlAzure SynapseCi/CdDelta LakeGitGithub CopilotInformaticaMicrosoft PurviewOracle Hcm CloudPysparkPythonSpark SqlSQLUnity Catalog
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
Designs and deploys scalable Azure Databricks data pipelines using PySpark, Spark SQL, and Delta Lake across medallion layers. Builds HR data models, source-to-target mappings, reconciliation controls, governance, lineage, documentation, and data quality processes. Modernizes legacy Informatica and SQL integrations, including Oracle HCM feeds. Collaborates with distributed agile teams, mentors junior engineers, and supports production cutovers for sensitive employee data.
Top Skills:
Ai/Copilot ToolingSparkAzure Data FactoryAzure Data Lake StorageAzure DevopsAzure SqlAzure SynapseCi/CdCrunchrDatabricksDelta LakeGitInformaticaOracle Hcm CloudPower BIPysparkSpark SqlSQLUnity Catalog
Financial Services
Designs, develops, and supports Java backend services and Python-based Databricks applications. Builds REST APIs, event-driven microservices, and AWS deployments while applying resilient architecture practices. The role includes automated testing, code reviews, production troubleshooting, observability, DevOps, CI/CD, containerization, Kubernetes, infrastructure as code, and MLOps. Collaborates with product, design, and engineering teams, uses approved AI-assisted development tools responsibly, and validates AI-generated code for quality, performance, security, and reliability.
Top Skills:
Ai-Assisted Software Development ToolsApi GatewayAWSCi/CdCloudwatchDatabricksDockerEc2EcsEksElkGrafanaInfrastructure As CodeJavaKafkaKubernetesLambdaMicroservicesMlopsPrometheusPythonRabbitMQReactRest ApisS3
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
