Design and optimize scalable ETL/ELT pipelines and modern data platforms using Databricks, PySpark, Azure Data Factory, ADLS, Delta Lake, and Synapse Analytics. Develop data ingestion and transformation workflows, optimize Spark performance, implement CI/CD automation, and uphold data quality, security, and governance standards. Collaborate with architects, analysts, and business stakeholders on cloud-based data engineering solutions.
Job
Description – Lead/Senior Databricks Engineer
Experience: 8+ Years
Location: Pune / Hyderabad (Hybrid)
Employment Type: Full-Time
Role Summary
We are looking
for a skilled Data Engineer with strong experience in Databricks and Azure Cloud to design, develop, and optimize scalable data pipelines
and modern data platforms. The ideal candidate should have expertise in big
data technologies, ETL development, and cloud-based data engineering.
Key
Responsibilities
- Design and develop scalable ETL/ELT pipelines using Databricks and PySpark.
- Build and optimize data ingestion frameworks from
multiple data sources.
- Develop data pipelines using Azure Data Factory
(ADF).
- Work with Azure Data Lake Storage (ADLS Gen2) for storing and processing large datasets.
- Implement Delta Lake and Lakehouse
architecture concepts.
- Optimize Spark jobs for performance and scalability.
- Develop data transformations using SQL and PySpark.
- Collaborate with data architects, analysts, and
business stakeholders.
- Implement CI/CD pipelines and deployment automation.
- Ensure data quality, security, and governance
standards.
Required
Skills
- Strong hands-on experience with Databricks
- Proficiency in PySpark and Spark SQL
- SSIS, Essbase, SQL Server, Data Warehouse, Stored
Procedure, Views Triggers
- Experience with Microsoft Azure
- Good knowledge of:
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Delta Lake
- Azure Synapse Analytics
- Strong SQL and data modeling skills
- Experience with Git and CI/CD processes
- Knowledge of performance tuning and optimization
Preferred
Skills
- Experience with:
- Streaming using Spark Structured
Streaming
- Kafka/Event Hub
- Data Warehousing concepts
- DevOps and Infrastructure as Code
- Knowledge of:
- Data Governance
- Data Quality frameworks
- Medallion Architecture (Bronze,
Silver, Gold)
Educational
Qualification
- Bachelor's degree in Computer Science, Information
Technology, or related field.
- Relevant Azure or Databricks certifications are
preferred.
Nice to Have
Certifications
- Databricks Certified Data Engineer Associate
- Microsoft Certified Azure Data Engineer Associate
Requirements
Required
Skills
- Strong hands-on experience with Databricks
- Proficiency in PySpark and Spark SQL
- SSIS, Essbase, SQL Server, Data Warehouse, Stored
Procedure, Views Triggers
- Experience with Microsoft Azure
- Good knowledge of:
- Azure Data Factory (ADF)
- Azure Data Lake Storage (ADLS)
- Delta Lake
- Azure Synapse Analytics
- Strong SQL and data modeling skills
- Experience with Git and CI/CD processes
- Knowledge of performance tuning and optimization
DATAECONOMY Pune, Mahārāshtra, IND Office
1001, 10th Floor, AP81, North Main Road, Koregaon Park Annex, Mundhwa,, Pune, India, 411006
Similar Jobs
Fintech • Professional Services • Consulting • Energy • Financial Services • Cybersecurity • Generative AI
Designs, develops, and maintains scalable batch and streaming data pipelines using Python, Scala, Spark, and Databricks. Responsibilities include optimizing large-scale data workloads, building ETL/ELT processes, modeling data warehouses and lakes, implementing data quality and governance standards, troubleshooting production issues, and collaborating with architects, analysts, and data scientists.
Top Skills:
SparkAws DatabricksAzure DatabricksDatabricksDatabricks WorkflowsDataframesDelta LakeEtl/EltGcp DatabricksPythonScalaSpark SqlSQL
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Lead the architecture and development of scalable batch, streaming, and event-driven data platforms supporting analytics, machine learning, generative AI, and agentic AI. Build governed lakehouse architectures, reusable data products, ingestion and orchestration pipelines, and AI lifecycle capabilities. Drive security, governance, observability, reliability, performance, and cost optimization while translating R&D concepts into production systems. Provide hands-on technical leadership, architecture reviews, mentorship, and cross-functional collaboration.
Top Skills:
Apache AirflowSparkAWSAws Step FunctionsAzure Ai FoundryAzure Data FactoryAzure Machine LearningCi/CdDatabricksDatabricks WorkflowsDockerInfrastructure As CodeKafkaKubernetesMicrosoft FabricMicrosoft PurviewMlflowPysparkPythonSQLTerraformUnity Catalog
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Lead the architecture and hands-on development of scalable batch, streaming, and event-driven data platforms supporting analytics, machine learning, generative AI, and agentic AI. Build governed lakehouse architectures, reusable data products, ingestion and orchestration pipelines, and ML lifecycle capabilities. Establish standards for security, governance, testing, CI/CD, observability, reliability, and cost optimization while providing technical leadership, mentoring, architecture reviews, and cross-functional collaboration.
Top Skills:
Apache AirflowSparkAWSAws Step FunctionsAzure Ai FoundryAzure Data FactoryAzure Machine LearningCi/CdCloud StorageData GovernanceData LakesData LineageData MeshesDatabricksDatabricks WorkflowsDockerEvent-Driven ArchitectureInfrastructure As CodeKafkaKubernetesLakehousesMetadata ManagementMicrosoft FabricMicrosoft PurviewMlflowPysparkPythonServerlessSQLStreamingTerraformUnity Catalog
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
