Design and optimize data pipelines using Azure Databricks and Spark, ensuring data reliability and quality through ETL/ELT workflows and CI/CD automation.
Key Responsibilities Design, develop, and optimize data pipelines using Azure Databricks. Implement ETL/ELT workflows leveraging Spark (PySpark/Scala). Integrate and manage data from multiple sources such as Azure Data Lake, SQL DB, Synapse, and external APIs. Design and enforce data quality rules, validation checks, and monitoring frameworks. Work with data quality tools to profile, cleanse, and validate data. Ensure data pipelines follow best practices for performance, scalability, and cost optimization. Collaborate with data engineers, analysts, and business stakeholders to ensure data reliability. Implement CI/CD pipelines and automation for Databricks deployments. Maintain documentation for data processes, pipelines, and quality rules.
Hexaware Technologies Pune, Mahārāshtra, IND Office
North Block, Plot No. 19, Rajiv Gandhi InfoTech Park, MIDC - SEZ, Phase 3, Hinjawadi, Pune, Maharastra, India, 411057
Similar Jobs
Information Technology • Consulting
Design, build, and optimize scalable data pipelines and analytics solutions on Snowflake. Collaborate with teams for data products and enforce security and governance.
Top Skills:
SnowflakeSQL
Information Technology • Consulting
Design and optimize scalable data pipelines and analytics platforms using Snowflake, Airflow, and SQL. Ensure data quality and drive performance optimization.
Top Skills:
AirflowAWSAzureDbtDebeziumDockerFivetranGCPGitKafkaKinesisKubernetesPub/SubPythonSnowflakeSQLTerraform
Information Technology • Consulting
The role requires leading big data projects with hands-on experience in Azure Databricks, Pyspark coding, advanced SQL, and data warehouse management.
Top Skills:
Azure DatabricksPysparkPythonSQL
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
