Design, develop, optimize, and maintain scalable data pipelines using Databricks, PySpark, Delta Lake, SQL, and Azure services. Apply ETL, data modeling, governance, and DevOps practices while troubleshooting data issues, conducting root-cause analysis, and improving pipeline performance. Collaborate within an agile scrum team, independently deliver production-ready code, and contribute to architecture and design discussions.
Key Responsibilities:
• Design, develop, and optimize data pipelines using Databricks, PySpark, Delta Lake, and related Azure components.
• Work across pipeline development, data modeling, and production code optimization, ensuring scalability and performance.
• Build new pipelines from scratch as well as enhance and maintain existing ones.
• Apply strong understanding of ETL design principles, data modeling concepts, and data governance standards.
• Collaborate within a scrum team, taking ownership of assigned stories while independently delivering high-quality, production-ready code.
• Demonstrate proficiency across PySpark, SQL, Delta Lake, Unity Catalog, and Databricks Workflows, with solid understanding of logic and data flow.
• Work in Azure environments, leveraging tools like ADLS, ADF, and Synapse (as applicable).
• Contribute inputs to architecture and design discussions where appropriate, while primarily focusing on hands-on development.
• Troubleshoot data issues, perform root-cause analysis, and optimize performance for existing pipelines.
Requirements
Desired Skills and Experience:
• 2-3 years of experience in data engineering, preferably with Azure Databricks.
• Strong technical expertise in PySpark, SQL, and Delta Lake.
• Familiarity with Unity Catalog, data governance, and DevOps practices (Git, CI/CD).
• Ability to work independently as well as collaboratively within an agile delivery team.
• Excellent problem-solving, debugging, and communication skills.
Similar Jobs
Financial Services
Lead design, build, deployment, and support of IBM Planning Analytics/TM1 solutions on AWS. Architect TM1 models, develop PAW/PAfE artifacts, manage TI processes and integrations, establish governance and SDLC controls, lead a TM1 team, drive AI-assisted engineering adoption, and partner with business stakeholders to optimize planning processes and ensure compliance and resiliency.
Top Skills:
ArchitectAWSCi/CdCognos AnalyticsContainerizationDatabricksIbm Cloud For DataIbm Planning AnalyticsPerspectivesPlanning Analytics For Excel (Pafe)Planning Analytics On CloudPlanning Analytics Workspace (Paw)PythonTm1Turbointegrator (Ti)
Artificial Intelligence • Analytics • Business Intelligence • Consulting
Design, develop, optimize, and maintain scalable data pipelines using Databricks, PySpark, Delta Lake, SQL, and Azure services. Responsibilities include data modeling, ETL development, governance, troubleshooting, root-cause analysis, production-code optimization, and performance improvement. The role collaborates within an agile Scrum team, independently delivers production-ready code, contributes to architecture discussions, and uses DevOps practices such as Git and CI/CD.
Top Skills:
AdfAdlsAzureCi/CdDatabricksDatabricks WorkflowsDelta LakeGitPysparkSQLSynapseUnity Catalog
Cloud • Fintech • Software • Business Intelligence • Consulting • Financial Services
Lead and deliver full-stack .NET web projects, write and maintain code (.NET Core, C#), integrate cloud and web technologies, drive technical decisions, ensure unit testing and CI, collaborate with stakeholders and overseas teams.
Top Skills:
.Net.Net CoreAngularAWSAzureAzure DevopsBootstrapC#CSSEntity FrameworkJenkinsJqueryJSONMocking FrameworksMstestNinject (Di Frameworks)NunitOrmReactSQLTeamcityTypescriptUnit TestingWeb Api
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
