Palo Alto Labs Logo

Palo Alto Labs

Data Engineer- DataBricks

Posted 2 Months Ago
Be an Early Applicant
In-Office
Pune, Maharashtra, IND
Senior level
In-Office
Pune, Maharashtra, IND
Senior level
Designs and operates Databricks data platforms, pipelines, workflows, CI/CD processes, and cloud infrastructure on Azure. Responsibilities include cluster and workspace management, automated testing, Infrastructure as Code, access controls, FinOps, Airflow orchestration, monitoring, incident response, and reusable Terraform and Databricks templates. The role also improves platform reliability, security, developer productivity, and production code promotion processes while partnering with engineering, architecture, and security teams.
The summary above was generated by AI

Role: Data Engineer
Location:
Remote
Employment Type: Full-Time
Experience: 4-7 years

 

KEY RESPONSIBILITIES

·       Design and implement CI/CD pipelines for data pipelines (ingestion jobs) and transformation projects (e.g., dbt on Databricks, SQL, notebooks)

·       Orchestrate Databricks jobs and workflows end-to-end (ingestion, transformation, quality checks)

·       Integrate automated testing into CI/CD, including schema and contract checks for data models and tables

·       Implement FinOps best practices to support cost monitoring and allocation across the EDP

·       Automate platform operations for Databricks and related services, such as workspace and cluster provisioning, library and runtime mgmt. and job deployment/config

·       Implement and maintain identity and access management for Databricks and supporting cloud resources, including workspace- and cluster-level permissions, table- and view-level access controls (e.g., Unity Catalog or equivalent), service principals, groups, and roles for automated workloads, and RBAC & TBAC models in collaboration with EDP Architect

·       Provide patterns, templates, and reusable modules (Terraform modules, Airflow DAG patterns, Databricks job templates) to accelerate onboarding of new projects

·       Continuously evaluate and improve tooling, pipelines, and platform architecture to increase reliability, security, and developer productivity on Databricks

·       Define the code promotion process to minimize impacts across domains as code is promoted to production

·       Manage end-to-end orchestration using managed Airflow. Contribute to defining and tracking SLA/SLO/SLIs for the platform and participate in incident response (triage, root cause analysis)

·       Practical knowledge of IT Infrastructure technologies, cloud computing Azure), cybersecurity, and disaster recovery.

·       Working knowledge of Azure ecosystem, including hands-on experience designing, building, and optimizing scalable data pipelines within cloud-native environments.

QUALIFICATIONS

·       Hands-on experience with Databricks in production environment, including workspace and cluster management, jobs/workflows and integrations with orchestration tools

·       Strong experience with CI/CD pipelines (e.g., GitHub Actions, GitLab CI, Azure DevOps, or similar) and Git-based workflows.

·       Strong experience with Infrastructure as Code (IaC) and orchestration tools for provisioning and managing Databricks and cloud infrastructure.

·       Experience implementing automated tests and quality gates in CI/CD pipelines.

·       Ability to partner effectively with data engineers, analytics engineers, architects, and security teams.

MINIMUM EXPERIENCE & EDUCATION

·       Bachelor’s degree in Computer Science, Information Technology, or related field preferred, or equivalent work experience.

·       4–7+ years in DevOps, Cloud Engineering, Site Reliability Engineering, or Platform Engineering, with at least 2+ years supporting data/analytics platforms.

·       Experience operating production data workloads, including monitoring, logging, performance tuning, and incident response.

·       Scripting skills (e.g., Python, Bash, PowerShell) for automation and integration.

·       Experience in CPG, retail, manufacturing, or distribution environments preferred.

 

 



Palo Alto Labs Pune, Mahārāshtra, IND Office

Pune, India, 411006

Similar Jobs

9 Days Ago
Hybrid
Pune, Maharashtra, IND
Senior level
Senior level
Fintech • Information Technology • Insurance • Financial Services • Big Data Analytics
Designs and deploys scalable Azure Databricks data pipelines using PySpark, Spark SQL, and Delta Lake across medallion layers. Builds HR data models, source-to-target mappings, reconciliation controls, governance, lineage, documentation, and data quality processes. Modernizes legacy Informatica and SQL integrations, including Oracle HCM feeds. Collaborates with distributed agile teams, mentors junior engineers, and supports production cutovers for sensitive employee data.
Top Skills: Ai/Copilot ToolingSparkAzure Data FactoryAzure Data Lake StorageAzure DevopsAzure SqlAzure SynapseCi/CdCrunchrDatabricksDelta LakeGitInformaticaOracle Hcm CloudPower BIPysparkSpark SqlSQLUnity Catalog
4 Days Ago
In-Office
Senior level
Senior level
Digital Media
Lead data engineering delivery on the Databricks Platform, building scalable batch, micro-batch, and streaming pipelines. Design lakehouse solutions using Delta Lake, Iceberg, Delta Live Tables, Unity Catalog, Spark, and cloud data platforms. Develop workflow orchestration, data quality, observability, CI/CD, and infrastructure automation solutions. Collaborate with architects and cross-functional teams, contribute to Databricks Center of Excellence initiatives, and deliver solution accelerators for clients.
Top Skills: Amazon EmrAmazon RedshiftApache AirflowApache IcebergSparkAuto LoaderCi/CdCollibraContainer Management SystemsDatabricksDatabricks Asset BundlesDbtDelta LakeDelta Live TablesGoogle BigqueryGreat ExpectationsJavaLakeflow Declarative PipelinesMlflowNosql DatabasesPrestoPrompt EngineeringPysparkPythonSnowflakeSQLStored ProceduresTerraformUnity CatalogYaml
13 Days Ago
Hybrid
Pune, Maharashtra, IND
Senior level
Senior level
Software • Business Intelligence
Build and operate Databricks data pipelines for Onit’s customer-facing benchmarking platform. Responsibilities include integrating and modeling multi-source data, resolving duplicate records, implementing data quality checks, orchestrating reliable workflows, optimizing Spark performance and costs, and applying governance, privacy, and multi-tenant isolation. The role collaborates with architects, data scientists, product, and engineering teams to productionize analytics and expose trustworthy curated data.
Top Skills: Auto LoaderAWSCi/CdClaudeDatabricksDatabricks AssistantDatabricks JobsDatabricks WorkflowsDelta LakeDelta Live TablesGitGithub CopilotPysparkPythonSpark SqlSQLStructured StreamingUnity Catalog

What you need to know about the Pune Tech Scene

Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account