Design and implement Databricks Lakehouse architectures and Medallion pipelines; build and optimize scalable ETL/ELT using PySpark/Spark SQL; integrate multi-source data (RDBMS/NoSQL/cloud); manage Databricks workspaces, governance, CI/CD, and platform integrations across Azure and AWS to enable analytics and high-performance data solutions.
Job Description: Databricks Developer / Data Architect
Position Title
Databricks Developer / Data Architect
Location
Hybrid/ Remote
Employment Type
Full-Time / Contract
Job Summary
We are seeking an experienced Databricks Developer / Data Architect to design, implement, and optimize modern data platforms and ETL pipelines using Databricks and cloud-native technologies. The ideal candidate will have strong expertise in data architecture, Lakehouse implementation, Medallion Architecture, and scalable ETL development across Azure and AWS environments.
The role involves setting up Databricks workspaces, configuring data integrations and Lakehouse Federation, building enterprise-grade ETL workflows, and enabling high-performance analytics solutions.
Key Responsibilities
Data Architecture & Modeling
- Design and implement scalable enterprise data architectures using Databricks Lakehouse platform
- Configure and manage Databricks workspaces, clusters, access controls, and governance
- Implement Medallion Architecture (Bronze, Silver, Gold layers) for data processing and analytics
- Set up and manage Lakehouse Federation and data connectors for multi-source integration
- Develop logical and physical data models for structured and semi-structured datasets
- Ensure data quality, security, scalability, and performance optimization
ETL Development
- Develop and maintain scalable ETL/ELT pipelines using PySpark, Spark SQL, and Databricks workflows
- Build reusable data ingestion frameworks for batch and streaming workloads
- Optimize Spark jobs for performance, cost efficiency, and reliability
- Integrate data from relational and NoSQL databases, cloud platforms, and external systems
- Automate deployment and monitoring of ETL workflows
Cloud & Platform Engineering
- Work with Azure and AWS cloud services to deploy and manage data solutions
- Configure integrations with Snowflake, Postgres, MongoDB, DynamoDB, Cloudera, and Domino Server
- Support CI/CD, infrastructure automation, and environment management
- Collaborate with cross-functional teams including Data Scientists, Analysts, and Business stakeholders
Required Skills & Qualifications
- 5+ years of experience in Data Engineering / Data Architecture
- Strong hands-on experience with Databricks platform
- Expertise in:
- Python
- Apache Spark
- PySpark
- SQL
- Strong understanding of:
- Lakehouse Architecture
- Medallion Architecture
- Data Modeling
- ETL/ELT Design Patterns
- Experience with cloud platforms:
- Microsoft Azure
- AWS
- Experience integrating with:
- PostgreSQL
- DynamoDB
- MongoDB
- Snowflake
- Cloudera
- Domino Server
- Knowledge of performance tuning and optimization in Spark/Databricks
- Experience with version control and DevOps practices
Preferred Qualifications
- Databricks Certification(s)
- Experience with Delta Lake and Unity Catalog
- Familiarity with streaming frameworks and real-time data processing
- Knowledge of data governance and security best practices
- Experience in Agile/Scrum delivery models
Technical Stack
- Databricks
- Python
- Spark / PySpark
- SQL
- Azure
- AWS
- Snowflake
- PostgreSQL
- MongoDB
- DynamoDB
- Cloudera
- Domino Server
Soft Skills
- Strong analytical and problem-solving abilities
- Excellent communication and stakeholder management skills
- Ability to work independently and collaboratively in fast-paced environments
- Strong documentation and solution design capabilities
Nice-to-Have
- Experience with Terraform or Infrastructure as Code
- Exposure to ML/data science platforms
- Experience with orchestration tools such as Airflow or Azure Data Factory
Job Description: Databricks Developer / Data Architect
Position Title
Databricks Developer / Data Architect
Location
Hybrid/ Remote
Employment Type
Full-Time / Contract
Job Summary
We are seeking an experienced Databricks Developer / Data Architect to design, implement, and optimize modern data platforms and ETL pipelines using Databricks and cloud-native technologies. The ideal candidate will have strong expertise in data architecture, Lakehouse implementation, Medallion Architecture, and scalable ETL development across Azure and AWS environments.
The role involves setting up Databricks workspaces, configuring data integrations and Lakehouse Federation, building enterprise-grade ETL workflows, and enabling high-performance analytics solutions.
Key Responsibilities
Data Architecture & Modeling
- Design and implement scalable enterprise data architectures using Databricks Lakehouse platform
- Configure and manage Databricks workspaces, clusters, access controls, and governance
- Implement Medallion Architecture (Bronze, Silver, Gold layers) for data processing and analytics
- Set up and manage Lakehouse Federation and data connectors for multi-source integration
- Develop logical and physical data models for structured and semi-structured datasets
- Ensure data quality, security, scalability, and performance optimization
ETL Development
- Develop and maintain scalable ETL/ELT pipelines using PySpark, Spark SQL, and Databricks workflows
- Build reusable data ingestion frameworks for batch and streaming workloads
- Optimize Spark jobs for performance, cost efficiency, and reliability
- Integrate data from relational and NoSQL databases, cloud platforms, and external systems
- Automate deployment and monitoring of ETL workflows
Cloud & Platform Engineering
- Work with Azure and AWS cloud services to deploy and manage data solutions
- Configure integrations with Snowflake, Postgres, MongoDB, DynamoDB, Cloudera, and Domino Server
- Support CI/CD, infrastructure automation, and environment management
- Collaborate with cross-functional teams including Data Scientists, Analysts, and Business stakeholders
Required Skills & Qualifications
- 5+ years of experience in Data Engineering / Data Architecture
- Strong hands-on experience with Databricks platform
- Expertise in:
- Python
- Apache Spark
- PySpark
- SQL
- Strong understanding of:
- Lakehouse Architecture
- Medallion Architecture
- Data Modeling
- ETL/ELT Design Patterns
- Experience with cloud platforms:
- Microsoft Azure
- AWS
- Experience integrating with:
- PostgreSQL
- DynamoDB
- MongoDB
- Snowflake
- Cloudera
- Domino Server
- Knowledge of performance tuning and optimization in Spark/Databricks
- Experience with version control and DevOps practices
Preferred Qualifications
- Databricks Certification(s)
- Experience with Delta Lake and Unity Catalog
- Familiarity with streaming frameworks and real-time data processing
- Knowledge of data governance and security best practices
- Experience in Agile/Scrum delivery models
Technical Stack
- Databricks
- Python
- Spark / PySpark
- SQL
- Azure
- AWS
- Snowflake
- PostgreSQL
- MongoDB
- DynamoDB
- Cloudera
- Domino Server
Soft Skills
- Strong analytical and problem-solving abilities
- Excellent communication and stakeholder management skills
- Ability to work independently and collaboratively in fast-paced environments
- Strong documentation and solution design capabilities
Nice-to-Have
- Experience with Terraform or Infrastructure as Code
- Exposure to ML/data science platforms
- Experience with orchestration tools such as Airflow or Azure Data Factory
Preferred Qualifications
- Databricks Certification(s)
- Experience with Delta Lake and Unity Catalog
- Familiarity with streaming frameworks and real-time data processing
- Knowledge of data governance and security best practices
- Experience in Agile/Scrum delivery models
Technical Stack
- Databricks
- Python
- Spark / PySpark
- SQL
- Azure
- AWS
- Snowflake
- PostgreSQL
- MongoDB
- DynamoDB
- Cloudera
- Domino Server
Soft Skills
- Strong analytical and problem-solving abilities
- Excellent communication and stakeholder management skills
- Ability to work independently and collaboratively in fast-paced environments
- Strong documentation and solution design capabilities
Nice-to-Have
- Experience with Terraform or Infrastructure as Code
- Exposure to ML/data science platforms
- Experience with orchestration tools such as Airflow or Azure Data Factory
EXL Pune, Mahārāshtra, IND Office
Pune, India
Similar Jobs
Big Data • Fintech • Information Technology • Business Intelligence • Financial Services • Cybersecurity • Big Data Analytics
Processes and transforms transactional and batch data requests for internal and external customers. Reviews technical requirements, configures scripts, creates and executes test plans, performs quality checks, audits results, and delivers accurate files and reports within service-level agreements. Collaborates with client engagement, production, and batch teams while supporting modernization initiatives.
Top Skills:
Apache HiveSparkETLLinuxMS OfficePysparkPythonSQLUnix
Big Data • Fintech • Information Technology • Business Intelligence • Financial Services • Cybersecurity • Big Data Analytics
Lead credit risk analytics and modeling engagements, develop and validate predictive models and scorecards, perform portfolio and segmentation analysis, and deliver data-driven recommendations to financial-services clients. Manage workstreams, collaborate with stakeholders, support proposals, and contribute to analytics capability development and automation initiatives.
Top Skills:
PythonRSQL
Big Data • Fintech • Information Technology • Business Intelligence • Financial Services • Cybersecurity • Big Data Analytics
Designs and builds production-ready AI systems using classical AI, search, planning, ranking, retrieval, knowledge representation, and agentic automation. Develops scalable APIs, microservices, data pipelines, and orchestration layers; integrates AI with enterprise applications, databases, and workflow platforms. Evaluates architectures for reliability and business impact while applying software engineering, distributed systems, cloud, observability, and security practices. Collaborates with stakeholders and contributes to design and code reviews.
Top Skills:
AutogenAWSAzureCi/CdDockerGCPGitGraph DatabasesJavaKafkaKubernetesLangchainLanggraphLlamaindexMicroservicesNeo4JNlpNosql DatabasesOntopPythonRest ApisSemantic KernelSQLVector Databases
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
