Design, build, and maintain data pipelines using Databricks and Apache Spark, ensuring data quality while collaborating with data teams.
Job Title: Data Engineer – Databricks
Company: V4C.ai
Type: Full-time
About V4C.ai
V4C.ai is a premier IT services consultancy and a proud partner of Databricks, the Data Intelligence Platform, driving strategic business transformation.
We partner with organizations to accelerate their journey towards AI-driven success by offering a comprehensive suite of Dataiku and generative AI services. Our expertise in implementation, optimization, and enablement empowers clients to harness the full potential of their data, unlocking significant competitive advantages and fostering innovation..
Key Responsibilities
- Design, build, and maintain scalable data pipelines using Databricks and Apache Spark
- Integrate data from various sources into data lakes or data warehouses
- Implement and manage Delta Lake architecture for reliable, versioned data storage
- Ensure data quality, performance, and reliability through testing and monitoring
- Collaborate with data analysts, scientists, and stakeholders to meet data needs
- Automate workflows and manage job scheduling within Databricks
- Maintain clear and thorough documentation of data workflows and architecture
- Work on Databricks-based AI/ML solutions, including machine learning pipelines, in collaboration with data science teams
Requirements
- Experience: 3+ years in data engineering with strong exposure to Databricks, AI/ML, and big data tools
- Technical Skills:
- Proficient in Python or Scala for ETL development
- Strong understanding of Apache Spark, Delta Lake, and Databricks SQL
- Familiar with REST APIs, including Databricks REST API
- Cloud Platforms: Experience with AWS, Azure, or GCP
- Data Modeling: Familiarity with data lakehouse concepts and dimensional modeling
- Version Control & CI/CD: Comfortable using Git and pipeline automation tools
- Soft Skills: Strong problem-solving abilities, attention to detail, and teamwork
Nice to Have
- Certifications: Databricks Certified Data Engineer Associate/Professional
- Workflow Tools: Experience with Airflow or Databricks Workflows
- Monitoring: Familiarity with Datadog, Prometheus, or similar tools
- ML Pipelines: Experience with MLflow or integration of machine learning models into production pipelines
Top Skills
Airflow
Spark
AWS
Azure
Databricks
Databricks Sql
Datadog
Delta Lake
GCP
Git
Mlflow
Prometheus
Python
Rest Apis
Scala
Similar Jobs
Big Data • Cloud • Fintech • Financial Services • Conversational AI
The role involves collaborating on software systems for BlackRock's Aladdin, focusing on high availability applications, and requires strong Java skills and problem-solving abilities.
Top Skills:
AWSDockerGitGCPHibernateJavaKafkaKubernetesMavenAzureNoSQLSpring BootSpring FrameworkSQL
Big Data • Cloud • Fintech • Financial Services • Conversational AI
As a Backend Developer at BlackRock, you'll build high-performance financial software, collaborate globally, and contribute to a multi-asset trading platform. This role emphasizes innovation, collaboration, and leadership within a fast-paced environment.
Top Skills:
AnsibleAWSAzureC++CassandraCloud-Native Design PatternsDevOpsDockerGCPGrpcIgniteJavaKafkaMongodbMs SqlNoSQLPerlPythonRedisRestSQLSybaseTerraform
Big Data • Cloud • Fintech • Financial Services • Conversational AI
As a Software Engineer at BlackRock, you will design, develop, and maintain high-performance java applications for Aladdin's trading platform, collaborating in a global team environment.
Top Skills:
Apache CassandraAzure DevopsHibernateIgniteJavaJunitKafkaMavenMongodbRedisSpring
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
