Senior Data Engineer (Databricks)

Experience Level: 3 to 6 Years

Work location:  Mumbai, Bengaluru, Trivandrum

Role & Responsibilities:

• Design and implement data solutions (ETL pipelines, Processing and Transformation logic) using Snowflake as a key platform.

• Design Virtual warehouses which are optimized for performance and cost (auto scaling, idle time etc)

• Write SQL , stored procedures to implement business logic, other critical requirements like encryption, data masking etc

• Implement solutions  with features like Snowpipe, tasks, streams,dynamic tables etc

• Design the solutions by leveraging key concepts around the architecture of Snowflake, its data distribution and partitioning mechanism, data caching etc.

• Identify queries which take more time , look at the logs etc and provide solutions to optimize those queries

Must have skills:

• 3+ years’ experience of Hands-on in data structures, distributed systems, Spark, SQL, PySpark, NoSQL Databases 

• Strong software development skills in at least one of:  Python, Pyspark or Scala.

• Develop and maintain scalable & modular data pipelines using databricks & Apache Spark

• Experience building and supporting large-scale systems in a production environment    

• Experience working  on either of the cloud storages - AWS S3, Azure data lake, GCS buckets

• Exposure to the ELT tools offered by the cloud platforms like, ADF, AWS Glue, Google dataflow

• Integrate Databricks with other cloud services like AWS, Azure or Google Cloud

• Implement and optimize Spark jobs, data transformations, and data processing workflows in Databricks

• Managing and configuring databricks environments, including clusters and notebooks

• Experience working with different file formats - parquet, databricks delta lake

• Experience working with data governance on databricks, implementing unity catalog, delta share