Build and maintain scalable data pipelines on Azure Databricks using PySpark, SQL, and Delta Lake. Design dimensional and Gold-layer data models, ingest structured and semi-structured data, implement testing and monitoring, troubleshoot data-quality issues, and follow cataloging, lineage, security, privacy, and regulatory standards. Collaborate with data leads, analysts, data scientists, and AI engineers to deliver reliable datasets and participate in code reviews and platform standards discussions.
This is a remote position.
Shuru is an AI-native technology and engineering company that partners with businesses to build and scale high-impact digital products, platforms, and AI-driven solutions. We work with fast-growing organizations across industries, helping them solve complex technology challenges through strong engineering, product, data, and AI capabilities.
We are looking for a Senior Data Engineer to join Shuru and help design and build scalable, reliable, and high-performance data platforms for complex business environments.
1. Pipeline Development
- Build, test, and maintain data pipelines on Azure Databricks for the assigned business line, using PySpark and SQL.
- Implement Bronze, Silver, and Gold layer transformations using Delta Lake, following the platform’s established lakehouse standards.
- Ingest structured and semi-structured data from sources such as MariaDB, Azure Event Hubs, APIs, and file systems.
2. Data Modelling
- Design and build data models (dimensional models, Gold-layer tables) for the business line’s use cases, in line with patterns set by the Lead Data Engineer.
- Flag where a business line’s data model needs a shared, cross-line solution rather than a local one, and raise it to the Lead Data Engineer.
3. Quality & Reliability
- Implement automated testing, data validation, and pipeline monitoring for everything owned.
- Respond to and troubleshoot pipeline incidents and data quality issues for the assigned business line, escalating where needed.
- Follow the platform’s cataloguing, lineage, and metadata practices so the data produced is discoverable and trustworthy.
4. Collaboration & Delivery
- Work directly with the Forward Deployed Data Lead to turn scoped business requirements into build plans and delivery timelines.
- Partner with analysts and, where relevant, data scientists and AI engineers consuming the pipelines built, to make sure outputs meet their needs.
- Participate in code reviews and platform standards discussions led by the Lead Data Engineer.
5. Compliance
- Ensure pipelines and datasets meet data security, privacy, and regulatory requirements (e.g., GDPR and PII handling) relevant to the business line.
Requirements
- 5–8 years of experience in data engineering, including hands-on work with cloud data platforms.
- Experience building pipelines on Azure Databricks, Delta Lake, or comparable lakehouse/cloud-native tools.
- Exposure to version control (Git), Agile methodologies, and CI/CD practices.
- Experience in fintech, broking, or trading platforms is a plus, not required.
- Bachelor’s degree in Computer Science, Engineering, or a related field.
Benefits
- Work on challenging, large-scale engineering and data problems.
- Build modern platforms using cutting-edge cloud and data technologies.
- Work closely with experienced engineering, product, and technology leaders.
- Opportunity to take ownership of architecture and key technical decisions.
- Remote-first and collaborative work environment.
- Strong opportunities for learning, leadership, and career growth.
Similar Jobs
Information Technology
Build and maintain scalable data pipelines for a global fintech engagement using Azure Databricks, PySpark, SQL, and Delta Lake. Develop Bronze, Silver, and Gold data layers, optimize data models, implement validation and monitoring, troubleshoot data quality issues, and follow governance, security, cataloging, and lineage standards while collaborating with data and engineering teams.
Top Skills:
AzureAzure DatabricksCi/CdDelta LakeGitPysparkSQL
AdTech • Digital Media • Marketing Tech • Mobile
Designs and develops scalable data architectures, cloud data solutions, data warehouses, business analytics tools, and complex data pipelines. The role involves data wrangling, exploration, and discovery across heterogeneous sources, using AWS, SQL, Python, and relational database technologies. The engineer also contributes to data engineering innovation, knowledge sharing, and project planning while enabling data-driven organizational decision-making.
Top Skills:
Amazon Web Services (Aws)Data WarehousingDaxPythonRelational DatabasesSQL
Information Technology • Consulting
Develop and maintain Oracle APEX applications, Oracle Analytics Server/OBIEE reporting solutions, BI Publisher reports, and Oracle database objects. Build dashboards, workflows, integrations, REST APIs, and data models; write complex SQL and PL/SQL; troubleshoot and tune application and database performance; support deployments, testing, production issues, documentation, and stakeholder collaboration.
Top Skills:
Agile/ScrumBi PublisherCi/CdCSSHTMLJavaScriptLinux/UnixObieeOracle Analytics ServerOracle ApexOracle Autonomous DatabaseOracle Cloud InfrastructureOracle Data IntegratorOracle DatabaseOracle Rest Data ServicesPl/SqlRest ApisSource ControlSQL
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
