Design, develop, and maintain scalable AWS-based data pipelines and infrastructure. Build and optimize data warehouses and distributed query engines, implement ETL/ELT with tools like NiFi/Airflow/Airbyte and PySpark/Flint, ensure data quality, performance tuning, BI dashboards, troubleshooting, documentation, and mentoring junior engineers.
Job Description
Role Summary:
As a Senior Data Engineer, you will design, develop, and maintain robust and scalable data pipelines and infrastructure on AWS Ubuntu. You will leverage your expertise in Python-based frameworks and a wide range of data platforms to build and optimize data warehouses, distributed query engines, ETL processes, and business intelligence solutions. You will be responsible for ensuring data quality, performance, and reliability while collaborating with cross-functional teams to deliver impactful data products.
Responsibilities:
- Data Pipeline Development: Design, develop, and implement efficient and scalable ETL/ELT pipelines using Apache NiFi, Airflow, and Airbyte.Efficient data processing via Pyspark,Apache Flink
- Data Warehousing: Build and maintain data warehouses using ClickHouse, StarRocks, and Apache Doris, ensuring data integrity and performance.
- Distributed Query Engines: Deploy, configure, and optimize distributed query engines like Trino, Drill, and Dremio for efficient data access and analysis.
- Business Intelligence: Develop and maintain dashboards and visualizations using Superset, Grafana, Metabase, and Redash.
- AWS Infrastructure Management: Deploy, configure, and manage data infrastructure on AWS Ubuntu, including EC2, S3, VPC, IAM, and CloudWatch.
- Performance Optimization: Tune and optimize data platforms and pipelines for high performance and scalability.
- Data Quality and Governance: Implement data quality checks and monitoring to ensure data accuracy and reliability.
- Troubleshooting and Support: Diagnose and resolve complex data infrastructure issues.
- Collaboration: Work closely with data scientists, analysts, and software engineers to deliver data-driven solutions.
- Documentation: Create and maintain comprehensive documentation for data pipelines and infrastructure.
- Mentorship: Mentor junior data engineers and share best practices.
- Stay Updated: Continuously research and evaluate new technologies and tools.
Requirements
Required Skills and Experience:
- 4-6 years of experience in data engineering roles.
- Strong proficiency in Python and related data manipulation libraries (Pandas, NumPy, Dask).
- Expertise in SQL and database technologies.
- Extensive experience with AWS Ubuntu infrastructure (EC2, S3, VPC, IAM, CloudWatch).
- Deep understanding and hands-on experience with any one in each category:
- Data Warehousing: ClickHouse, StarRocks, Apache Doris.
- Data Processing: Pyspark,Apache Flink.
- Distributed Query Engines: Trino, Drill, Dremio.
- ETL/Data Orchestration: Apache NiFi, Airflow, Airbyte.
- Business Intelligence: Superset, Grafana, Metabase, Redash.
- Experience with containerization (Docker) and orchestration (Kubernetes) is a plus.
- Strong understanding of data modelling, data warehousing, and data pipeline design principles.
- Excellent troubleshooting and problem-solving skills.
- Strong communication and collaboration skills.
- Ability to work independently and as part of a team.
Preferred Qualifications:
- AWS certifications (e.g., AWS Certified Data Analytics - Specialty).
- Experience with cloud-native data platforms.
- Contributions to open-source projects.
- Experience with data governance and security best practices.
- Bachelor's or Master's degree in Computer Science, Engineering, or a related field
About Company / Benefits
SarvaGram leads India's fintech revolution, focusing on rural households. Our mission: empower rural communities via credit-led household-centric solutions blending high-tech innovation with a personal touch. We catalyze change, bridging financial gaps, and boosting agriculture productivity.
—-----------------------------------------------------—-----------------------------------------------------—------------
🔍 Looking for expert help in landing your next job?
Explore Nexthire Talent Assist – our premium career support offering to help you get interview calls, negotiate better hikes, improve your resume, and connect with 100,000+ hiring managers & founders.
👉 Check it out here: https://www.nexthireconsulting.com/
Similar Jobs
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Designs, builds, and operates enterprise AI platforms across public and private clouds. Develops MLOps, model deployment, monitoring, observability, governance, automation, and infrastructure capabilities supporting machine learning and generative AI workloads. Responsibilities include troubleshooting production systems, improving reliability and scalability, collaborating with engineering, security, and governance teams, evaluating emerging AI technologies, contributing to architecture and platform strategy, and mentoring peers.
Top Skills:
Ai GatewaysCi/CdDockerDrift DetectionGenerative AiGitopsGpu ComputingHelmInfrastructure As CodeKubernetesLarge Language ModelsMlopsObservabilityOpenshiftPrivate CloudPublic CloudPythonTelemetryVector Databases
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Designs, builds, and operates Mastercard’s enterprise Developer Workbench platform. Responsibilities include integrating developer tools, managing tooling lifecycles, creating standardized onboarding environments, advancing AI-assisted development capabilities, implementing CI/CD and cloud-native solutions, measuring developer productivity, and ensuring security and regulatory compliance. The role collaborates across product, platform, security, compliance, and engineering teams while contributing to technical strategy and engineering standards.
Top Skills:
ArgocdAWSAzureBackstageClaude CodeDevcontainersDocker DesktopGCPGithub ActionsGithub CopilotJenkinsKubernetes
Blockchain • Fintech • Payments • Consulting • Cryptocurrency • Cybersecurity • Quantum Computing
Designs, implements, secures, and supports multi-site LAN/WAN networks. Manages network projects, upgrades, troubleshooting, vendor engagement, change processes, baseline hardware checks, and mentors junior staff while ensuring PCI/DSS compliance and 24/7 network support.
Top Skills:
BgpCcnpCiscoEncryptionFirewallsLanLoad BalancingMplsNetwork DiagnosticsPci DssSnmpSshTcp/IpVpnWan
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
