Develop, optimize, and maintain scalable ETL/ELT data pipelines for structured and unstructured data using Python and PySpark. Ensure data quality through validation, cleansing, transformation, and reconciliation. Collaborate with technical and business stakeholders, troubleshoot performance issues, conduct code reviews, and implement coding standards and CI/CD pipelines. Work with big data frameworks, cloud platforms, databases, data models, schemas, and warehousing solutions.
Description: Responsibilities: • Development and Maintain Data Pipelines: Design, implement, and optimize end-to-end ETL/ELT pipelines for ingesting, processing, and transforming large volumes of structured and unstructured data. • Utilize Python and Pyspark: Write efficient, scalable and maintainable code in Python and leverage Pyspark for large-scale data processing in distributed computing environments. Also be able to review existing code and identify areas of improvement. • Ensure Data Quality and Integrity: Implement data validation, cleansing, transformation and reconciliation processes to ensure data accuracy and consistency throughout the data lifecycle. • Collaborate with Stakeholders: Work closely with IT teams and business stakeholders to gather data requirements and translate them to technical solutions. • Troubleshoot and Optimize: Monitor job performance, troubleshoot complex data issues and fine-tune for performance and scalability. • Adhere to Best Practices: Participate in code reviews, establish coding standards, and implement CI/CD pipelines for automated testing and deployment. Skills: • Strong hands-on coding proficiency in Python, Pyspark and SQL (Microsoft SQL Server preferred) • Experience with big data frameworks (Hadoop, Spark). • Experience with cloud platforms ( AWS, Azure or GCP) • Experience with Code versioning tools ( Bitbucket, Github ) • Experience with CI/CD and setting up pipelines. • Solid understanding of database design principles, data modelling, schemas and data warehousing solutions. • Excellent problem-solving and analytical skills to troubleshoot complex data issues independently.
Hexaware Technologies Pune, Mahārāshtra, IND Office
North Block, Plot No. 19, Rajiv Gandhi InfoTech Park, MIDC - SEZ, Phase 3, Hinjawadi, Pune, Maharastra, India, 411057
Similar Jobs
Information Technology • Consulting
Designs and maintains enterprise data architecture, including conceptual, logical, and physical data models. Establishes data standards, metadata conventions, governance policies, security frameworks, and data dictionaries. Leads Master Data Management initiatives to improve consistency and integrity across platforms. Collaborates with business, analytics, application, and engineering teams, while supporting database, data warehouse, integration, and ETL pipeline efforts aligned with business, regulatory, security, and performance requirements.
Top Skills:
Data ArchitectureData GovernanceData ModelingData WarehousingDatabase ManagementETLMaster Data Management (Mdm)Metadata Management
Cloud • eCommerce • HR Tech • Information Technology • Analytics
Manage and support AKS-based enterprise data platform infrastructure, administer Apache NiFi, Azure Event Hub and Functions, implement CI/CD with Azure DevOps, automate infrastructure with Terraform/Bicep/ARM, monitor with Azure Monitor and Log Analytics, troubleshoot production incidents, ensure security, capacity planning, and collaborate with data engineering teams.
Top Skills:
Apache NifiArgocdArm TemplatesAzure Application GatewayAzure Application InsightsAzure Container RegistryAzure Data FactoryAzure DevopsAzure Event HubAzure FirewallAzure FunctionsAzure Kubernetes Service (Aks)Azure MonitorAzure Synapse AnalyticsBashBicepCluster AutoscalerDockerFluxGitGrafanaHpaIstioKqlKubernetesLog AnalyticsMicrosoft Defender For CloudNetwork Security GroupNginxNifi RegistryOpentelemetryPeeringPowershellPrivate EndpointsPrometheusPythonRbacTerraformVirtual NetworkVpaVpnYaml Pipelines
Software
Design, implement, and lead large-scale generative AI and ML systems (LLM fine-tuning, RAG, agents, Graph ML). Build REST APIs, deploy with Docker/Kubernetes, optimize big-data workflows, mentor engineers, and operate ML services on cloud platforms.
Top Skills:
Anomaly DetectionAWSAzureBig DataCi/CdData MiningDockerDocument DatabasesFastapiGenerative AiGCPGraph MlKnowledge BasesKubernetesLlmsNoSQLPythonPyTorchRagRestful ApisSagemakerTensorrtTriton
What you need to know about the Pune Tech Scene
Once a far-out concept, AI is now a tangible force reshaping industries and economies worldwide. While its adoption will automate some roles, AI has created more jobs than it has displaced, with an expected 97 million new roles to be created in the coming years. This is especially true in cities like Pune, which is emerging as a hub for companies eager to leverage this technology to develop solutions that simplify and improve lives in sectors such as education, healthcare, finance, e-commerce and more.
