Sep 2023 – Apr 2025 · 1 yr 7 mos · Gurugram, Haryana, India · Hybrid

• I am leading the platform data engineering team, I am responsible for architecting a scalable 70TB+
• Hudi data lakehouse platform, achieving 30-seconds query response, enabling seamless data access,
• improve governance, and analytics across the organisation
• > PySpark, EMR, MSK, Debezium, Python, SQL, EC2, Apache Hudi, Docker, Prometheus,
• Lambda, Grafana, Athena and Airflow
• Engineered Change Data Capture (CDC) architecture with Debezium 2.7, Kafka connect clusters for
• real-time CDC ingestion, improving scalability and reducing operational overhead by 99%, and
• resulting in a remarkable 40% decrease in costs associated with MySQL queries
• Led Migration of Shipments multi data to BigQuery, saved cost of ~ 3000$ per month
• > MySQL, CDC, EC2, Debezium, Python, Kafka, BigQuery
• Led AWS cost optimization initiatives, saving the organization ~$40,000/month
• Created Athena workgroups and optimized queries to enable secure, cost-effective data access and
• analytics across multiple teams
• Design, coordinate, and execute pilots, prototypes and POCs (e.g, Flink, Iceberg, DataBricks, BigQuery,
• Snowflake, Apache Pinot, DMS, Redshift); document and share technical best practices
• Developed a centralized automatic alerting and monitoring system using Grafana, and Prometheus
• I am responsible for hiring and mentoring a strong engineering team across the organisation while
• setting technical direction across backend, cloud infrastructure, security, quality, and DevOps
• Design, Architecture, Code Reviews, Cloud Infra, Team Management, Hiring, Mentoring
• Built CI/CD pipelines, standardizing deployments and cutting code deployment time by 30%
• Maintain internal forks of Apache Hudi 0.14.1, fixed multiple Hudi open issues in lower version for
• production hot fixes
• Implemented, a highly scalable, and completely server-less architecture for CDC data ingestion for multi-cloud use cases

Oct 2021 – Sep 2023 · 1 yr 11 mos · Bengaluru, Karnataka, India · Hybrid

• Built Spark Streaming data pipeline to fetch real time pricing data in and generate real time analytical insights
• Migrated the existing application from hive to Spark 3.0, reduced the processing time to ~35%
• Developed the automated data quality framework, to ensure data quality across all data sources, improved the accuracy
• Developed 10+ Airflow DAGs for Pipeline automation, scheduling, and monitoring, reduced maintenance tasks by 60%
• Automated alerting system for job monitoring using Airflow, Shell Scripting, Python, Slack, it saved 1.5 hrs/day/Developers
• Deep dive into massive data sets to answer key business questions, creating data pipelines and spark jobs for analysis

Aug 2019 – Oct 2021 · 2 yrs 2 mos · Bangalore

• ▪ Project: Migration of our On-Premise GE Healthcare Services Enterprise Data Lake to a cloud-based solution on AWS.
• Building "One Data Platform" where TB's of data is sourced from multiple systems of various LOB's into Datalake built on the cloud(S3).
• Architected and implemented a reusable data quality framework for the Central DataLake on S3, enabling efficient and consistent data quality management.
• Engineered a versatile data compaction framework that effectively reduced data overhead, achieving an exceptional 70% decrease in data fat from incoming sources.
• Designed and developed an automation framework that streamlined routine development tasks, yielding a remarkable 4-hour reduction in daily effort per engineer.
• Provided mentorship and guidance to a team of 4+ members, empowering them to implement optimized solutions and successfully navigate technical challenges.
• Developed data pipelines from scratch; optimized data aggregation from 30+ independent sources and automated the ETL development process.
• Led the automation initiative for onboarding new data pipelines, resulting in ~ 80% reduction in manual efforts.
• Implemented an SNS notification system that includes comprehensive error information for Data Pipeline activities.

Jul 2018 – Aug 2019 · 1 yr 1 mo · Bengaluru Area, India · On-site

• ▪ Design ELT Pipelines for clinical visit reports and surveys.
• Build generic & optimized pipeline for highly critical & confidential clinical survey Data
• Developed HQL scripts to create external tables and analyze incoming and intermediate data for analytics applications in Hive.
• Used Partitioning, and Bucketing concepts in Hive and designed both Managed and External tables in Hive to optimize performance
• Optimized spark jobs using optimization techniques like broadcasting, executor tuning, persisting, etc.
• Written configurable automation scripts in Python to test the codes and migration.
• Documented and shared best practices for Spark optimization within the team and organization.

Jan 2016 – Jul 2018 · 2 yrs 6 mos · Bengaluru Area, India

• ▪ Tech Modernization for State Farm Insurance (Client)
• Worked on Hadoop migration project from Informatica and Oracle to Cloudera distribution.
• Developed pipelines to handle data of 1.5 TB/day from ingestion to reporting layer using Hadoop, Spark, and Shell script.
• Redesigned and refactored project architecture and Spark Scala ETL code, bringing down costs by 70%
• Performed performance tuning in Spark, SQL, and Sqoop, resulting in a 60% response time reduction
• Gained experience in stakeholder interaction, requirements gathering, data analysis, design document creation, solutions, performance tuning, and enhancements.

May 2015 – Jun 2016 · 1 yr 1 mo · Bengaluru Area, India

• Developed tables, views, and materialized views using SQL
• Automated the ETL processes for data ingestion and transformation using Spark, Scala, and Hive.
• Developed and maintained technical documentation from source to target load process.
• Involved in project estimation and planning.