May 2021 – Mar 2023 · 1 yr 10 mos · Mumbai, Maharashtra, India · On-site

• Data Migration Expertise:
• Spearheaded the migration initiative from a traditional Redshift warehouse system to a scalable and efficient data lake infrastructure.
• Successfully executed migration tasks, ensuring seamless transition and minimal disruption to ongoing operations.
• Adoption of Apache Iceberg:
• Implemented Apache Iceberg as the storage format for the data lake, enhancing data reliability, versioning, and schema evolution capabilities.
• Conducted a proof of concept (POC) on Trino DB to serve as a compute layer for Apache Iceberg, optimizing query performance and data retrieval.
• Big Data Optimization:
• Worked on optimizing Spark jobs on Amazon EMR clusters, improving processing speed and resource utilization.
• Implemented best practices to enhance the overall efficiency of the big data processing framework.
• Data Modeling:
• Applied expertise in data modeling to design efficient and scalable data structures, ensuring optimal data organization and storage.
• Real-time Data Platform:
• Played a key role in developing and maintaining a real-time data platform capable of handling an event concurrency of 200 million requests per minute and processing 24 billion records.
• Utilized Apache Flink to create a robust real-time alerting mechanism, enhancing the system's responsiveness to critical events.
• KPI Report Generation:
• Developed KPI report generation processes to provide actionable insights to stakeholders, enabling informed decision-making.
• Ensured the accuracy and timeliness of key performance indicator reports.

Sep 2018 – May 2021 · 2 yrs 8 mos · Noida, Uttar Pradesh, India

• 1. Pulse - Real Time Stateful Stream Processing & Analytics System (Fast Data)
• Flink on YARN - Realtime Stateful Stream Transformation
• DRUID - A TimeSeries database
• Ingestion time data cubes to achieve Sub-second query response
• ThetaSketches - count distinct values over incrementally rolled up data.
• Analytics Dashboards - forked Apache-Superset
• Eagle Eye (Health and performance monitoring) - Play framework, Scala, ELK Stack
• Batch Processing & Data Warehousing - Parquet, AWS S3, Hive, AWS Athena
• more details in Projects section
• 2. GA Facts & Data Pipeline
• Conceptualised & Created a complete SPARK, BigQuery & Python based framework to partially
• process data on Google BigQuery and partially in our Spark clusters to produce business useful
• insights.
• 'GA Facts' completely changed the way GA data was leveraged at PayTM as it solved multiple use-
• cases across different business verticals.
• Optimised batch data processing (1TB/Day Google Analytics clickstream Data) from a staggering
• 21hrs to 4.5hrs.
• 3. ElasticSearch Based Near Realtime Pipeline with Grafana.
• Optimisations in ES-Push (NRT dashboards) to reduce the size of stored data and reducing pipeline
• failures with better exception handling.

Oct 2017 – Sep 2018 · 11 mos · Gurugram, Haryana, India

• Responsible for the maintenance, improvement, cleaning, and manipulation of
• data from bank's business and analytics databases using PySpark.
• Created RESTful API ( in Django, DRF, Pandas, Numpy) for Campaign Management
• Tool.

Sep 2016 – Oct 2017 · 1 yr 1 mo · Gurugram, Haryana, India

• Developed an ERP solution for Snapdeal and EComm logistics from Scratch(Python, Django,
• Javascript, Jquery, Mysql, Google Chart).
• Created complex modules of reporting, challans generation, route planning of vehicles, pick up
• trip,shipment and Pallet handling.
• API Integration with various 3rd party systems for shipment manifest.
• Real time tracking of the shipment(Kafka, PySpark,MongoDB).