Jul 2022 – Apr 2024 · 1 yr 9 mos

• Part of Data platform leadership group.
• Team handled scale of 60M concurrent users during the World Cup which translates to:
• 1. Hundreds of millions of request per minute on Ingestion API
• 2.Billions of messages for a match on open source Kafka
• 3. Terrabytes of compressed data (Parquet) in Data lake
• Team delivered:
• Moving from JSON to Proto right from ingestion layer to data lake changes across countries
• Isolation of Kafka resources based on priorities, managed open source Kafka with 25 nodes only at such a high scale, we took care of resource management, rebalancing of data on Kafka clusters
• Degradation levers built to route traffic to different Kafka clusters, drop events in case of panic, write to file for specific events if need be in case of Panic.
• Fixing Kafka connector issues to deliver events to third party analytics system in real time.
• Cost improvement
• No SLA breach for any of the critical reports after the match

Mar 2021 – Jun 2022 · 1 yr 3 mos

• Migrated over 1000 plus tables from HBase to Hudi which enabled to query data in near real time.
• Reduced SLA from hours to minutes for data availability
• Creating data products and landing them in multiple geo's like SEA, MENA,UK, CA,US

May 2019 – Mar 2021 · 1 yr 10 mos

• Building the Data Platform at Hotstar

Sep 2014 – Apr 2019 · 4 yrs 7 mos · Pune/Pimpri-Chinchwad Area · On-site

• + Worked closely with Trade exchange to handle more than 1 billions trades per day
• + Designed and developed Real time pipelines using Flink which handles
• stream to stream joins
• Enrichment with ES
• Out of order
• Late events
• Side outputs
• + Involved in making sure that the data is securely stored in data lake by using SSE, having the right set of policies on the table using Apache Ranger.
• + Involved closely in performance tuning of pipelines bringing down the job runs from hours to minutes.
• + Developed batch pipelines for daily trade reporting where we evaluated Apache Beam with Flink and Spark runners. These pipelines were dealing with terabytes data every day.
• + Involved in setting up the infra as well where we setup Flink, ZK on premise.
• + Worked on downstream Data Lake processing by strategizing Data formats, compression, partitioning, and scheduling of Hive ETL jobs.
• Tech Stack : Apache Flink, Kafka, Spark, Zookeeper, Cassandra, ElasticSearch, HIVE, AWS, Ranger

Sep 2013 – Sep 2014 · 1 yr · Pune/Pimpri-Chinchwad Area

• Worked in Storage and Virtualisation domain
• Developed interface between VMware SRM and replicated storage arrays.
• Worked on backend development for vCenter plugin,