Sep 2022 – Jan 2025 · 2 yrs 4 mos · Gurugram, Haryana, India · Remote

• Leading the Data charter at Hotstar. Managing two teams- Data Platform and Analytics Engineering. Solving challenging data intensive problems at a scale of 60 million concurrent viewership. Curated and executed multi-quarter roadmaps across teams to create a modern Data Platform and served multiple Data Products on top of these platform abstractions. Continuously worked with other Engineering, Product, Analytics and Customer Support as a horizontal team leader to make Data@Hotstar successful.
• Led the efforts to scale Hotstar's Data Platform to 4x - to handle multi million RPS
• throughput and processing hundreds of billions of clickstream and async messages per day. Platform scaled for world record traffic of 60 million concurrent live stream viewers for Cricket World Cup'23.
• Led the migration of self managed Hive system to Databricks-based Spark, improving execution speed by >5x and reducing cost by ~150K $ every year.

Mar 2022 – Sep 2022 · 6 mos · Delhi, India · Remote

• Leading engineering efforts for the Spark Execution team of 4 engineers - end
• to end ownership for managing lifecycle and execution of distributed spark
• code on cloud for a Low-Code Data Engineering Platform.
• Designed and implemented a solution to compute IO metrics and data preview
• and column statistics for Spark jobs. This allowed customers to track data
• changes and optimize their jobs by seeing historical metrics of their jobs
• without affecting job execution times.
• Designed and implemented a new Databricks based Job Scheduler to schedule
• Spark Jobs on Databricks and manage the job lifecycle from a single pane of UI.

Sep 2021 – Mar 2022 · 6 mos · Bengaluru, Karnataka, India

• Managing 3 engineers. Leading the end to end integration of Domino Data Lab MLOps and Model Development platforms to capture model predictions in real-time and scaling it to Terabyte scale.

Dec 2020 – Sep 2021 · 9 mos · Bengaluru, Karnataka, India

• MLOps Team - creating an industry leading Model Monitoring system to enable enterprises to handle ML operations in production at scale.
• Scaled Domino Model-Monitor's compute engine ~1000x, to support computing Model Quality metrics for machine learning models with datasets upto a few Terabytes in size, which earlier supported only a few Gigabytes. This involved using Spark for distributed compute, producing hourly aggregate metrics and supporting queries on top of these aggregate metrics.
• Led the project to improve the Security posture of the enterprise Domino Model-Monitor product. Reduced the security vulnerabilities to almost 10% of the original number.

Mar 2020 – Dec 2020 · 9 mos

• Led the initiative to revamp the end to end Data pipeline setup and search experience for Sumo Logic's log analytics platform. This involved running a team of 6 engineers (backend, frontend and QE), owning the technical design and project delivery, and deep diving into product requirements with multiple brainstorming sessions with Product and UX teams to come up with an intuitive and customer-friendly product design backed by a scalable and modular technical design.
• Re-designed Sumo Logic's in-house Map Reduce framework architecture to support large scale (billions of groups) aggregate queries on top of unstructured log data.

Mar 2017 – Feb 2020 · 2 yrs 11 mos

• Technical Lead for a team of 4 engineers, designing and developing a new lookup database service to handle multi-million read/write operations every second. I also led the planning and end-to-end delivery of the entire project.
• Team/Technical Lead for end-to-end support for column based data storage format in ingest and search pipeline to build the new on-demand pricing model for log analytics.
• Technically mentored and trained new engineers in my team.
• Know-how of running a production system: managed multiple services and fires in production.
• Member of the Change Management Board to control production changes.
• Evangelist for adopting best scrum and agile development practices in my team.
• Contributed to defining the technical vision of the log analytics product.

Jun 2015 – Feb 2017 · 1 yr 8 mos

• Developed a new highly available Interval Tree Database microservice to improve scalability and reliability. Upgrading the new service now took few seconds, as compared to a few hours before.
• Developed an offline microservice to re-process customer data. Improved performance of search queries by as much as 8x, and reduced lag from few days to an hour.

Jan 2014 – Mar 2014 · 2 mos

• Creating programming questions for recruitment. Formalizing everything, from problem statement to test cases.