Sep 2021 – Present · 4 yrs 8 mos

Jun 2018 – Sep 2021 · 3 yrs 3 mos · Palo Alto, CA

• Architect for Streaming Infra
• Kafka
• Kafka is the data backbone of Uber, it transfers 4+ Tri messages and 2+ PB of data per day for thousands of services and use cases across the company.
• Kafka availability is exceptionally critical and directly impacts business.
• When I joined Uber, Kafka service reliability is pretty low. Identified vital aspects that we can improve operationally and audited configs across 30+ clusters, and fixed several issues that were causing L4/L5 outages.
• Drove best practices for zookeeper and Kafka, upgrades through automation, critical metrics to monitor, and reduced on-call alert noise. Increased service reliability to 4 9's and with zero L4/L5 outages during 2019.
• Published vision for streaming infra, drove several vital projects such as
• Kafka Tiered Storage (https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage),
• Kafka Security at Uber using SPIRE (https://spiffe.io/spire/)
• Containerization of Kafka for better management and roll-out of upgrades
• Kafka Metadata Proxy, a central proxy to handle metadata requests for a topic across multiple clusters
• Kafka hardware identification, on-prem and cloud
• uWorc (Unified Workflow Orchestration Platform)
• At Uber, data scientists use our data infra to drive insights to better user experience and business growth.
• Often it requires them to build data pipelines, ETL, queries, etc.. to analyze, process, transform the data. This involves engineering chops, or they need to work data engineering to build a pipeline for them, and any changes to this pipeline needs to go through the loop of data engineering.
• With uWorc, we wanted to build a self-serviceable, no-code, drag-drop UI to build data pipelines, be it batch or streaming or ML-based.
• I proposed to use Streamline as the foundation for uWorc and built a POC. Demo'ed to Eng LT, partner teams to drive the consensus to build the uWorc
• uWorc became the default workflow editor that users love at Uber.

Sep 2015 – Present · 10 yrs 8 mos

Aug 2014 – Present · 11 yrs 9 mos

Aug 2014 – Present · 11 yrs 9 mos

Apr 2014 – Jun 2018 · 4 yrs 2 mos

• Introduced Apache Kafka as part of the Streaming Platform, Hortonworks is the first company to do so. Contributed several features to Storm & Kafka, Kafka security designed and developed by myself.
• Apart from contributing in the engineering role, took the responsibilities of Manager and grown the team to 14 engineers. Hired and mentored high-caliber engineers. Lead the team in delivering several releases and set the technical direction for the team.
• Lead the architecture, engineering team and UI to build Schema Registry (http://github.com/hortonworks/registry) and SAM (http://github.com/hortonworks/streamline) real-time streaming analytics platform.
• Both of these products are extremely well received.
• 1. http://www.crn.com/slide-shows/applications-os/300089006/the-10-coolest-big-data-products-of-2017-so-far.htm/pgno/0/6?utm_content=buffer7f3a1&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
• 2. https://www.datanami.com/2017/06/14/hortonworks-shifts-focus-streaming-analytics/

Jul 2012 – Apr 2014 · 1 yr 9 mos · Mountain View,CA

• Lead the team at Mozilla to build the Metrics collection from firefox (downloads, telemetry, perf) installation across the world. Built using Apache Kafka, Storm to consume and process up to TBs of data per day.https://github.com/mozilla-metrics/

Jan 2010 – Jan 2010 · 0 mo

Jan 2008 – Jul 2012 · 4 yrs 6 mos

• Worked at Yahoo! Vertical Search Engine team. Scaling vertical search engines for Yahoo Product Search.
• Designed and developed new GEO RESTFUL API’s. Several teams inside Yahoo uses this API and also part of Yahoo! Developer network. Implemented Route thinning algorithm for drawing directions routes efficiently.
• Used memcached as our edge caching for the api.

Jun 2007 – Aug 2007 · 2 mos

• Shopping search engine

Aug 2006 – Dec 2007 · 1 yr 4 mos

• User Identity, Online Privacy

Jul 2004 – Aug 2006 · 2 yrs 1 mo