Aug 2023 – Aug 2024 · 1 yr · Remote · Remote

• ● Leading the creation of a data quality framework capable of connecting to various data sources for comprehensive data validations and quality checks using Apache Spark, Flink, and Postgres.
• ● Designing the framework to interpret and execute data quality rules expressed in SQL, ranging from basic Null checks and row counts to complex anomaly detection.

Sep 2020 – Sep 2022 · 2 yrs · Seattle, Washington, United States

• ● Led a team of three to design and implement self-serve reporting frameworks and up-level the data quality for Microsoft 365.
• ● Architected and implemented a Centralized Ingestion framework and data-aware pre-computation and aggregation that handles hundreds of Terabytes of daily data.
• ● Solved complex organizational challenges involving disparate data sources by designing a Unified Metrics framework as the single source of truth. Defined the standards and created a process of how Microsoft 365 should onboard new metrics and datasets.
• ● Developed Autotune using historical data and ML models to predict spark resource usage, which saved Microsoft $2 million.
• ● Converted the 100 TB daily batch pipeline to streaming and improved the data freshness latency by 20X. Implemented Exactly-once semantics using Flink checkpointing and Delta Lake Atomic commits. Solved complex challenges related to small files by implementing dynamic partitioning and reduced the number of output files from millions to thousands.
• ● Built and Implemented a two-year roadmap for Microsoft 365 to have the highest quality data.

Jun 2018 – Aug 2019 · 1 yr 2 mos · San Francisco Bay Area

• ● Part of the API Gateway service framework team that migrated Versa Director from Monolith to SOA. Designed and implemented distributed Rate Limiter and Authorization Microservices using Spring boot, Java and SQL.
• ● Implemented Thrift IDL service configuration template, which auto-generates all the boilerplate code like observability, rate limit, circuit breaking, and backpressure for the microservices.
• ● Implemented smart load balancing between microservices using consistent hashing.
• ● Implemented query layer API microservice for Elasticsearch clusters. This query layer is responsible for generating efficient queries and rejecting expensive queries. Implemented distributed Rate-limiting to avoid coordination between query API nodes.

Oct 2008 – Mar 2018 · 9 yrs 5 mos · Palo Alto, California, United States

• ● Led a team of three in managing a complex ecosystem built on Kafka, Flink, and Cassandra, demonstrating leadership skills and proficiency in these technologies.
• ● Optimized ingestion pipelines resulting in reduction of daily storage from 3TB to 40GB, highlighting efficiency in resource management. Reduced the storage and computing costs by 60%. Saving VMware > $1 million per year.
• ● Scaled Kafka to ingest billions of messages daily across hundreds of nodes in geo-replicated Active-Active clusters, showcasing ability to handle large-scale data processing.
• ● Designed and implemented a robust monitoring and auditing system for Kafka clusters to improve production readiness and solve data quality challenges.
• ● Implemented Distributed Profiler on Apache Spark applications leading to a reduction of memory usage by 500GB, demonstrating expertise in performance optimization.