Jan 2021 – May 2023 · 2 yrs 4 mos · Vancouver, British Columbia, Canada

• Conceptualized and developed a versatile, API-driven control plane with plug-and-play capabilities to efficiently manage ephemeral Spark clusters, job scheduling, governance, and observability paving the way for a new data platform.
• Prototyped an intelligent co-pilot assistant that reduces on-call workload, allowing for self-serve user support through the utilization of OpenAI API's and their LLM models.
• Undertaken the development of infrastructure for a platform that can ingest real-time domain event streams into BigQuery using Google Dataflow, with the capability to handle up to 30,000 Kafka topics, ensuring scalability and optimal performance.
• Designed and implemented infrastructure and tooling for Shopify's next-generation streaming storage and query platform, utilizing Apache Druid OLAP store, to enable merchants with sub-second latency real-time analytics capabilities.
• Optimized analytical queries for merchants by employing an in-house proprietary roll-up engine to achieve faster query performance

Nov 2019 – Dec 2020 · 1 yr 1 mo

• Designed and developed highly scalable API gateway in Go Lang to handle high volume
• traffic at the edge of Qubole stack with features like Response caching, Compression,
• Reverse proxy, Socks Proxy, Auth caching, StatsD metrics instrumentation and Jaeger
• tracing
• Enhanced Qubole Command flow's fault tolerance on Redis outages and implemented
• automatic reconciliation of stuck commands impacted due to Redis outage
• Designed and implemented a micro service to provide big data cluster images for various
• generation like AL1, AL2.
• Developed a framework to provide HTTP request stickiness between two micro-service
• at Qubole using Kubernetes and istio
• Prototyping open source ruby based godrb processing monitoring framework into
• go-based in house service to make it extremely scalable.
• Implemented secure service on Qubole Data Plane to talk from Qubole Control plane
• securely.
• Designed a lighter version of Qubole-in-a-Box(QIB) which allowed to run software in a
• smaller instance from r3.2xlarge to r3.large

Mar 2019 – Oct 2019 · 7 mos

• Designed and developed custom/external HPA auto scaling using StatsD, collectd
• exporter and Prometheus to scale all the micro services in Qubole Kubernetes control
• plane
• Moving Qubole Control plane from chef based monolith tiers to micro service based
• Kubernetes control plane supported on all major clouds like AWS, GCP & Azure
• Implemented critical core functionality to support multiple active releases running on
• control plane, allowing Qubole to do staggered roll out of new releases in controlled
• fashion with quick rollbacks.

Mar 2017 – Feb 2019 · 1 yr 11 mos

• Contributed critical functionality in implementing Highly Distributed Asynchronous
• Command Processing Platform, which is processing millions of commands across all
• Qubole offerings
• Helped to implement platform-wide background jobs framework and implemented many
• auxiliary background jobs
• Developed critical functionalities in a peer-to-peer real-time command monitoring,
• reconciling and alerting system called as ATC over Redis
• Designed and implemented critical functionalities by moving client tier from Qubole
• control-plane to cluster data plane allowing Qubole to run big data commands for more
• than 36 hours
• Implemented lua script in Redis to automatically trim live logs of big data commands and
• keep the redis memory in check

Jul 2015 – Feb 2017 · 1 yr 7 mos

• Developed critical functionality on tool to build big data cluster images using Hashi Corp
• packer across all major clouds AWS, Azure, GCP and Oracle
• Developed core features on package building tool and optimized it for better developer
• efficiency
• Implemented ability to install any Qubole support packages at run time on Qubole Data
• Plane without having to re-bake the image

Jan 2015 – Jun 2015 · 5 mos

• Developed APIs to enable/disable a Qubole account
• Developed APIs to suspend/active a Qubole account
• Moved existing rspec test cases to new pytest based automation framework
• Developed a tool to run DB/Data/Settings migrations standalone