Jul 2022 – Dec 2025 · 3 yrs 5 mos

• Designed and led development of Analytics and Reporting pipeline using Apache Kafka, Flink and Druid for warehousing and Big Query for Reporting
• Real-time Session Attribution for e-commerce signals using Flink's TableAPI and Session Windowing
• Customizable Aggregation jobs for users, segments etc using CockroachDB
• Schema management microservice for Apache Solr written in golang, supports smart schema switches on catalog refreshes and supports multiple languages and configuration templates
• Architected and developed a high-performance data ingestion pipeline from the ground up, processing 134M documents in 90 minutes with 5,000 full uploads and 7,000 delta updates daily. Built a cloud-native solution using microservices API, Argo Workflows on Kubernetes with Spot instances, Kafka event streaming, and NATS.io for state management. Implemented advanced features including multi-tenant resource isolation, dynamic scaling and sharding, intelligent queuing for multiple upload types, AI-powered enrichment models, multi-level document hierarchy support, and Solr 9 integration
• Git trigger based Release Automation using FluxCD, used for managing 37 kubernetes clusters

May 2021 – Jul 2022 · 1 yr 2 mos

• Developed a custom Solr Routing microservice implementing Envoy's dynamic resource discovery (xDS) protocol over gRPC, driven by real-time replication events. Built an intelligent replica placement system leveraging Solr and traffic metrics to optimize performance, featuring smart query routing that surpasses default Solr routing capabilities while seamlessly supporting multiple Solr clusters for enhanced scalability and load distribution.
• Setup Nats super cluster, links 7 regions across the globe, used for cross region sync
• Developed centralized configuration management service using Dgraph with config change sync using Nats bus
• Developed Query ranking service, override default Solr's document scoring, with custom user defined rules

Oct 2016 – Apr 2021 · 4 yrs 6 mos

• Implemented a Disaster Recovery solution featuring periodic synchronization of production clusters to disaster recovery environments with template-based backup and restore jobs using Argo Events and Argo Workflows
• Designed and developed the initial iteration of a pub-sub based ingestion pipeline from scratch, achieving 1M document processing in 20 minutes. Built a comprehensive system featuring an API server for catalog data handling, Kafka for document streaming, Storm topology for downstream processing, and a polls-based state management system. Implemented multi-topic distribution, queue and Solr-level site isolation, and batch file upload capabilities to ensure scalable and efficient data ingestion.
• Trending product/queries calculator using Apache Storm for capturing e-commerce website's realtime trends

Apr 2014 – Sep 2016 · 2 yrs 5 mos · Chennai Area, India

• > Upgrade Lucene 3.0.3 to 4.10.4, including migration utilities
• > Instant results for Zoho CRM using Lucene edge grams
• > Unified Zoho Search with SSE for dynamic results

Jan 2014 – Apr 2014 · 3 mos · Chennai Area, India

• > Monitoring module for search and indexing load on HDFS