Jan 2025 – Present · 1 yr 4 mos

• 1. Engineered a long-running FaaS execution runtime supporting 2-hour workloads (beyond the typical 15-minute serverless limit), with durable progress tracking and safe retries.
• 2. Implemented dynamic ACK extension and heartbeat-based in-progress signaling on NATS JetStream, enabling stable execution across 10K+ concurrent worker pods.
• 3. Optimized worker runtime with parallel S3 data ingestion and Redis metadata fetch, significantly improving throughput for heavy data workloads.
• 4. Led KEDA-based autoscaling for FaaS, enabling intelligent pod scaling and delivering substantial infrastructure cost reductions across environments.
• 5. Designed a multi-tenant routing module using Scala and Kubernetes Java Client, leveraging Kubernetes’ etcd-backed control plane to safely store shared metadata while maintaining strict tenant isolation.
• 6. Re-architected Kafka → Snowflake ingestion from continuous Snowpipe/Streams to Redis offset–based on-demand sync, cutting Snowflake warehouse costs and preparing the system for Databricks migration.
• 7.Time-based partitioning for output-status archival with automated retention policies, significantly improving database performance and reducing ingestion latency
• 8. Developed smart health checks for Kafka Connectors to detect ingestion drift, connector stalls, and automatically recover, significantly improving system reliability.
• 9. Optimized PostgreSQL query performance by tuning random_page_cost, delivering measurable gains not only for FaaS but across multiple Innovaccer teams.
• 10. Resolved AWS STS throttling issues rapidly by implementing jitter-based retries, ensuring system stability under high concurrency.
• 11. Resolved S3 throttling by redesigning key/path structure to distribute load and improve throughput
• 12. Added Kafka Tiered Storage with S3 offloading, reducing broker disk pressure by enforcing 1-day local retention while maintaining 3+ days of remote retention, preventing broker failures and improving cluster reliability

Jun 2022 – Dec 2024 · 2 yrs 6 mos

• 1. Redesigned FaaS architecture by removing Postgres from the hot path (10K+ pods caused contention) and moving to S3/ADLS with Snowflake-native loading; reduced runtime from ~80 to ~32 minutes for 100K patients.
• 2. Automated Kafka → Snowflake ingestion using a Kubernetes-native deployment workflow for the Snowflake Connector, enabling zero-touch and reliable data delivery.
• 3. Built selective retry architecture that reprocessed only failed entities using existing S3 mappings, avoiding full reruns and reducing Snowflake/S3 costs.
• 4. Led migration of legacy Scala services to Scala 2.12, unified repositories, and refactored large codebases to improve maintainability and build stability.
• 5. Resolved critical FaaS and distributed-system failures using Scala, Python, Kubernetes, and Kafka to stabilize and optimize production systems.
• 6. Implemented configurable retries with automatic DLQ routing to prevent poison-message loops and simplify reprocessing.
• 7. Replaced polling with webhook-based async execution notifications, adding retries and exponential backoff to improve reliability and reduce latency.
• 8. Enabled Redis durability via AOF with PVC-backed storage to ensure crash recovery, prevent data loss, and strengthen fault tolerance.
• 9. Optimized memory management by removing RLimit constraints and tuning OOM scores to protect the main process during memory pressure.
• 10. Designed and implemented a Circuit Breaker pattern to isolate failing downstream services and prevent cascading failures, enhancing system resilience and fault tolerance.