Dec 2020 – Dec 2022 · 2 yrs · On-site

• Platform Engineering:
• 1. Kubernetes Orchestration:
• Setup bare-metal cluster of Kubernetes on an on-premises datacentre.
• Wrote automation scripts & groovy deployments pipelines for hassle free onboarding.
• Over 300 pods running across more than 25-30 namespaces improved env bandwidth.
• 2. AWS Cost Reduction & On-Premises Data Center:
• Lead the project for two business units to revisit all the AWS services under usage as per AWS bills and optimized following costs: Data-
• transfer, EC2, EBS, EB, ALB, S3, Elastic Cache, RDS, Elastic IPs, ECR, ECS, EKS, NAT & Internet Gateways, Site-to-Site VPNs, etc.
• Migrated prod, non-prod, common resources into separate VPCs, Subnets, AZs & On-Premises Data Center for improved security, efficient traffic routing & low operating cost.
• Reduced overall monthly cost by $50K across all business verticals and ~$1.6K per day for the 2 BUs I lead.
• Microservices:
• 1. Process Standardizations:
• Rest API request/response headers, exception contracts, error codes and exception handling process.
• Code documentation (HLD/LLD), code review standards, code review process, release process.
• Migrated location data from MongoDB to Neo4j for efficient graph traversal queries and PostgreSQL for recursive queries on hierarchal data.
• Data Science:
• 1. Fleet Network Planner:
• Lead, designed and developed a fleet network routing and scheduling engine for a vehicle fleet of 200+ vehicles across 500 warehouses
• spanning 19000 Pan India PIN codes via 1800+ trips for national, regional, and intra city movements.
• Formulated constraints into a linear programming model. Used solvers provided by Jsprit and OR-Tools, concepts from Graph Theory, Unsupervised learning for optimal network topology, OSMnx & PostgreSQL for processing shapefiles and querying geo-data.
• Shipped production model with Airflow DAGs for ease of execution, retries, & notifications.
• Reduced network planning time from 15 days to 3 hrs.

Jan 2019 – Dec 2020 · 1 yr 11 mos · On-site

• Data Engineering:
• 1. Real-time CDC Data Pipeline:
• Contributed in setting up an self-managed CDC stream platform consisting of clusters of Zookeeper, Kafka,Kafka Connect, Schema Registry & CDC Debezium connectors processing millions of events per day across all BUs.
• 2. Distributed ETL Workflow Orchestration:
• Setup in-house ETL workflow orchestration platform with Airflow, Redis & Celery Workers handling hundreds of tasks per day.
• Wrote a codebase with coding standards driven by Julien Danjou's "Serious Python".
• 3. Data Warehousing & BI:
• Setup self-managed DW-BI stack consisting of Apache Superset for analytics, TimescaleDB for data warehousing & ETL/ELT jobs with Spark & Athena.
• Used Timescale's HPO & monitoring tools - Hypertables, Columnar compression, PgHero, PgBadger, BARMAN.
• Used Superset’s HPO via Gevent based Gunicorn workers & Domain sharding.
• 4. Centralized Logging:
• Setup in-house Elasticsearch cluster with nodes running in multi-roles mode ingesting & processing TBs of logs per day.
• Wrote ingest pipelines for processing logs to be pushed by Filebeat & visualized at Kibana.
• 5. HA Redis-Sentinel Cache:
• Introduced a self-managed Sentinel based Redis cluster handling billions of keys per db.
• Microservices:
• 1. Notification microservice:
• Designed and implemented a service for publishing notifications to different channels such as Webhooks, WhatsApp, SMS, Email.
• Used vavr monads, fluent design pattern, GraphQL for mutations & abstract factory pattern for orchestrating enricher, fabricator, publisher factories.
• Provided flexibility for custom workflows.
• 2. Generic client library for Kafka Streams:
• Designed and implemented a client to abstract out standard APIs for regular/idempotent consumer/stream topologies, KStreams, KTables, Serde, Stream Orchestrators, streams health indicators & state notifiers.
• Used SOLID design principles & design patterns - Template, Flyweight, Observer, State etc.

Jul 2017 – Dec 2018 · 1 yr 5 mos · On-site

• Data Science:
• 1. Bin Packing for Optimal Vehicle Utilization:
• Lead, designed and developed an LP model and shipped the project with a Flask based microservice with 97% unit test coverage.
• Used CBC solver to solve the linear equations and optimize the objective function, ML techniques such as Decision Trees, Random forest, XGBoost and TSA algos such as Holt-Winters, ARIMA using statsmodel lib to learn the optimal weights of the objection function.
• Achieved utilization of ~99% for warehouses with surplus load and a revenue impact of INR 1.5 Crs per week due to improved trip planning.
• 2. Information Retrieval from POD Images:
• Designed and implemented a CV and QR logic based product to segment ROIs from POD document images and video-frame sequences.
• Used key-frame extraction techniques and solved for linear transformations, rotations, reflections, scaling using OpenCV3.0.
• Better classification of images expedited dispute resolution process.
• 3. Recommendation algorithm for Load Farming:
• Implemented gradient based strategies, analysed time-series using EMA, Holt-Winters’ & SARIMA reinforced via feedback loops.
• Better understanding of user behaviour lead to timely approach and convert thousands of corporate clients, retail, and micro-market users.
• 4. Pilot Allocation:
• Developed a KDTree based solution to find and allocate the nearest pickup captain w.r.t requesting user’s location.
• Reduced overall transit time of the shipments increased CPM by ~18% per week at a scale of 10K CNs per day.
• Microservices:
• 1. Monolith to event-driven microservice architecture:
• Used CDC pipelines to write identification and enrichment logic of more than 50 events involved in a consignment’s journey.
• Significantly improved API latencies and scaling issues due to monolith.
• 2. Secured API Gateway using OAuth2:
• Implemented OAuth’s Client-Credentials flow using JWT, Netflix’s Zuul for traffic routing and managed roles & permissions via bipartite graph.

May 2016 – Jul 2016 · 2 mos · Bangaon Area, India

• Worked with Fidelity's Fixed Income team to provide a solution to the constraints and challenges faced by FI Shared Web Services (which use SOAP as the fundamental communication protocol) without actually migrating towards RESTful architecture but to meet growing demands and need to reduce overall Time to Market.
• Developed a service with multi-versions deployed on the same server which can incorporate changes made in Oracle database dynamically with ZERO down time.
• Configured MEMBRANE, an open source API Gateway for SOAP and REST services to direct and channel the client calls with the respective version of service they want to avail.
• The feature allows client applications such as money market trading, bond trading etc. to move at their own pace without getting affected and without knowing about the changes on the server side.