Apr 2020 – Jul 2021 · 1 yr 3 mos · Bengaluru, Karnataka

• Developed a microservice to create Kubernetes clusters on public/private clouds and setting up YuniKorn scheduler on them. YuniKorn is Cloudera's custom pod scheduler for big data jobs. The microservice is consumed by the Quality Engineering team to test the builds of YuniKorn scheduler on multiple versions of Kubernetes
• Link: https://yunikorn.apache.org/
• Developed a service to trigger notifications to internal teams based on microservices' events and also execute workflows. The notifications can be sent to any communication medium like Slack, Email, SMS and the workflows can be created and executed to take actions based on events
• Single-handedly automated the end-to-end deployment of all the services of Cloudera's engineering infrastructure. This is getting used by the partner and quality engineering team to validate Cloudera's products on partners' infrastructures. It reduced the infrastructure setup time from approx 30 days to just 1 hour with 0 errors
• Revamped an internal library to make it thread-safe so that it can be used with REST-APIs. The library is used to report call stacks to a central service which is used for debugging issues in the internal services of Cloudera
• Decoupled all the common operations of core microservices to a central microservice, following a concept similar to .SO/.DLL (shared library) files. This helped in revoking secrets access from the core microservices for common operations like S3 reads/writes, also giving the liberty to update common operations without redeploying the core microservices, making the system more reliable and secure
• Developed a microservice to deploy and test the builds of Liftie which is a multi-cloud kubernetes provisioner used by CDP to provision kubernetes clusters for each customer account
• Owns the Kubernetes stack of production microservices
• Won Cloudera Commitment Award
• Helped the team in breaking monoliths into microservices and innovating in developer productivity

Sep 2018 – Apr 2020 · 1 yr 7 mos

• Rearchitected the data layer of the product to make it isolated at the per-user level to make multi-tenancy possible
• Developed an Incident Alerting Service using serverless on AWS which is used for triggering alerts to customers via Email, Slack, SMS, TAXII, Pager Duty etc. The SLA of the service is 99.9% and handles 5-20K requests per day with a total monthly burnout of just 3-10$, following pay-as-you-go model
• Developed a domain impersonation/infringement detection service which monitors millions of websites using Kubernetes. The service has successfully detected 1000+ phishing/impersonating domains until now for companies like HDFC Bank, PayTM, Flipkart, Amazon
• Single-handedly deployed Kubernetes in production and led the revamp of backend services to stateless/cloud-native design. It helped the business to grow from handling the load of 30 businesses to 100+ business customers and enhanced the product performance and resilience by 5x
• Redesigned AWS VPC to make it more secure. Added services like NAT Gateways, VPC endpoints for private access to API gateways, VPC Flow Logs, Cloudwatch events/alarms, Route53 etc
• Mentored a team of 15+ engineers
• Led the development of webapp vulnerability assessment service which auto-discovers customers’ web applications and performs a deep vulnerability assessment on it
• Developed backend services for data filtration, transformation and aggregation in Elasticsearch clusters

Apr 2017 – Aug 2018 · 1 yr 4 mos

• Developed a service to auto-discover the internet exposed digital infrastructure of customers and perform periodic vulnerability assessment/misconfiguration scans on it
• Developed a semi-automated crawler using selenium to get discussions data from darkweb forums
• Decomposed the monolithic central data acquisition pipeline into REST-based microservices design following the separation of concerns principle. It gave the liberty to the data science team and data acquisition (crawling) team to maintain/scale their services independently
• Mentored the transformation of the entire backend into microservices architecture
• Developed a service to store/search billions of leaked credentials. The service houses 20+ billion records in Elasticsearch and starts/stops on-demand to cut down the monthly EC2 cost from 80K to 500rs per month using boto3
• Single-handedly maintained the provisioning/maintenance of the infrastructure of the entire company on AWS

Dec 2016 – Mar 2017 · 3 mos

• Developed a scalable central data pipeline to gather data from web crawlers, clean/transform the data, classify and ingest it into Elasticsearch resiliently
• Contributed to product architecture to handle the processing of millions of crawled documents per day
• Created a multi-zone Elasticsearch cluster which housed billions of documents on EC2 instances with EBS storage (GP2).
• Solved gazillion shards problem with a max search latency of 1sec per query
• Deployed, hardened and maintained production services on AWS