Apr 2021 – Aug 2022 · 1 yr 4 mos

• I was part of the Data Collection team which owns a highly available, multi-tenant, distributed log, and metrics ingestion pipeline. We are operating at a massive scale where every day 300 TB+ data are ingested. At peak, we scale up to the rate of 4 PB/day logs ingestion.
• Projects worked on:
• Worked on ingest infrastructure in Scala and Java on AWS using EC2, S3, Kafka, Elasticache (Memcache/Redis), ELB/ALB load balancers, Zookeeper, RDS (SQL/Aurora), DynamoDB, SNS.
• Scaled Amazon S3 log ingest over to 10+GB/second:
• a. Used Apache Kafka as a persistent queue.
• b. Improved S3 objects discovery using SNS
• c. Migrated from RDS to DynamoDB.
• Worked on telemetry agent in Scala sending petabytes of logs and metrics to the cloud.
• Lead a team of 4 developers & QE to implement projects:
• 1 Ingest Budget 2. Logs Archival 3. Logs Metadata Enrichment
• Implemented data auditing and accounting to track ingestion pipeline SLI and monitor per customer SLA.
• Member of Change Management Board, Build management and Developer productivity team.
• Key Tech- Scala, AWS, SOA, Kafka, Spring, Memcached, Redis, DynamoDb, MySql, and IntelliJ. (Also, Java, Python)

Jan 2019 – Mar 2021 · 2 yrs 2 mos

Aug 2017 – Jan 2019 · 1 yr 5 mos · Bengaluru, India

Jun 2016 – Jul 2016 · 1 mo · Bengaluru, Karnataka, India