Dec 2022 – Present · 3 yrs 4 mos

Dec 2020 – Nov 2022 · 1 yr 11 mos

Nov 2018 – Dec 2020 · 2 yrs 1 mo

• 1. Designed and coded Query Engine to Perform Join and Query Data over 50 Peta byte Data Stored in Object Storage. We used best of all worlds. Used Presto Query Parser for Parsing and optimizing sql queries, spark for executing tasks and designed own query task scheduler and custom join logic. We are able to do joins on Peta byte scale data with very little memory footprint and without shuffle.
• 2. Designed and coded federation layer, which serves OLAP queries with millisecond latency. Integrated latest ORC with Cloudera , enabled ZSTD compression and cached ORC files in Alluxio . Query done using presto and we are able to scan billion records per second over 4 node cluster.
• 3. Currently writing data store for streaming key value based system, which will provide OLAP queries over streaming data with miilisecond latency. It is well known problem in industry and no single existing system works out of box to solve this.
• 4. wrote system to ingest trillion records per day. System writes data over to object storage . Implemented custom S3 committer to support our purpose . It is horizontally scalable system and all data is sorted with custom logic before writing. System daily receive 25 Tera byte of data.
• 5. Implementing Unified customer Master for all categories of Airtel.