Dec 2020 – Present · 5 yrs 6 mos

Feb 2019 – May 2022 · 3 yrs 3 mos

• 1. Designed and coded Query Engine to Perform Join and Query Data over 50 Peta byte Data Stored in Object Storage. We used best of all worlds. Used Presto Query Parser for Parsing and optimizing sql queries , Planning SQL Execution Plan, spark for executing tasks and designed own query task scheduler to perform scheduling beyond the framework's limits and custom join logic. We are able to do joins on Peta byte scale data with very little memory footprint and without shuffle.
• 2. Designed federation layer, which serves OLAP queries with millisecond latency. Integrated latest ORC with Cloudera , enabled ZSTD compression and cached ORC files in Alluxio . Query done using presto and we are able to scan billion records per second over very minimum node cluster.
• 3. Currently writing data store for streaming key value based system, which will provide OLAP queries over streaming data with miilisecond latency. It is well known problem in industry and no single existing system works out of box to solve this.
• 4. wrote system to ingest trillion records per day. System writes data over to object storage . Implemented custom S3 committer to support our purpose . It is horizontally scalable system and all data is sorted with custom logic before writing. System daily receive 25 Tera byte of data.
• 5. Commited several changes in Opensource community includes ORC Bloom Filter Support in Presto and Curruped JVM while writing in ZSTD Compression.