Jul 2020 – Jun 2022 · 1 yr 11 mos · Gurugram, Haryana, India

• OLAP Engine using Presto and Spark : Created Aggregated datasets using Spark Batch, added concept of cuboid partition to store multi dimensional aggregates in the same external hive table. Also enhanced code for presto core to intercept a SQL query analyse its AST and rewrite original query at run time with optimal cuboid partition.
• Exactly Once For Spark Batch: Spark Batch guarantees exactly once for only 1 source 1 sink system.
• I enhanced it to provide exactly once semantics for a job having multiple sources and multiple sinks, to achieve this I reused and enhanced internal API(s) from Spark Structured Streaming in Spark Batch, making batch jobs completely fault tolerant.

Aug 2019 – Jul 2020 · 11 mos · Gurugram, Haryana, India

• Feature Factory ETL Framework : Enhanced In House ETL framework responsible for running Spark Job using JSON configuration files. Added support for in memory master management and wrote various UDF(s) and UDAF(s) like approximate distinct measure internally using hyperloglogplus data structure.
• Analytics using PySpark and Presto : Using various data analysis techniques like Row-Pattern Matching, I devised rules to catch various telecom fraud types like FAS and Fixed Line Fraud.

Jun 2018 – Aug 2019 · 1 yr 2 mos · Gurugram, Haryana, India

• Digital Gold : Worked on the backend side to enable application users to buy and sell gold in real time. This feature helped me understand widely used MVC architecture. Application was based upon Spring,
• Hibernate and MySQL .