Aug 2019 – Feb 2020 · 6 mos

Jan 2016 – Jul 2019 · 3 yrs 6 mos

• ● Worked on Spark CBO stats estimation fixes for Aggregate and Sort operators. Speed up of almost 2x in select queries like Q83 of TPCDS. Contributed back to open source - (commit id b1857a )
• ● Fixed deadlock issue in Spark’s UnsafeExternalSorter affecting one of the largest Qubole customer workloads. Contributed back to open source - (commit id 6c4552c6 )
• ● Spark - S3 Select connector for Qubole Spark to push down projects and filters for CSV and JSON automatically; TPCDS benchmarks Geo-mean - 2.9x ; Max speedup - 5x (Blog - https://www.qubole.com/blog/amazon-s3-select-integration/)
• ● Serverless Spark on AWS Lambda - Spark executors completely runs as Lambda functions with S3 being the external storage to manage shuffle data (Blog - https://www.qubole.com/blog/spark-on-aws-lambda/)
• ● Worked on Qubole Spark autoscaling based on stage progress - pluggable, custom auto-scaling policies can be defined.
• ● Implemented Workload based Scaling limits leveraging Apache YARN’s Fair Scheduler queue limits.
• ● Implemented HDFS auto-scaling , scales up nodes based on DFS disk capacity and incoming data velocity.
• ● Mentoring new grads and interns; PR reviews; Spark version upgrades; On-call, Customer issues troubleshooting etc.

Jun 2014 – Dec 2015 · 1 yr 6 mos · San Jose

• ● Worked on a ​Real time performance monitoring/troubleshooting dashboard for large MapR Hadoop clusters; Contributed from the POC stage
• ● Worked on Apache projects like Drill, Spark and Hive; Contributed to Apache Drill; Participated in
• user/dev groups of other open source projects like Apache Samza.
• ● Developed ​Real time Log analysis for MapR Hadoop clusters using ELK stack.

Aug 2013 – May 2014 · 9 mos · Chandler

• ● Designed and developed a proof of concept on "Machine Learning as a Service" - Cloud based framework.
• ● ML as a service framework exposes machine learning algorithms available in various packages (Weka, Scipy, Numpy, Mahout etc) as web services.
• ● Developed visualizations using D3.js to analyze data samples over a timeline graph
• ● Set up a single node Apache Hadoop cluster to demonstrate the idea.

Jun 2013 – Aug 2013 · 2 mos · Phoenix, Arizona Area

• ● Constructed Data Pipelines to aggregate and summarize instrumentation logs; Extracted behavioral attributes from the generated logs using Hive UDF’s; Created an automated Workflow using Oozie.
• ● Social Graph Analysis on discussion data stored in HDFS. Calculated Prestige score of each participant in order to measure the importance of the participant in the network.
• ● Graph Visualization of the discussion forum using D3.js is also developed.
• Used: Hive, Oozie, Spring MVC, JAVA, D3.js

Aug 2012 – May 2014 · 1 yr 9 mos · Tempe, AZ

• Masters in Computer Science