Apr 2018 – Feb 2020 · 1 yr 10 mos · Bengaluru Area, India

• Started my Qubole journey by optimizing old ETLs in the system, to get up to speed with the system.
• Identified and reduced S3 listing calls cost. Migrated Hive jobs to Apache Spark and
• tuned them for high resource utilization, to avoid resource wastage
• Contributed to cost explorer for clients, enabling them to monitor and track cost at the query, user, cluster level
• Productionised Bifrost, Qubole’s in-house platform for data sharing with customers over s3.
• Sharing insightful data with 1000s of customer accounts daily.

Nov 2016 – Apr 2018 · 1 yr 5 mos · Bengaluru Area, India

• Built a powerful analytics platform for the Media industry.
• Key features include query response time on hundreds of terabytes of data in seconds, real-time alerting and reporting, cloud-enabled environment, SQL-enabled with optimization on count distinct queries, ad-hoc analytics, REST, and Thrift APIs, etc. Involved in setting up ETL pipelines on the scale of 100 GB per hour for various clients. Worked on query optimization on hundreds of terabytes of data. Worked on client-side for orchestration of multiple ETL pipelines using Airflow. Worked on customization of our platform to accommodate various client requirements like data replacement, collating data from multiple sources. Have good knowledge of the internals of Apache Spark. Have worked extensively on spark job optimization. Can identify the root cause of slowness of a spark job.

Jan 2016 – Jun 2016 · 5 mos · Mumbai Area, India

• Worked in the Risk Aggregation team of Morgan Stanley as a spring intern.
• Worked on an internal web dashboard. Contributed by writing two end to end applications.
• Responsible for creating view(UI), service(REST) and an automation script(PERL).