Sep 2020 – May 2022 · 1 yr 8 mos

• Leading the Apache Spark team at Microsoft India. Working on making Apache Spark performant, resilient, scalable and cloud native.
• Improvising Spark catalyst optimizer to improve the query performance. We were able to improvise Spark performance by >2x.
• Leveraging the flexibility of the cloud to scale-up and scale-down Spark clusters based on the requirements of job.
• Using hardware accelerators(FPGA) for speeding up some of heavily used Spark operators.
• Native execution layer for Spark operators like hash aggregation and sorting.
• https://techcommunity.microsoft.com/t5/azure-synapse-analytics/apache-spark-in-azure-synapse-performance-update/ba-p/2243534

Oct 2018 – Aug 2020 · 1 yr 10 mos

• Leading the Apache Hadoop team to run Cosmos (300 thousand node cluster)
• Yarn Federation.
• Yarn opportunistic container support.
• HDI Auto-scaling
• Dhalion - a generic healing and auto-scaling platform.
• https://github.com/microsoft/Dhalion
• Started Apache Spark India team for improvement on Apache Spark.

Dec 2017 – Sep 2018 · 9 mos

• Working on scaling Hadoop and YARN to run at largest clusters in the world. Worked on various features including node partitioning, opportunistic containers support, Application timeline server, federation support and SLS enhancements. Also worked on a generic auto-scaling framework Dhalion which can be used to auto-tune and auto-scale various big-data technologies.
• I am also leading team to deploy Spark on the scale of Cosmos and do various enhancements in Spark for same.

Jul 2019 – Present · 6 yrs 8 mos

Sep 2015 – Dec 2017 · 2 yrs 3 mos · Bengaluru Area, India

• Working as a Lead developer on Hadoop 2 and YARN stack. Developed features like YARN based autoscaling framework, spot nodes rebalancing for ephemeral clusters, persistent job history storage service, s3a filesystem enhancements. I have added responsibility to debug and fix customer issues with hadoop jobs.

Feb 2015 – Aug 2015 · 6 mos

• I worked on integration of various online services in Adobe Reader. For this, we needed to integrate CEF within Reader and then expose a bridge between Javascript and C++. Since I was among the senior persons in team, I have added responsibility to groom up new joinees and lead them.

Feb 2013 – Feb 2015 · 2 yrs

• Server side development for Acrobat.com services.
• Design and implementation of new services for Acrobat.com

Jun 2011 – Feb 2013 · 1 yr 8 mos

• Design and implementation of various features in Acrobat - Currently moved to Acorbat.Com team which requires Agile development and server side development skills.
• Technical owner for IPM(In-Product-Messaging). The feature includes a server through which new messages can be created and pushed dynamically and a client in Acorbat which consumes those messages to display them appropriately with appropriate actions.
• Integration of Blueheron with Acrobat/Reader: The feature includes providing the Blueheron tasks like Save to Blueheron Server, Open from server, Overwrite a file etc. directly from Acrobat/Reader for Blueheron hosted files.
• Languages/Platform: C, C++, Python on Windows and Mac OS X

Apr 2010 – Jul 2010 · 3 mos · Gurgaon, India

• Designed and developed a full application which provides full functionality of Control Room. Through which multiple sources may be connected and controlled through a common place.

May 2009 – Jun 2009 · 1 mo

• I have worked there on fabrication of chip, chip designing and sensors.