Jun 2024 – Dec 2024 · 6 mos · Menlo Park, California, United States

• ◦ Improving performance for Llama 70B and 405B models on NVIDIA H100.
• ◦ Scaling techniques to make FP8 datatype work efficiently with acceptable numerical accuracy.
• ◦ FP8 GEMMs with blockwise scaling on NVIDIA GPUs that can be extended to groupwise scaling.
• ◦ FP8 Blockwise Scaling GEMM work is also used in Deepseek's V3 and R1 models.
• ◦ I left Meta on Dec 27, 2024 to explore the space of test-time compute, long-context, and reinforcement learning.

Sep 2023 – May 2024 · 8 mos · Mountain View, California, United States

• ◦ Improving GPU performance and programmability for LLMs and more.
• ◦ Planning, scoping, and evaluating the codegen efforts for NVIDIA Hopper architecture.

Jun 2022 – Sep 2023 · 1 yr 3 mos · Mountain View, California, United States

• ◦ MLIR/LLVM codegen in OpenXLA compiler
• ◦ Codgen GEMMs for NVIDIA A100 Tensor Cores using OpenXLA/IREE MLIR compiler to match handwritten (CUTLASS) and library (cuBLAS) performance.
• ◦ Improved half-precision and single-precision NVIDIA A100 codegen performance from 144 to 238 TFLOPs and 77 to 118 TFLOPs, respectively.
• ◦ Also implemented support for batch matmul, split-k, bfloat16, and mixed-precision datatype.

Dec 2017 – May 2022 · 4 yrs 5 mos · Santa Clara, California, United States

• One of the early engineers working on the CUTLASS project from 2017 to 2022, when the team operated with a headcount of 3-4 engineers.
• During this time, accelerating ML on four generations of NVIDIA GPU architectures (Volta [V100], Turing, Ampere [A100], and Hopper [H100]).
• Notable highlights
• Many GEMM optimizations including split-k (serial and parllel), complex gaussian GEMMs, improvements to TensorFloat32 (TF32) GEMMs.
• Designed and implemented Implicit GEMM convolution supporting Fprop, Dgrard, and Wgrad.
• Improved backward strided data gradient by 4x vs. cuDNN 8.2.
• Presented the work at GTC 2021 and GTC 2022 (slides and talk in the links below).

Jan 2016 – Sep 2016 · 8 mos · Austin, Texas Metropolitan Area

• Compiler passes using LLVM for GPU kernels written in C++AMP
• Reliability vs. performance trade-offs study for memories
• Worked on exascale computing project as a part of resiliency and reliability team

Mar 2014 – May 2015 · 1 yr 2 mos · San Diego, California, United States

• Basic Data Structure & Object Oriented Design, Teaching Assistant, Fall 2015
• Software for Embedded Systems, Teaching Assistant, Spring 2014

Aug 2010 – Sep 2017 · 7 yrs 1 mo · San Diego, California, United States

• Static taint analysis for CUDA kernels using LLVM
• Software-based recovery mechanisms for code executing on unreliable hardware
• Compiler infrastructure to analyze and generate reliable code
• Big Data analytics and improving Hadoop performance via reuse

Aug 2008 – Jul 2010 · 1 yr 11 mos

• DSP Firmware Team: Assembly implementation of audio codecs.
• Coded and integrated features such as Dolby Digital decoder, echo-canceller & noise suppressor,
• automatic gain control, and IIR filters on Qualcomm ADSP
• Acquired hands-on knowledge of processor architecture, instruction decoding, memory management,
• OS fundamentals, assembly & C language, RTOS understating, design and development
• of firmware, system simulation and software testing using Tcl programming and JTAGs
• Worked on audio codecs as a part of DSP firmware team

May 2007 – Jun 2007 · 1 mo

• Studied electrical, aeronautical, and hydraulic systems of A-320 aircraft.

May 2006 – Jun 2006 · 1 mo · Greater Delhi Area

• The project involved installation of new transformers, motors, generators and other electrical equipment. I was responsible for transformer sizing for given load considering the impact of highest rated motor. The final system was modeled in Auto-CAD.