Jan 2025 – Mar 2025 · 2 mos · Menlo Park, California, United States · Hybrid

• Worked on optimizing Data Pipelines feeding into the Apache Hudi based Data Lakehouse organized in a Medallion architecture. The data consisted approximately half of all card-based transactions in the U.S. and a quarter in the U.K, and powers rewards experiences in various banking and finance apps. Was able to achieve close to 50% cost and latency savings through Spark SQL query analysis and optimizations. Also Lead the effort to run spark jobs on spot EKS instances for more cost savings.

Aug 2022 – Dec 2024 · 2 yrs 4 mos

• Leadership and Growth:
• Stepped up to lead the Yahoo Knowledge (YK) Engineering Team, expanding it threefold across the US, India, and Canada. Mentored, and successfully advocated for team members' promotions.
• Led the Consumer Research Org's Digital Transformation (DT) to public clouds, advocating for Infrastructure-as-Code with Terraform/Terragrunt and leveraging cloud-native solutions. Collaborated with the Central Big Data Team to identify issues, build a strategic roadmap, develop, and deliver custom Terraform modules and other solutions for AWS, tailored for Yahoo-specific needs. Adopted by several teams inside Yahoo within a year.
• Implemented best practices in software engineering in the YK Team, including enhanced documentation, unit/integration testing, Agile development, and CI/CD with infrastructure-as-code (IAC). Achieved AWS proficiency, leading the team to be among the first at Yahoo to handle live production traffic on AWS, while also reducing latency by 30%. Set up several data pipelines in AWS utilizing MWAA, EMR, Spark, ECS/Fargate, and Lambda.
• Successfully secured funding to set up a new Production Engineering (PE) team in India to support YK and other Consumer Research Org teams. Alongside setting processes and coached PEs to ensure best devOps practices.
• Managed external staffing by working with agencies like Infosys and Toptal to recruit and train contractors, driving DT progress.
• Product Achievements:
• Successfully launched the Watch-To-Watch module in collaboration with the Web Search team, receiving company-wide acclaim. Partnered with Business Development and external vendors to introduce affiliate streaming links creating new revenue streams.

Oct 2018 – Aug 2022 · 3 yrs 10 mos

• Implemented the Image Pipelines for the Yahoo Knowledge graph end to end, which involved setting the Image Acquisition pipeline(including integration with Getty Images), image batch processing, thumbnail generation, ML feature extraction, indexing, ranking, and serving. This system powers most of the image experiences on Yahoo Web Search Knowledge Cards and other modules
• Transitioned the Yahoo Knowledge API from RestFul API to a Graphql-based API which dramatically increased the adoption and usefulness of the API within the company. Also onboarded the Knowledge Graph on AWS Neptune to unlock the power of the cloud and horizontal scaling. Also enhanced the Graphql API by connecting to different Vespa indexes and federation to other Yahoo APIs. It now powers many systems with Yahoo including Knowledge Cards, Showtimes, X-ray, Related Entities
• Helped build the Question Answering system over the Yahoo Knowledge Graph which provides direct answer modules in Yahoo Search Search to questions of Age, Kids, net-worth Etc.
• Spearheaded the modernization of all serving systems including containerization, and moving to Kubernetes from Bare Metals/VMs. Added open telemetry integration, performance, and integration tests and was able to reduce hardware requirements by 70-90%

Oct 2015 – Sep 2018 · 2 yrs 11 mos

• Rebuilt the Yahoo Knowledge Graph using Apache Spark to scale it from ~10 million entities to hundreds of Millions of entities with Billions of facts. Worked on several modules for Web mining, structured data extraction on the Web, large-scale knowledge graph construction and management (incl. schema alignment, entity reconciliation, anomaly detection), and entity recommendation. The data is used to power several knowledge Card and experiences in Yahoo Web Search and other properties

Mar 2014 – Sep 2015 · 1 yr 6 mos

Feb 2011 – Feb 2014 · 3 yrs

• Worked at Yahoo's CPG( cloud and Platform Group), handling web-scale knowledge extraction using Hadoop,Map-Reduce, PIG, Oozie, Hbase etc.
• Setup up several critical open source data pipelines including
• 1. Wikipedia and Freebase: Included download and processing of monthly dump, and live wiki page changes by tracking IRC Channel updates
• 2. Expanded Yahoo's Internal GEO warehouse by gathering and merging data from Wikipedia and OpenStreetMap, greatly increasing the coverage of data including Labels in different locales

Jul 2007 – Feb 2011 · 3 yrs 7 mos · Noida, Uttar Pradesh, India · On-site

• Was one of the founding engineers in the team that built Limedomains.com, a domain registration and web hosting company. Worked as a full stack engineer, using Ruby on Rails, JQuery, Mysql etc, following Agile Methodologies.
• Worked on several modules including payment processing, Reports generation, Order Management, Management console, Shopping cart and offers etc