Dec 2021 – Present · 4 yrs 5 mos · San Francisco Bay Area

• Consulting, teaching on A/B testing and controlled experiments, and how to make organizations more data-driven.
• For my 10-hour live class, see https://bit.ly/ABClassRKLI
• For the 6-hour advanced class, see https://bit.ly/AdvancedABRKLI

Dec 2021 – Present · 4 yrs 5 mos · Virtual

• The class is taught about quarterly.
• I taught at Sphere until 4/2023, then moved the class to Maven when Sphere shutdown live cohorts.
• See https://bit.ly/ABClassRKLI and https://bit.ly/AdvancedABRKLI

Nov 2019 – Nov 2021 · 2 yrs · San Francisco Bay Area

• VP and Technical Fellow [text below approved by Airbnb]
• Led Relevance and Experimentation, a team of world-class engineers and data scientists that included two principals and ten senior staff members. The team delivered 6%+ improvements to booking conversion from about 20 successful product changes out of over 250 ideas that were tested in controlled experiments.
• Surfaces impacted include search, product detail page, home page, post-booking cross-sells, and email. See Improving Deep Learning for Ranking Stays at Airbnb (https://medium.com/airbnb-engineering/improving-deep-learning-for-ranking-stays-at-airbnb-959097638bde)
• Improved data-driven decision making by increasing use of controlled experiments, helped design the OEC (Overall Evaluation Criterion) and OKRs (Objectives and Key Results) for multiple teams, improved experimentation processes and trust, helped build near-real-time experimentation for safe deployments, analyzed hundreds of experiments. See Designing Experimentation Guardrails (https://medium.com/airbnb-engineering/designing-experimentation-guardrails-ed6a976ec669).
• Consultant 3/2021-11/2021

Mar 2014 – Nov 2019 · 5 yrs 8 mos

• As Distinguished Engineer and then Technical Fellow and Corporate VP, evangelized data-driven software development using rigorous controlled experiments at Microsoft, enabling improvements in user experience and surprising ideas that generated incremental revenue of hundreds of millions of dollars.
• Grew the experimentation platform (ExP) team to over 110 Data Scientists, Developers, and Program Managers to accelerate innovation at Microsoft through trustworthy analysis and experimentation.
• Platform now used to run 25,000+ experiment treatments per year (about 100 new per workday) across more than 15 major Microsoft product groups, including Bing, Edge, Exchange, Identity, MSN, Office client, Office online, Photos, Skype, Speech, Store, Teams, Windows, and Xbox.
• Oversee data scientists, who provide a center of excellence for analysis and experimentation for Microsoft.

Oct 2010 – Mar 2014 · 3 yrs 5 mos

• Enabled trustworthy and scalable experimentation at Bing, championing metrics, processes, and algorithms that improved trust, and provided clear data for decision-making.
• Introduced A/A tests, correctness tests, variance reduction techniques, and Overall Evaluation Criterion (OEC) for user-experience vs. revenue tradeoff (Ship User Value).
• Designed and helped execute an important slowdown experiment that showed the value of performance for Bing: each 10msec improvement in server performance generates incremental profit to fund an engineer.

Jun 2005 – Oct 2010 · 5 yrs 4 mos

• Founded the Experimentation Platform team, later merged into Bing, enabling the scalable execution and analysis of controlled experiments on websites and services at Microsoft.
• Hired and led the original ExP team from its inception to 50 high-performing members in Dev, Test, PM, Analyst, Ops, and Support teams, including six principal-level team members.

Sep 2003 – Jun 2005 · 1 yr 9 mos · Seattle, WA

• Headed multiple teams, including ad automation (SEO/SEM), consumer behavior/data mining, site experimentation, automated email, and two personalization teams. Introduced multiple features with total incremental revenue of several hundred million dollars. Grew Amazon’s Data Mining & Personalization organization from about 50 to over 90 people.

Oct 1998 – Jul 2003 · 4 yrs 9 mos · San Mateo, CA

• Vice President, Business Intelligence: Led the Analytic Services and Business Intelligence Sales Demo teams throughout Blue Martini’s IPO (2001).
• Senior Director, Data Mining Applications: Led engineering team responsible for data collection, transfer (ETL), analysis, reporting, visualization, and campaign management for Blue Martini’s product suite. Designed, architected, and performed initial Java coding for data mining, reporting, and transfer models.
• Joined the startup as employee #10.

Oct 1995 – Sep 1998 · 2 yrs 11 mos · Mountain View, CA

• Manager, MineSet, Silicon Graphics Inc.
• Managed the MineSet data mining and visualization engineering team of 15.

Jan 1985 – Jan 1988 · 3 yrs · Israel

• Manager (lieutenant), Israeli Defense Forces
• Managed 12 people at a computer center in a large army base in Israel.

Jan 1981 – Jan 1984 · 3 yrs · Tel Aviv, Israel

• Programmer, International Software, Tel-Aviv, Israel.
• Developed a database application generator, IRIS, while in high school.
• The program was sold commercially, including sales to the Israeli Defense Forces.