Jun 2019 – Dec 2020 · 1 yr 6 mos

Jan 2019 – Present · 7 yrs 4 mos

• Creator of the Hudi Project at Uber. Continuing to lead its evolution at the Apache Software Foundation.

Oct 2014 – Jun 2019 · 4 yrs 8 mos

• Drove technical vision, design & architecture across 2 technology areas core to Uber's business, during the transformative phase of the business as it scaled 100x
• Data: Founding engineer on the data team and architect of Uber's data architecture including streaming data capture, schematizing data, Kafka/Hadoop/Spark/Presto ecosystem. Architected major chunks of strategic roadmaps, including - comprehensive Next generation Data Architecture, china/us active-active. Creator of Apache Hudi, bridging the batch & streaming gap to bring near real-time data processing to Uber's 100s+ PB Data Lake.
• Wireless Transport : Managed a team of engineers, researchers with the mission of keeping the Uber experience real-time. Innovating on the wireless links connecting Uber's cars & riders to the data-centers, by engineering udp-based transport protocols. Optimizing traffic patterns, intelligent edge selection, failover/redirects and observability for networking.
• Helped scaled engineering processes and hone operational excellence across various areas of engineering, to ultimately keep Uber available and reliable, while engineering grew from 200 to 4000+

Jun 2014 – Oct 2014 · 4 mos

• Engineer on the Content Creation group, building the Box Notes app
• Distilled & Documented the Etherpad lite Operational Transformation library
• Designed & prototyped operational transformation code, for building collab editable tables in Box Notes.
• POC for real-time database technologies to power collaborative apps.
• Designed low latency data-store ala Firebase, with delta pushes to standing queries
• Doubled down as a pseudo PM, identifying future use cases & collaborative apps.

Nov 2011 – May 2014 · 2 yrs 6 mos

• Shaped the vision for Linkedin's private cloud "Nuage" to provide horizontally scalable data systems like Voldemort/Espresso/Kafka as a service, to accelerate product development. Built out the first prototype using Voldemort.
• Explore new data infrastructure projects that can be useful for linkedin.
• Continuously Scaling Voldemort to over a million operations/sec across 1000+ servers with several hundred terabytes of data, spanning multiple data centers.
• Mentored an intern to restify the Voldemort Server
• Also still a strong individual contributor to Project Voldemort
• Streamlining usage of Vector Clocks across fast path and data loads from streaming client
• Working with the sleepycat team to move voldemort storage to BDB-JE 5
• Worked on fundamental building blocks for next gen Voldemort Cluster/datacenter expansion algorithms
• Designed and Implemented abortable cluster expansion/datacenter addition
• Built vector clock pruning code to eliminate data bloats after cluster expansion

Nov 2011 – Oct 2013 · 1 yr 11 mos

• Engineer on Voldemort, NoSQL k-v store
• Scaling Voldemort across hundreds of servers, powering large part of Linkedin's online services
• Voldemort server GC tuning
• System Characterization and Capacity Modelling
• Stabilizing multi tenant cluster performance
• Improve Voldemort monitoring to the point of being able to accurately pinpoint problems in production
• Led performance efforts and driving several high impact store ramps on Voldemort (eg: Search, Sponsored Updates), working hands-on with users on the client app architecture
• Putting together building blocks that enabled stream lined provisioning of Voldemort stores, to keep up with exponential growth for Linkedin Product teams.
• Rewrite Berkeley DB storage layer for Voldemort
• Providing improved GC performance and faster response time for online operations
• 20x speedup of Rebalance and Restore operations to maintain voldemort clusters
• System Characterization and Capacity modelling to support multi-tenant Voldemort Clusters
• Involved in database replication for Linkedin's active-active efforts to operate the site with higher availability. Helped users of Voldemort, re-engineer their applications to work with a master-master replication model.

Jun 2009 – Nov 2011 · 2 yrs 5 mos · San Francisco Bay Area

• Oracle Streams/Goldengate/XStream distributed data replication products.
• Performance: Developed shared memory management/caching infrastructure for Oracle XStream Out, a high performance native streaming interface to get relational data out of Oracle, producing 2X throughput gains.Implemented intelligent throttling mechanisms for shared memory usage to achieve a stable balance of safety & performance.
• DML Reordering: Conceptualized and implemented a DML reordering algorithm, to improve data apply throughout, by reorganizing DML operations across transactions, so as to maximize opportunities for batch processing.
• Coordination Control: Designed a coordination layer to synchronize multiple Oracle GoldenGate Replicat processes, on conflicting operations such as PK-Updates, DDL, event markers to preserve consistency even in the face of failures.

Jan 2009 – May 2009 · 4 mos · Austin, Texas Area

• Data Flow shell project, building shell extensions to facilitate large scale parallel processing.
• Links:
• http://sites.google.com/site/ewalker544/research-2/dataflowshell
• http://dl.acm.org/citation.cfm?id=1645175

May 2008 – Sep 2008 · 4 mos · Austin, Texas Area

• Worked with Prof.Yin Zhang and Prof.Lili Qiu on 'Content Delivery in delay tolerant networks', enabling high bandwidth communication to mobile users on the road, by leveraging novel opportunistic replication & mobility prediction techniques.
• Publications:
• http://dl.acm.org/citation.cfm?id=1921199

Jun 2007 – Nov 2007 · 5 mos · Hyderabad Area, India

• Involved in development of Peer-to-Peer Realtime Trading Systems. Incorporated Synchronization algorithms to achieve intended runtime behavior of different distributed components