Oct 2014 – Oct 2015 · 1 yr · Menlo Park

• Backend AI infrastructure development - building scalable distributed infrastructure for enabling complex data analytics algorithms and machine learning jobs execution on top of YARN.
• Develop distributed infrastructure for high performance batch and interactive data analytic processing on top of native YARN. Implemented application master to interact with resource manager and node managers to schedule and launch various machine learning data analysis jobs on top of Hadoop cluster.
• Design and implement highly scalable and flexible system architecture to allow the infrastructure adaptive to physical clusters. The system employs a group of application masters to separate the job scheduling load; dynamically discovers and monitors the liveness of application masters; balances the work load among the application masters.
• Update cluster to CDH5 and employ Cloudera manager to config HDFS and YARN parameters; implement kerberos based authentication to support security hadoop cluster; repackaging software package to be hadoop cluster ready.

May 2007 – Oct 2014 · 7 yrs 5 mos · New York City Metropolitan Area

• Data platform for large scale distributed financial instrument pricing.
• Research and implement approaches for distributed process profiling and compute cost reduction.
• Edit tracking process, automatic report generation.
• Selected projects:
• Stock pricing infrastructure: developed the core dollar price components of stock pricing infrastructure. It extracts the common factors and defines a general pricing environments to allow various stock pricing model ( linear or non-linear ) to plugin and run on enterprise cloud.
• Large scale position attribute calculation infrastructure: developed the infrastructure to calculate financial instrument attributes (terabyte data daily), such as denomination, on data centers; employed timeout and multilevel retry mechanisms (immediately, delayed, and final) to address calculation failure caused by transient application exceptions, machine, network or other resource issues; provided tools to allow ad-hoc testing and issue investigation locally; developed task status check and rerun to ensure calculation completeness which directly impacts the federal stress-test results.
• Compute cost reduction via task batching: independently researched on the project for compute cost reduction. Proposed and evaluated effective batching approaches (split inputs to create tasks) to reduce total compute cost. Implemented and deployed lazy batching approach to reduce compute cost (average of 5%) by batching more computation units in a task as possible to minimize task management overhead, where time-insensitive task creation is delayed until accumulating enough computation units or reaching time limits.

Jun 2006 – Sep 2006 · 3 mos · New York City Metropolitan Area

• Develop an automatic failure monitor and detector in Perl.

Aug 2002 – Jul 2004 · 1 yr 11 mos · New York City Metropolitan Area

• Teach undergraduate C programming language.