May 2024 – Present · 2 yrs · Salt Lake City, Utah, United States · On-site

• Building a micro-degree to bring you from "Coder" → "Software Engineer" → "Cloud/DevOps Engineer" → "MLOps Engineer". Visit mlops-club.org!

Jan 2024 – Jul 2024 · 6 mos

• Lead in the end-to-end design, planning, and implementation of company-wide AI initiatives.

Jun 2023 – Jul 2024 · 1 yr 1 mo

• The ML Platform team exists to empower data scientists to rapidly develop and own AI services end to end by making the DevOps and software aspects trivial.
• I am accountable for
• the ML Platform team meeting our quarterly and annual OKRs
• having clear ML Platform architecture and ownership/support model
• platform outages
• interviewing; writing job descriptions
• acting as an individual contributor--but as a secondary priority to managing

Jan 2022 – Aug 2023 · 1 yr 7 mos

• Some projects:
• Championing a company-wide "constructs library". Like https://constructs.dev/, but just for BENlabs.
• This is a great balance of deploying infrastructure to AWS in a consistent, secure way, but still gives individuals and teams freedom to deviate for new architectures.
• Sidenote: we set a goal to have all of our engineers and many data scientists become "IaC-literate". It turned out to be doable. Great boost in speed & quality. Very exciting. Exploring LocalStack to make testing of IaC faster and easier. That's been a pitfall of developing IaC.
• Championing a project, boilerplate code tool (with projen, https://github.com/projen/projen). Goal: data scientists and engineers should be able to "launch" new experiments and products in minutes instead of weeks.
• I made these projects as open-source POCs before continuing them closed-source at BEN and digging deeper. POC for creating repos with IaC: https://www.youtube.com/watch?v=7TDrysCv5xg; POC of Python-projen library https://github.com/phitoduck/phito-projen
• Doing a POC of BentoML as opposed to FastAPI, Seldon Core, etc. for model serving. Gave a meetup talk on it at Utah's MLOps.community chapter. Being active in the BentoML community.
• Technically, I'm a co-organizer of the MLOps Utah meetup, but I need to contribute more to claim credit for that.
• I'm one of the "local experts" on Auth0, NewRelic, DNS, Pulumi, projen, AWS CDK, packaging, and BentoML.
• Supporting an MLFlow POC, mostly led by Jon Jones.
• Pushing hard for support plans: AWS Support, BentoML dedicated Slack channel, NewRelic support.
• Seeking ways to form relationships with AI teams at "sister tech companies", maybe with a mutual-NDA.
• Organized a hackathon with some co-workers and friends outside of work. We made a self-hostable, serverless Minecraft Platform-as-a-Service using AWS CDK. ("pip install awscdk-minecraft") https://github.com/mlops-club/awscdk-minecraft

Apr 2021 – Jan 2022 · 9 mos

• Write/extend tools to make DS models trivially easy to "deploy" as REST APIs and in batch jobs on AWS.
• Lead "hand-offs" of AI products from DS to DE.
• Research/implement MLOps practices: experiment tracking, shadow deployments, catching model drift, etc
• Worked with Steve Jackson and Daniel Clark to build an important JSON:API (web service) whose purpose I probably shouldn't say here. We've continued to use the framework we established here for new AI services. Now, heavily researching observability vendors for correlated metrics, logs, and traces (which trigger alerts).
• Established patterns for testing applications in docker. Currently developing CI framework to run a battery of tests against services *after* they have been deployed to the cloud, and then on a schedule thereafter.
• Working on a modeling project with DS. I enjoy this and the learnings should feed back into the ML platform tools.
• Did/doing heavy research into no-downtime deployments of REST APIs--particularly when the underlying transactional database schema needs to change with the REST API version.
• Write loads of infrastructure as code. Heavily researching pro's/con's of AWS CDK vs CDKtf vs Pulumi. Also, establishing standards for those.
• Wrote a CLI-tool wrapping Sphinx to generate versioned documentation for all DE/DS repos. The docs build and are published in CI. Docs can be viewed in PRs as well :)
• Extended a project template framework: DE/DS engineers can generate boilerplate Python projects, ready with CI, docs, Dockerfiles, packaging configuration, tests, semantic versioning for internal PyPI server, and other nice things. Every team should have this!
• Actively mentor.
• Actively participate in recruiting and interviewing. Went to career fairs at BYU and UVU. Holding an info session at BYU promoting BEN and offering career advice.
• Participate in our internal AI paper club. Presented on the transformer architecture.

Aug 2022 – Aug 2023 · 1 yr · Lehi, Utah, United States

• We offered Salesforce analytics via SaaS--as well as mentorship and training on best practices for tracking your sales process. This is ideal for groups who have not hired a full-time analyst but want affordable, high-quality visibility into their sales funnel.

May 2021 – Jul 2022 · 1 yr 2 mos

• Conduct technical interviews on behalf of client organizations. My goal is to make interviews an uplifting experience for the candidates－whatever the outcome.

Jul 2020 – Apr 2021 · 9 mos

• Wrote many data pipelines and support tools using Apache Airflow. I also "Docker-ized" Apache Airflow for our team, managed its deployment in Kubernetes, and worked with the AWS support team to resolve mysterious Airflow/Kubernetes issues.
• Researched "Infrastructure as Code", then led an initiative to standardize deployments of all new services using AWS CloudFormation templates; later leveraged that work to spin up all data engineering services in a staging environment (Terraform, CloudFormation, pretty diagrams)
• Led migration and rearchitecting of a REST API for machine learning inferences, which was consumed by our core product. I used my data science background to reduce the HTTP request latency from 2+ minutes to <20 seconds in the worst case (Flask, python, Airflow, DynamoDB)
• Led an initiative to create and standardize unit testing and CI/CD pipelines for all Python projects (Bitbucket/Bamboo, AWS Container Registry)
• Assisted in maintaining massive data ingest pipelines using Kafka and other tools to land data in S3, Snowflake, and curated relational/non-relational databases
• Made lifelong friends and awesome software with brilliant people

Aug 2019 – Jul 2020 · 11 mos

Aug 2019 – Aug 2019 · 0 mo · Provo, Utah Area

• Wrote code to develop data infrastructure including SQL and NoSQL databases, Kafka, RabbitMQ, Airflow, and many services within AWS

Apr 2019 – Aug 2019 · 4 mos · Provo, Utah Area

• Led trainings 2x/week for analytics team on Python, Docker, Linux, and AWS which enabled multiple team members to independently write and deploy scripts to production.
• Developed a customer churn model from Salesforce and customer usage data. Gathered data SQL and a 3rd-party JavaScript-based query language (JQL)
• Deployed Airflow on a VM to run data pipelines daily. Helped another engineer dockerize and deploy an R script with Airflow that sent alerted account managers of accounts they hadn't reached out to in over a week.
• Learned SQL and DOMO. Also learned to make BI dashboards with a lot of handholding from my manager.
• Lessons, looking back years later:
• (1) After I left, Airflow fell over and the analysts stopped using it. It was unrealistic to expect a team of traditional B.I. analysts to maintain such a software-engineering-oriented tool.
• Positive note: some Airflow pipelines proved valuable, and were reimplemented in simpler tools.
• (2) The customer churn model was not useful so it was abandoned. I think the sad reality is most A.I. projects are not worth building, and it's easy to get distracted to complexity.

Feb 2019 – Oct 2022 · 3 yrs 8 mos

• Building an AI startup “in the open”, using the project as a mentorship platform for people wanting to get experience in all sorts of engineering and engineering-adjacent roles.
• See the explainer video at https://www.rootski.io (looks awful on mobile 🙂)
• Learn about the project and how to contribute here: https://github.com/rootski-io/rootski. Hope to see you on Slack!

Dec 2018 – Apr 2019 · 4 mos · Provo, UT

• Worked with strategy team to identify and implement software solutions that eliminate bottlenecks in 2 million parts per month global supply chain
• Developed application for systems administration of U.S. and Chinese accounts (Python, Bash)

Dec 2017 – Apr 2018 · 4 mos · Provo, Utah Area

• Started a rewrite of the lab's landing page in AngularJS. Never finished, but was a valuable learning experience.

Sep 2017 – Dec 2017 · 3 mos · Provo, Utah Area

• Helped a few hundred beginner CS students with clean code and OOP fundamentals in C++

Jul 2015 – Jul 2017 · 2 yrs · Samara Region, Russian Federation

• Created an internal support program in 3 cities—training Russian church members to visit, teach, and care for each other
• Lead training meetings with 21 volunteers to adjust to a major legal policy change in 2016
• Served as a Russian translator for top church leaders during regional conferences