Migrating a machine learning pipeline to Kubernetes

Zach Lipp

he/him

Senior Software Engineer, Lumere

19 February 2020

Problem overview

We want to help our team of expert medical researchers classify hospital purchases

Problem overview

You don’t need to be an expert for some of these

Enter machine learning!

We can use the text descriptions as inputs to classification models. This is called short text classification.

Machine learning deployment

Results

• Our reconfigured pipeline is faster end-to-end
• We no longer require manual modeling runs
• Improved monitoring and observability
• Models are written to disk
• We parallelized model training, predicting, and preprocessing
• We distribute and schedule work with Dask

Configuration

• Two Deployments (Dask workers, scheduler)
• Three CronJobs (training, predicting, refreshing training data)

Lessons learned

1. Know your APIs

• scikit-learn has great functionality for building pipelines
  • Pipeline
  • FunctionTransformer
  • ColumnTransformer
• pandas can save your database some munging
  • DataFrame.groupby
  • .to_sql()

Lessons learned

2. Treat ML code like application code

From Hidden Technical Debt in Machine Learning Systems, NIPS 2015

Lessons learned

2. Treat ML code like application code

By adapting old code to meet our new data model and make use of pandas over SQL, we avoided some costly joins and aggregations, leading to a 5-6 orders of magnitude speedup

Lessons learned

3. Avoid premature optimization

Lessons learned

1. Know your APIs
2. Treat ML code like application code
3. Avoid premature optimization

Fin