It is Thanksgiving week for me so a smaller ML team weekly
-Benthos work is on hold. We’ve been experimenting with Benthos as a lightweight tool to stream model prediction scores to the larger event stream. It looks like it would work great, but we are putting the work on hold. The Data Engineering team is working on Flink, which would solve all the functionality we were thinking of using Benthos for. Flink isn’t ready yet, however, based on our timelines we can hold off having a streaming solution while the Data Engineering team gets Flink ready and then use that. If that doesn’t work we can always fall back to Benthos.
- We are deploying some brand new models into production as part of an agile development process, specifically, Revert Risk (language-agnostic prediction an edit is reverted) and Outlink Topic (language-agnostic prediction of an article’s topic). I’ll talk about these models more in the future, but for now, I wanted use them to highlight the improvements Lift Wing has created in model deployment.
Previously, deploying models like these on ORES would take a few days for each model. Okay, but lots of room for improvement. With Lift Wing, deploying these models takes less than an hour:
- Upload new model to Thanos Swift. (~10min)
- File a patch to deployment charts to update STORAGE_URI, wait for ML SRE +2 and merge (~10min if ML SRE is available)
- Deploy to staging (ml-staging-codfw) and test the model. (~10min)
- Deploy to production (ml-serve-eqiad &ml-serve-codfw) and test the model (~20min)