Note: The title of this article is a play on the term "Rube Goldberg machine". Acccording to Wikipedia, a Rube Goldberg machine is a contraption, invention, device, or apparatus that is deliberately over-engineered to perform a simple task in a complicated fashion, generally including a chain reaction. Keep that in your mind.
Everyone (that cares about ML) knows about supervised / unsupervised / semi-supervised learning pipelines. I have now come across an entirely new class of ML pipelines that I shall call "Rube Goldberg Machine Learning" pipelines. Before I go on and explain what I mean, let me provide some context.
Last week, I attended Velocity conference in Santa Clara, CA. For those of you who are unfamiliar, Velocity conference is a popular enterprise-oriented (non-academic) conference focused on web performance and DevOps topics. I was very excited to see a machine learning talk in the program: "Using machine learning to determine drivers of bounce and conversion". Apparently it was the first ML talk ever at this venue (for a field that produces insane amount of data, I don't know why ML doesn't show up more often in Velocity). So, yay! Given what I know about the prior work of Pat Meenan and Tammy Everts, I had high hopes for the talk and the potential findings. Neither of them are ML practitioners, but both of them have done stellar work in web performance community before. Unfortunately my excitement quickly dissipated within the first few slides (click the link to the slides if you are curious). Several web performance folks told me informally that the conclusions of the talk didn't seem right, because the conclusions appeared to go against the conventional wisdom in the web performance field. I have no problem with going against conventional wisdom; sometimes it is nice to correct long-held misconceptions if there's good evidence. My disappointment mainly stems from the misuse of ML models in this talk. Instead of simply venting, let me break down the good/bad/ugly aspects of this talk:
I could go on further about other nitty-gritty, but let me stop here and summarize. I saw a first-ever ML talk at Velocity that didn't really teach me anything about web performance despite analyzing a million sessions worth of data. The talk generated a lot of buzz on the basis of disrupting conventional wisdom while the models are highly questionable. I think what allowed the talk to fly through is the use of over-engineered and overly-complicated ML pipelines that aren't interpretable. This is the class of ML that I am going to call "Rube Goldberg Machine Learning" from now on.
As a recent ICML talk title says: "Friends Don’t Let Friends Deploy Models They Don’t Understand".