Note: This blog post describes collaborative work that had contributions from myself, Clark Gao, Matthew Mok and Karan Kumar (all at Instart Logic Inc.). The image above is borrowed from Paul Irish's talk slides with my hand drawn squiggly circle to highlight this blog post's focus.
Sometime last year when I started to dig deeper into web performance and how human end-users perceive the process of webpage loading, it quickly became clear that typical W3C standard metrics weren't sufficient. I also learned (painfully) that popular page-level metrics like onLoad could be easily gamed via scripting tricks. I became particularly fascinated with the problem of measuring above-the-fold webpage loading process - one that's almost a computer vision type of a problem (but my computer vision friends don't know about it yet). Couple of early discoveries kept me going:
I suspected that the histogram-based comparisons used in Speed Index's visual progress calculations could lead to some issues (see the examples in the slide deck below for a gory exploration of this aspect). I was annoyed by the fact that visual jitter (caused by layout instability, weird ad behavior or carousal elements etc.) wasn't being captured by histogram based calculations. I spent some time working out the math, and finding a way to address this issue in the computation of visual progress. By March/April 2015, I had arrived at a different formulation that relied on Structural Similarity (SSIM). Fortunately, I found colleagues (Karan at first, Clark and Matthew later on) who were also interested in this topic, so we kept taking stabs at it whenever we found time. I reached out to Pat and it started a collaboration that ended up leading to some cool insights. What we have developed is Perceptual Speed Index (PSI), a novel metric that measures the above-the-fold visual performance of a webpage, while accounting for the layout stability. The code is open-source, so you can go play with it if you like.
Some key take-aways:
Parting thoughts: Between SI and PSI, we have two complementary metrics that do a good job of informing whether above-the-fold loading process is fast and visually stable. Normalized differences between these two metrics will be interesting to play with. All the code is in public domain. If you want to see the data, send me a message, and I will post that too.
Looking ahead: I don't think the problem of measuring above-the-fold visual QoE (quality of experience) is fully solved by either Speed Index (SI) or Perceptual Speed Index (PSI) - but they are good starting points on that path. I feel that we are just scratching the surface of a very interesting visual perception problem. We will continue to explore this. Stay tuned on that front.
Here is a slide deck that captures our work, timelines, results etc. Feel free to leave me a comment or ask questions, if any.