What’s better than free? Speed
I recently watched The Playlist on Netflix and was heavily inspired by Spotify on Daniel Ek’s obsession in creating the best product that loads instantly.
The human brain conceives anything that happens in less than 200ms as “instant”, and Daniel from Spotify wanted nothing short of that.
At Mida, we share a similar obsession; to develop the fastest A/B testing platform on the market that won't compromise the speed of your website. We want to ensure you won't need to sacrifice performance in order to optimize conversions.
So how do we build something better than free? The answer was speed. Imagine an A/B testing platform that loads instantly.
Fast, Faster, Fastest
While we're proud of Mida's performance, which is already remarkably quick according to our previous speed test article, we believe we can make it even faster.
We discovered that our script issues two additional HTTP requests, each one adding an extra 100ms to the loading speed. This means we can effortlessly shave 100-200ms off our script load time with just a few tweaks -- such as removing extraneous requests and enhancing some conditional logic on our backend servers.
After closely examining our 20kb small script, we singled out several areas ripe for enhancement. We spent an evening refining these elements and then rerunning the test on Debug Bear.
And the results?
We have successfully brought the numbers (LCP and STTV) down reducing from 636ms to 490ms and 474ms to 333ms.
Updated Desktop Benchmark
This means we're now approximately 44% faster than our closest competitor. Super awesome!
However, upon deeper analysis, I was puzzled by an anomaly. Our script only seems to take 94ms to load, yet the Start Time to Variant (STTV) shows as 333ms. Why is this?
In case you're wondering, the formula for STTV is as follows:
Largest Contentful Paint - Script Start Time = STTV
This discrepancy led me to question whether STTV might be an inaccurate measurement.
Then, it strikes me. Could it be that it’s the website's inherent loading speed that’s impeding the script, rather than the script itself slowing down the site?
Revised Methodology
To test this theory, I removed all external script and ran the base site on Debug Bear. This revealed that the base site LCP is already at 472ms, indicating that no amount of optimization can take the LCP below this time.
Accordingly, we'll use the base LCP of 472ms as a constant baseline and examine the increase post the installation of each A/B testing tool to determine the potential speed reduction on the site.
By calculating the difference between the base LCP timing and the changes brought about by the installation of different A/B testing tools, we can gain a more accurate insight into speed performance.
The differences of base speed LCP and A/B testing installed LCP is as below:
We are not 44% but over 1000% faster than the next vendor on the list.
It may seems too good to be true but if we are factoring into the uncompressed file size then it would all make perfect sense since everything is correlated:
- Mida (ONE script): 20.9kb size vs 18ms speed
- VWO (MULTIPLE scripts): 250kb+ size vs 253ms speed
- Convert (ONE script): 200kb size vs 758ms speed
For every 1kb of uncompressed file size, we can expect 1 milliseconds of additional loading time which doesn’t sound like a big deal on its own. But, if your script is 200kb+ huge, then you’re expecting to add a minimum of 200ms loading speed to your site.
For every 1kb of uncompressed file size, we can anticipate 1 millisecond of additional load time. While this might not seem like a big deal in isolation, when your script is 200kb and above then you’re expecting to add a minimum of 200ms loading speed to your site.
Patrick Meenan:
“Larger JavaScript files also take longer to parse and run (1ms for every 1KB of uncompressed JS is a reasonable estimate). All else being equal, if you have the same amount of JS in a lot of files it will take much longer to load than if the same amount of JS was in a single file.”
While we're excited about our speed, we're still working on further optimizing our user interface to provide a more seamless experience for our users.
---
So yeah, this article may be a little bit biased in favor to Mida. Take it with a pinch of salt and give it a try yourself.
.svg)
