With the latest Windows 10 Insider release, build 18282, Microsoft is continuing a program of “A /B” testing for some features, this time for access to “window snip” a feature of the new Snip & Sketch that allows users to take a screenshot of an open window with just one click. Microsoft has long used A/B testing, the process of providing different sets of features to different groups for testing, but it wasn’t until the company first started testing Sets, back in November of last year, that it publicly made A/B testing a part of the Windows Insider program.
Since then, there have been flurries back and forth about how Insiders accepted this A/B testing, when a good portion of Insiders have indicated that they joined the program not only for the good of Windows, but also (and perhaps primarily) to access the latest features. Here’s the results of a poll we took right after the announcement last November:
Consternation has begun anew with the latest Windows Insider build 18282, as only half of Fast Ring Insiders have access to window snip, except if they install a tool built by Rafael Rivera, which allows users to turn on access to some features they weren’t initially offered. So far, almost 20,000 Insiders have downloaded the tool, and that’s not sitting well with some Microsoft employees:
Don’t do this. Please stop.
— Brandon Paddock (@BrandonLive) November 15, 2018
Brandon’s contention is that bypassing the A/B test messes with Microsoft’s desired telemetry, but for those Insiders who joined the program to get access to the latest bits, being told they’re on the outside looking in hasn’t sat well.
On the Windows Insider home page, the first thing users see is “Be the first to access new Windows features”:
And in reading through the Program Agreement, which is prefaced with “Registering to become a Windows Insider means you get early access to upcoming Windows features before they are publicly released,” there’s nothing about being subject to A/B testing. Of course, there’s nothing that explicitly guarantees access to *all* the latest features, either, but the implication clearly is “join the program, be the first to access new features.”
While A/B testing is useful, and while Rafael’s tool isn’t an official part of the program, it’s far from the “reverse engineering of the source code” that the Program Agreement restricts. As I said to Brandon Paddock on Twitter (and a bit of a caveat I’ve known both Brandon and Rafael for over a decade and consider them both to be good and dedicated people), while A/B testing of Insiders who signed up to get the latest bits may be useful, it certainly isn’t the only way to handle testing scenarios, and actually goes pretty far in failing to deliver what the Insider program promises, that is, early access to the latest Windows features.
Fortunately for some Insiders, and perhaps unfortunately for some Microsoft tests, the promised desire to “be the first to access new features” overrides the clean telemetry that is simply never going to be achieved with a wide open program like the Windows Insiders. The company is going to have to accept a certain amount of fudging of the data, especially as they’ve used early access as an enticement to gain the telemetry to begin with. This isn’t, and never was, a controlled experiment.
