You might have heard of Chrome Variations, sometimes known as Chrome Field Trials, or even the internal codename, Chrome Finch.
These are all names for the same thing: a mechanism to test new features or changes to the Chrome browser or to ChromeOS, the Chrome operating system that runs on Chromebooks.
What are Chrome Variations?
Chrome Variations enable Chrome to activate a new feature, deactivate a feature, or try a feature change for a subset of users.
Not everything in Chrome is introduced using Chrome Variations. However, the Chrome team can use Chrome Variations whenever we need. Chrome Variations can be used to:
- Allow Chrome engineers to be e specially careful about making a certain change. They may see a risk to browser performance or have other concerns.
- Validate a hypothesis about a change or an update. For example, for a percentage of users in a Chrome Variations group, we modified the QUIC networking protocol parameters to make Chrome faster for users' real network conditions.
- Control activation of an origin trial feature. By default, origin trials are enabled on all pages that provide a valid trial token. But in some cases, a feature may not be available to certain users, even if they visit a page that provides a valid trial token.
How do Chrome Variations work?
Every 30 minutes on desktop or mobile, or each time you start Chrome, the browser makes a request to the Chrome backend to get the Chrome Variations configuration file, which is known as the variations seed. There is a dedicated server to provide Chrome Variations seeds. Chrome makes an HTTPS request to the server, and the server responds with a delta-compressed seed.
When Chrome on your device contacts requests the seed file, Chrome provides the version number and the operating system it's running on. The file returned by the Chrome Variations backend is used to turn on features, turn off features, or specify variations in a feature.
Chrome uses the data in the seed file, and a locally stored randomization seed, to randomly assign the browser to a variation group. Your browser, on your device, keeps track of which variation group it belongs to.
Feature rollout out and Chrome channels
One of the most important uses for Chrome Variations is to gradually phase in changes or new features to a percentage of Chrome clients. That's crucial for a complex application like Chrome, which has billions of users, with thousands of types of devices on multiple platforms, using dozens of different languages for millions of different apps and websites.
Chrome gets valuable feedback from usage and testing in Chrome Canary, Dev, and Beta. We can use Chrome Variations to activate or deactivate features within any of these release channels. However, those channels are primarily used by developers and other specialists.
Chrome Stable users interact with Chrome differently, and in much larger numbers, so we always need to validate on the Stable channel. This allows us to address any problems encountered in normal browser usage. Chrome engineers can't always predict how real users will respond to changes and new features, at scale.
Chrome Variations make it possible to check usage and metrics for Chrome Stable. By measuring the impact of changes in Chrome Stable, we can ship the best features possible and build a better browser, even when the trade-offs are complex.
When do Chrome engineers use Chrome Variations?
There are three main reasons that Chrome needs Chrome Variations.
Activate a new feature
Using Chrome Variations to provide control over activation of a new feature is particularly useful for anything that might be more risky in some way, or that might affect performance.
Chrome Variations enable us to provide a new feature to a subset of users. Chrome engineers can then check for differences in performance, or look at other types of feedback from the variation group.
Deactivate a feature
Chrome Variations can be used as a safety mechanism to turn off a feature, although this happens rarely.
For example, a new networking feature might become subject to a denial-of-service attack. Chrome Variations can be used to quickly turn that feature, as the Chrome Variations configuration is downloaded every 30 minutes and settings are activated every time you restart Chrome.
By contrast, updating Chrome and waiting for the new version to be propagated to billions of users would be much slower.
Try changes to a feature
Chrome Variations can be used to validate changes and updates. For example, we could fine-tune the difficulty for the Chrome Offline Dino game to make the interaction feel more fun.
Chrome Variations can also be used to study the long-term effects of new features. This is done by using the Chrome Variations mechanism to hold back features for a small proportion of users, maybe 1% or less. A holdback group can be useful to check for changes and side effects that only become visible over time.
A holdback group can be particularly important for user interface changes, where users are likely to interact with a feature when it's new, but may behave differently in the long term.
For example, we launched a photo picker in Chrome, but the feature was temporarily held back from a small percentage of users, to compare outcomes. We shipped the new Android Photo Picker to a high proportion of our users, and initially saw a significant increase in the number of images that group shared on the Web. However, in the six months that we had the holdback variation, we saw the lift in usage decrease significantly. This was because availability of the new photo picker encouraged sites to add the accept attribute to file input elements—which led to a better experience for all users.
How are Chrome Variations used for validation?
If the setting to help improve Chrome's features and performance is enabled
(chrome://settings/syncSetup?search=improve) then Chrome can automatically
monitor and send metrics to the Chrome backend with User Metrics Analysis (UMA).
You can see examples of Chrome's anonymous usage statistics at chromestatus.com/metrics, such as the percentage of page loads that use a CSS property or an HTML or JavaScript feature.
Chrome Variations are particularly useful for comparing statistics from one group of users against another. For example, a feature can be turned on for some users and not others, and then Chrome can compare metrics for each group. These metrics might include memory usage, page load times, or the usage of a browser feature. This allows Chrome engineers to compare performance or other metrics between users that have a feature turned on, users that have the feature turned off, or users with different feature variations.
Field trials
Each set of variations for a feature is called a study or field trial, and each has a fixed duration. Once a study is complete, any users who were included in a non-default behavior group get the default Chrome setting for the feature: either enabled or disabled.
Most features that can be controlled by Chrome Variations correspond to a flag
that can be set from the chrome://flags page. Alternatively, if you
run Chrome from the command line,
you can use --enable-features and --disable-features flags to configure
Chrome Variations.
The --reset-variation-state
flag forces reset of the Chrome Variations state for the current client. This
means that the client may move to different Chrome Variations groups.
Chrome Variations for enterprise
For enterprise customers, Chrome also provides the Chrome Variations policy to manage Chrome Variations. We recommend leaving Chrome Variations enabled, so that Chrome can quickly provide critical security fixes.