EXPLAINER.md

Client-Side A/B testing

Client side A/B testing refers to the method of performing experimentation related changes to a web application in the browser — sometimes without integrating code changes to the actual application source code. This is popular in the industry as the method usually helps to cut down resources required for experimentation. Our objective is to explore ways of conducting the same outcome, without the performance penalties it comes with today.

Overview

The key to the approach being outlined here is considering control as the base document, and expressing each variant as a series of transformations being applied onto control.

This is analogous to how client side A/B testing is conducted today — except we want to improve it further and make it more performant. How do we do that?

1. Standardize the serialization of changes to a common schema.

2. Whenever possible, optimize for applying a transformation where it makes the best sense (pre-UA, or on-UA). This helps to:

   a. reduce the bytes transferred, b. potentially improve cache hit rates at serving c. reduce computation needs on each client.

3. Apply transformations in a performant way, without blocking rendering of the page — ideally without any performance metric degradation.

Standardizing the representation of transformations

We could represent the transformations required for reaching a variant as a serialization of an ordered set of operations as follows:

[
  [flags, selector, operation, ...payload],
  [flags, selector, operation, ...payload],
  ...
]

where each field can be defined as follows

flags	A bit field that indicates the type of transformation. For an initial version, we can support the following flags: 0x1 PRE_UA Transform can be applied on the server, or prior to UA parsing. It could be done at a CDN/Edge, or at the Origin server itself. Most of the static document transformations would use this flag. 0x2 ON_UA Transform can be applied on the User Agent/the browser. Unless combined with the ONCE flag below, the transformation is expected to be continuous — i.e., it is applicable any time during the web application lifecycle within the UA. 0x4 ONCE Transform needs to be applied only once, after which it can be discarded to free up memory and computational resources.
selector	A CSS selector that targets the `HTMLElement` for transformation. Depending on the target platform and capability, only a subset of CSS selectors might be applicable here (and that needs to be documented and revisioned as the support changes).
operation	A numeric value indicating the operation to be performed. For a list of operations and their supported parameters, see the section "Operations" below. The operations are expected to be idempotent, i.e., repeated applications of the operations should be possible and not have unintended side effects.
payload	A variable number of arguments to support the operation. Should follow the specification of the operation used.

Operations

Operation	Code	Description	Arguments
OP_CUSTOM_JS	0	Executes a custom Javascript block of code against the element.	code : Javascript code serialized as a string. The applicator code will call this code as a function ($) => code, $ referring to the element selected.
OP_INSERT_BEFORE	1	Inserts content right before the element.	content: HTML markup
OP_INSERT_AFTER	2	Inserts content right after the element.	content: HTML markup
OP_PREPEND	3	Inserts content right after the start tag of the element.	content: HTML markup
OP_APPEND	4	Inserts content right before the end tag of the element.	content: HTML markup
OP_REPLACE	5	Replaces the element with the provided content.	content: HTML markup
OP_SET_INNERHTML	6	Replaces the content of the element with provided content.	content: HTML markup
OP_REMOVE	7	Removes the element and its children	none
OP_SET_ATTRIBUTE	8	Sets an attribute’s value on the element.	name: Name of the attribute value: Value to be set on the attribute.
OP_REDIRECT	9	Redirect the user to a different page or URL. During `PRE_UA` phase, this will perform an HTTP redirect and during `ON_UA`, this will result in client side redirection. Additional arguments supplied can select the response code during PRE_UA phase.	URL: URL to redirect the page to. code: HTTP code to use for redirection during PRE_UA phase.

Components of a solution, and their roles

1. A PRE_UA component — responsible for applying static markup transformations, and injecting the necessary ON_UA components into the document.
2. An ON_UA component — responsible for client-side transformations.
3. The set serialized transformations.

PRE_UA components

A PRE_UA component is responsible for applying the transformations flagged as PRE_UA, i.e., the transforms that make best sense to be applied before User-Agent. This could be done at:

1. the Origin itself, or
2. an Edge component at the Origin (like a web server plugin, or a proxy), or
3. a CDN compute node that fronts the Origin as a proxy, or
4. an implementation inside the UA/Browser prior to parsing the document (this has a latency impact).

The important aspect is that the PRE_UA transformations are best done at a step prior to UA’s parsing.

PRE_UA component has the following responsibilities:

• Apply all of the PRE_UA transformations for the selected experiment, onto the response.
• Collect the remaining ON_UA transformations if any, and serialize them along with the client-side transform applicator code. Inject this into the HEAD of the document.

ON_UA components

The client-side/ON_UA components are responsible for applying ON_UA transformations as necessary, and consists of two parts:

1. The remaining ON_UA transformations from the experiment configuration.
2. The client side transformation applicator code.

Client-side transform applicator

The applicator component injected into the HEAD of the document has the following parts to it:

• A MutationObserver client code that listens for DOM changes.
• The listening code looks for DOM changes that match the ON_UA selectors, as (1) the browser is parsing the document, or (2) the client side Javascript is making changes to the DOM.
• One or more identified DOM mutations that match the selectors are queued and deferred for processing until the next repaint, via a requestAnimationFrame callback.
• The queue is processed, applying each ON_UA transformation as needed — via running the matching DOMElement through the supplied transformation function.

An example client-side transform applicator

A simple implementation could be as follows:

(function applyTransformations(transformations) {
  const transform = (target) => {
    if (target.nodeType !== 1) return;
    for (const [flags, selector, transform] of transformations) {
      if (!(flags & ON_UA) || !transform) continue;
      const node = target.querySelector(selector);
      if (!node) continue;
      try {
        transform(node);
      } catch (e) { /* report back transform error */ }
    }
  }

  const queue = new Set();
  const processMutationsOnPaint = () => {
    for (const target of queue) {
      transform(target);
      queue.delete(target);
    }
  }

  var observer = new MutationObserver(function(mutations){
    for (const {target} of mutations) {
      if (!queue.size) requestAnimationFrame(processMutationsOnPaint);
      queue.add(target);
    }
  });

  return observer.observe(document.documentElement, {
    childList: true,
    subtree: true
  });
})([ /* Array of transformations */ ]);

The code example above doesn’t account for any selector performance optimizations, or error reporting.

This block results in approximately 365 bytes of minified code when processed via terser, and forms the Client-side Transform Applicator.

References

• WebPerfWG Open meeting notes 2021-02-04
• A/B Testing at the Edge with Servers Workers
• Performant A/B Testing with Cloudflare Workers
• A History of HTML Parsing at Cloudflare: Part 1
• The Case Against Anti-Flicker Snippets
• How CSS opacity animations can delay the Largest Contentful Paint