COMMERCE & EXPERIMENTATION
A/B testing
A/B testing compares two assigned versions of an experience to measure their effect on a defined outcome. A credible test uses a suitable comparison design, consistent measurement and a planned analysis.
A/B TESTING IN PRACTICE
- Define the testSpecify the versions, population and primary metric.
- Assign exposureUse comparable groups and stable tracking.
- Review the evidenceEvaluate uncertainty and downstream impact.
Use the agreed units, actual records and approved procedures for your operation.
A WORKED EXAMPLE
3.0% versus 3.3%. What changed?
Two checkout versions each receive 10,000 eligible sessions. Version A generates 300 orders and B generates 330. The observed conversion rates are 3.0% and 3.3%: a 0.3 percentage-point difference and a 10% relative increase. These numbers alone do not establish statistical reliability. The team also checks assignment, repeat visitors, test duration and whether the delivery promise offered by B can actually be fulfilled.
WHAT THIS MEANS FOR YOUR QUOTE
Agree the requirement before comparing quotes
Use the actual shipment, stock and service requirements. Record the billing unit, included work and responsibility for exceptions.
A LITTLE MORE CONTEXT
Questions about A/B testing
Must an A/B test change only one element?
No. It can compare bundles of changes, but that limits what can be inferred about each individual element.
Is a 50/50 split mandatory?
No. Allocation should suit the experiment design, risk and required precision.
Can we stop as soon as B looks better?
Repeatedly checking and stopping opportunistically can distort conclusions. Use a planned analysis appropriate to the design.
Source & example notes
Definition and operational example prepared by 3PL Hub. Examples describe hypothetical requirements; verify the actual records and agreed procedures for your operation.
Quantities, costs and scenarios are illustrative, not operating targets, approved handling instructions or guarantees. Source guidance reviewed 22 October 2026.
Last updated: · 3PL Hub
