A/B testing SMS messages works if you change one thing, keep both versions the same segment length, and judge the result on replies, clicks and opt-outs together. Most bad SMS tests fail before the first text goes out: two variables changed at once, uneven segment counts, or a winner called after 40 sends.

This post is for product and growth teams testing texts to users who've agreed to receive them, such as trial reminders, onboarding nudges and account alerts. It won't give you benchmark numbers, because we don't have honest ones to give. It gives you a design that makes your own numbers trustworthy.

Decide what you're testing and why

Write one sentence before you build anything: "We believe changing X will move Y, and we'll know because Z." Then pick a single X.

Good single variables:

  • The first line (a question versus a statement).
  • The call to action ("Reply YES" versus a link).
  • The send time, inside allowed hours.
  • Including the user's first name or not.

Bad tests change the tone, the link and the send time in one go. If B wins, you've learned nothing you can reuse.

Control the segment count

Segments are billed and metered, not messages. A 160-character GSM-7 text is one segment, longer ones split into 153-character parts, and one emoji can push the whole message to the 70-character unicode limit. If variant B has an emoji and variant A doesn't, you're testing cost and delivery behavior as well as copy. Check both variants with a counter first. The segment counter in code post shows how.

Also remember that Ready adds "Reply STOP to opt out" and the business name to the first text a contact gets, unless the account turns it off. That footer counts toward the segment length of a first text in both variants.

Pick the metrics, and rank them

MetricWhat it tells youWatch out for
Reply rateThe message invited a responseReplies include "stop" and complaints
Click rateThe link was worth tappingLink previews and scanners can inflate clicks
Product actionThe user did the thingNeeds a clean attribution window
Opt-out rateThe message annoyed peopleThe guardrail, not a goal

Treat opt-out rate as a guardrail. A variant that lifts clicks and doubles opt-outs is a loss, because every opt-out is a user you can't text again. Delivery status isn't a read signal either; the delivery receipt versus read gap post explains why.

Be honest about sample size

We won't quote a magic number, because the right size depends on your baseline rate and the lift you want to detect. The practical rules:

  1. Small lists can't detect small differences. With a few hundred users per arm, only a large gap means anything.
  2. Decide the sample size and the end date before you start. Don't stop the moment one arm looks ahead.
  3. Run one test at a time on the same audience, so results don't mix.
  4. If neither variant clearly wins, that's a result. Keep the simpler copy.

An assumed example: 600 eligible users, split 300 and 300. If A gets 30 clicks and B gets 36, that gap is small enough to be noise. Don't ship B on that alone.

Randomize, and keep a holdout

Assign users to A, B or a holdout by a stable random split, such as a hash of the user ID, not by signup date or alphabet. The holdout gets no text. It answers the question your A and B can't: whether the text beat doing nothing at all. For transactional-leaning messages like trial expiry, this matters, since some users would have acted anyway.

Also check that your audience is consented, and that quiet hours apply to every arm. An API text outside the window isn't sent unless the window opens within 6 hours, so an unlucky time zone mix can make one arm quietly smaller than the other.

Log what you need to read the result

Tag each send with its variant in your own database and join it to replies, clicks and opt-outs. Pull opt-outs from your own system or check GET /v1/opt-outs/:phone before reusing a list; see the opt-out management post. For trial-style messages, our trial expiry template pack gives you starting copy for the variants.

Wrapping up

Pick one variable, match the segment length, hold some users out, and let opt-outs veto a "winner." That's the whole method. When you're ready to send through an API, the messages docs have the request shape, and you can sign up to try it.

Frequently asked questions

How many people do I need for an SMS A/B test?

It depends on your baseline and the lift you're trying to detect, so no single number is honest. Smaller lists only reveal big differences, so fix your sample size and end date up front.

What should I measure in an SMS test?

Reply rate, click rate and the product action you actually want, with opt-out rate as a guardrail. A variant that wins on clicks but raises opt-outs shouldn't ship.

Should I test send time or copy?

Either, but not both at once. Test one variable per run so you know what caused the change.

Do emojis affect an SMS test?

Yes. An emoji can switch the message to unicode encoding and raise the segment count, so it changes cost and length as well as tone.