What's the difference between an A/B test and a growth experiment?
An A/B test splits traffic between two versions of a page or feature. A growth experiment tests any change to how you find or win customers, like outreach, events, partners or pricing, and measures it on a business number. Every A/B test is a growth experiment. Most growth experiments aren't A/B tests.
By Nomiki Petrolla, founder of Theanna
What is an A/B test?
An A/B test shows two versions of the same thing to two groups of people at the same time, then compares them. Version A might be your current sign-up page, and version B the same page with a shorter form. A testing tool splits the traffic at random, so the only difference between the groups is the change.
A/B tests are strongest for pages, emails and product features, where lots of people see the same thing. Tools like GrowthBook, Statsig, Optimizely and PostHog are built to run them.
What is a growth experiment?
A growth experiment is a short test of one change to how you find or win customers. It has a starting number, an end date and a clear result: keep it or stop it.
The change can be anything: a new first line in your cold emails, a partner co-hosting an event, a price change, or a new onboarding step. It's measured on a number the business lives on, like demos booked, sign-ups or revenue, read from the tools you already use.
How are they different?
What it tests
- A/B test:
- Two versions of one page, email or feature
- Growth experiment:
- Any change to how you find or win customers
Who sees what
- A/B test:
- Traffic split at random between versions
- Growth experiment:
- Often everyone, or a list you choose
What it's measured on
- A/B test:
- A click or conversion on that page
- Growth experiment:
- A business number: demos, sign-ups, revenue
Traffic it needs
- A/B test:
- Enough visitors for both versions to show a difference
- Growth experiment:
- None: 20 messages can be a test
Who usually runs it
- A/B test:
- Engineers, product or growth
- Growth experiment:
- Anyone on the team
Tools
- A/B test:
- GrowthBook, Statsig, Optimizely, PostHog
- Growth experiment:
- Theanna, Growth Method, a spreadsheet
Can you A/B test without an A/B testing tool?
Yes, and small teams do it all the time. You write two versions, send each to a similar group, and count what comes back. No traffic split, no code. It's a growth experiment with an A and a B.
Here's how Nomiki Petrolla, Theanna's founder, tests the first message a team sends to buyers:
- 01
Write 2 versions, change one thing
The problem you name, or the ask (a 20 minute call vs a pilot), or the length. Not all three.
- 02
Send each version to 20 people who could sign the deal
Same kind of company, same title, the person who owns the budget.
- 03
Send them the same days, same time of day
So the timing isn't what makes the difference.
- 04
Wait 3 days and count the right replies
Count replies that turn into a call with the decision maker, not "thanks, not now."
- 05
Keep the winner and test the next thing
The winner becomes your new normal. Then test the next one change against it.
“Most small teams don't need another channel. They need a better first sentence on the channel they already have.”
What does a growth experiment look like in practice?
Nomiki ran one on Theanna's own webinar funnel. She redesigned the landing page to cut friction and simplified the ad copy. Cost per lead dropped from $22 to $9, and registrations went from 148 to 653.
Then only 9.3% of people showed up, against 32% before, and it looked like a failure. But the closes kept coming. Judged on the page alone, the test failed. Judged on revenue, it worked. That's the difference between measuring a click and measuring the business.
“I was wrong. The data told a completely different story than what I felt in the moment.”
When does a small team need an A/B testing tool?
- You're testing two versions of a page, email or product screen.
- Enough people see it each week for both versions to show a real difference. A sample size calculator tells you how many.
- An engineer can set up the split, or the tool is already in your product.
- If your tests are outreach, events, partners or pricing, you don't need one. You need a place to track every experiment against the same number.
Which tool should you use for which?
Use an A/B testing tool for page and feature tests: GrowthBook, Statsig, Optimizely or PostHog. Use a growth experiment tool for everything else, and to see every test against your company's number in one place. Many teams use both.
Theanna is a growth experiment tool for small teams: set one company target, pick up to 3 goals, and run the experiments under each, with results read from Stripe, Google Analytics, Calendly and the rest of your tools. For the full list, see the best growth experiment tools for small teams.
Questions teams ask
Is every growth experiment an A/B test?
No. An A/B test is one kind of growth experiment: two versions shown to two random groups at once. Most growth experiments change something for everyone, like a new outreach message or a price, and compare the number before and after.
Do I need an A/B testing tool to run growth experiments?
Only for page and feature tests with enough traffic to split. Outreach, events, partner and pricing tests don't need one. You need a starting number, an end date and a way to read the result from your own tools.
Is Theanna an A/B testing tool?
No. Theanna doesn't split traffic on your site. It tracks every kind of growth experiment against your company's number and reads the results from your tools. Teams that A/B test their pages use an A/B tool next to it.
How long should a growth experiment run?
Long enough for the number to move: often 1 to 3 weeks for outreach and content, longer for pricing and retention. Set the end date before you start, so you don't call it early.
What should a small team measure a growth experiment on?
One number the business lives on, like demos booked, sign-ups or revenue. Not likes or page views. If the test doesn't move the number, it didn't work.