Short answer
Real proof in a Shopify agency case study has five parts: a named or confirmable client, a starting number with a date, the metric that changed, the time it took, and what the agency did to cause it. Trust revenue per visitor and conversion rate over raw revenue, ask how any split test was run, and ask to speak to the client.
Key takeaways
- A big revenue number with no baseline, no dates and no explanation of what the agency did is a story, not evidence.
- Raw revenue growth mixes the agency’s work with ad spend, seasonality and product launches. Ask what else changed in the same period.
- A split-test win means little unless the sample size was set before the test started.
- Testimonials show that a client was happy. Unless the advertiser can show those results are typical, FTC guidance says the ad must make clear what results people can generally expect.
What does a trustworthy Shopify agency case study include?
Most agency case studies are marketing, written to impress. Your job is to separate the part that is evidence from the part that is packaging. Read each one against this checklist:
| Element | What good looks like | What weak looks like |
|---|---|---|
| Client | Named, or described closely enough that you could confirm it | “A leading DTC brand” |
| Baseline | A starting number with a month and year | “Struggling with conversions” |
| Metric | One clearly defined number: revenue per visitor, conversion rate, average order value | “Growth” or “ROI” with no definition |
| Time period | A start date and an end date | “In record time” |
| Cause | The specific changes the agency made, in order | “Our proven system” |
| Other factors | What else changed: ad spend, pricing, new products | Not mentioned |
| Confirmation | A client you can call, a video, a public review | None offered |
A case study does not need all seven to be useful. But one that has only the client name and a headline number gives you nothing to check.
Book a call
A free 30-minute call about your store. Shopify Plus Partner, rated 5.0 from 8 reviews on Shopify’s Partner Directory.
Which numbers in a case study should you trust?
Some metrics move mainly because of the agency’s work, while others move for many reasons at once.
Raw revenue is the weakest proof. A store that doubled revenue in a year could have doubled its ad budget, launched a hit product or simply grown with its market. If a case study leads with revenue, ask what happened to traffic and ad spend over the same period.
Conversion rate is better, if it is defined. Shopify defines online store conversion rate as the share of online store visits (sessions) that resulted in a sale: sessions that completed checkout divided by sessions (Shopify developer documentation). Shopify also tracks a separate checkout conversion rate, which only counts visits that reached checkout. A case study that says “conversion went up” without saying which rate is leaving out the most important detail. Because checkout conversion rate only counts visitors who already reached checkout, it is always at least as high as store conversion rate and usually far higher, so mixing them up can make an ordinary result look like a big one.
Revenue per visitor is often the strongest single number. It is revenue divided by sessions, so it captures both conversion rate and order value, and it does not rise just because traffic grew. If an agency reports it with a baseline and dates, that is a good sign.
Tip
Percentages need their starting point. A “200% lift” from a tiny base and a “15% lift” on a large store are very different results. Always ask for the absolute numbers behind any percentage.
If you want a reference point for what normal conversion rates look like, our post on ecommerce conversion rate benchmarks sets out the ranges by industry and device.
How do you check a split-test result in a case study?
Agencies that run conversion programs often point to split tests (A/B tests) as proof. A test result can be strong evidence or close to meaningless, depending on how it was run.
Watch out
The most common problem is stopping a test the moment it looks like a winner. Checking results again and again and stopping at the first significant reading raises the rate of false positives, because the statistics behind the test assume the sample size was fixed in advance (“How Not To Run an A/B Test”). The fix is to decide on a sample size before starting and wait until the experiment is over.
Questions to ask about any test in a case study:
- How was the sample size decided, and was it fixed before the test started?
- How long did the test run? Ask whether it covered full weeks, since weekday and weekend shoppers behave differently.
- What metric decided the winner? Revenue per visitor and conversion rate can point in different directions.
- What happened after launch? A test that won in the experiment and then faded once rolled out to everyone is worth knowing about.
- How many tests lost? An agency that only shows winners is showing a highlight reel. Losing tests are normal, and a good partner records them.
What do testimonials and reviews actually prove?
Testimonials tell you a client was pleased. They are weaker evidence of what you will get, and US advertising rules say as much.
Worth knowing
The FTC’s guidance on endorsements says that when an endorser reports exceptional results and the advertiser lacks proof those results are typical, the ad “must make clear to the audience what the generally expected results” are (FTC, Endorsement Guides FAQ). The same guidance says connections between the endorser and the business that consumers would not expect, such as payment or something else of value, should be disclosed clearly.
In August 2024 the FTC also announced a final rule targeting fake reviews. Among the practices it lists are fake or false testimonials, buying positive or negative reviews, insider reviews and testimonials, company-controlled review websites and review suppression (FTC press release).
What that means for you as a buyer:
- Treat a testimonial as a lead, not a result. Ask the agency whether you can speak to that client.
- Prefer reviews the agency does not control. On Shopify’s Partner Directory, a merchant only gets a review request if the Partner was added as a collaborator and that access was active for at least a week (Shopify Help Center). That rule makes those reviews harder to manufacture than quotes on an agency’s own site.
- Look for the typical case. Ask the agency what a normal client sees in the first six months, not just its best one.
What should you ask an agency about its case studies?
Bring these to the sales call. A confident agency answers them easily.
- Which of these case studies is closest to my store in revenue, product type and traffic?
- What was the baseline, and over what dates did the change happen?
- What did your team do, in what order, and what did the client’s own team do?
- What else changed in that period: ad spend, pricing, new products, a site rebuild?
- Can I speak to this client, or to a client in a similar position?
- What does a typical engagement look like, rather than the best one?
Our Shopify agency vetting framework covers the rest of the interview, from technical depth to contract structure.
Which red flags should make you walk away?
| Red flag | Why it matters |
|---|---|
| No dates anywhere | You cannot tell whether the result is recent or from a different era of the platform |
| Only percentages, never absolute numbers | Small bases produce big percentages |
| Every test is a winner | Either the agency hides losses or it stops tests early |
| Results credited entirely to the agency | Ad spend, products and the client’s team also drive growth |
| No client will take a call | The case study cannot be confirmed |
| Guaranteed results | No honest agency can promise a number before seeing your data |
For the wider process behind conversion work, see our guide to ecommerce conversion rate optimization.
How Build Grow Scale presents its client results
Build Grow Scale’s client results page, where its Shopify case studies are collected, presents six case studies, each with a headline outcome and a linked video telling the story; four of them name the client. The page names the Ongoing CRO Partnership as the service behind most of these engagements. Apply the checklist in this article to that page the same way you would to any other agency’s: ask for baselines, dates and a client to speak to.
FAQ
How many case studies should a Shopify agency have?
There is no right number. Three case studies close to your store’s size and category are worth more than thirty from very different businesses. Ask the agency to pick the two or three most like your store and walk you through them, including what went wrong.
Is it a bad sign if an agency’s case studies don’t name the client?
Not always. Some clients ask not to be named, especially larger brands. In that case, ask the agency to describe the client closely (category, revenue range, platform) and offer a reference call. An agency that cannot name clients and also cannot arrange any reference call gives you nothing to confirm.
What is the most reliable metric in an ecommerce case study?
Revenue per visitor is often the most reliable single number, because it combines conversion rate and order value and does not rise just because traffic grew. Conversion rate is useful when the case study says which rate it means. Shopify separates store conversion rate from checkout conversion rate, and the checkout rate is usually far higher.
Should I trust video testimonials more than written ones?
A video makes it harder to invent a client, so it is a stronger signal that the client exists and was happy. It still shows one client’s view. Unless the advertiser can show those results are typical, FTC guidance says the ad must make clear what results people can generally expect, so ask the agency what a typical engagement produces, not just its standout story.
Book a call
A free 30-minute call about your store. Shopify Plus Partner, rated 5.0 from 8 reviews on Shopify’s Partner Directory.
Sources
- Federal Trade Commission, “FTC’s Endorsement Guides: What People Are Asking”: https://www.ftc.gov/business-guidance/resources/ftcs-endorsement-guides-what-people-are-asking (fetched 2026-10-23)
- Federal Trade Commission, “Federal Trade Commission Announces Final Rule Banning Fake Reviews and Testimonials” (August 14, 2024): https://www.ftc.gov/news-events/news/press-releases/2024/08/federal-trade-commission-announces-final-rule-banning-fake-reviews-testimonials (fetched 2026-10-23)
- “How Not To Run an A/B Test” (evanmiller.org): https://www.evanmiller.org/how-not-to-run-an-ab-test.html (fetched 2026-10-23)
- Shopify developer documentation, ShopifyQL sessions schema: https://shopify.dev/docs/api/shopifyql/latest/schemas/sessions_and_behavior/sessions (fetched 2026-10-23)
- Shopify Help Center, “Hiring and working with Shopify Partners”: https://help.shopify.com/en/manual/partner-directory/hiring-help (fetched 2026-10-23)