Short answer
A conversion rate optimization (CRO) agency studies why visitors to your store don’t buy, turns those findings into changes, tests the changes against your current pages, and keeps the winners. In a typical month that means research, a ranked list of ideas, building and checking test versions, running tests, reading results, shipping winners into your theme, and reporting what changed in revenue.
Key takeaways
- A working CRO month runs on a weekly loop: research feeds a ranked backlog, the backlog feeds tests, and tests feed a log of what was learned.
- Several specialists touch each test: research and analysis, design, development and quality checks.
- Not everything gets tested. Clear problems, such as surprise shipping costs, are fixed directly.
- A test only pays once the winning version is built permanently into your live theme.
Who does what in a CRO month?
Most of a CRO retainer pays for specialist time, and each specialist has a different job in the month. Titles vary by agency, but the work splits roughly like this:
| Role | What they do in a typical month |
|---|---|
| Strategist or lead | Runs the monthly review with you, ranks the backlog, writes or approves each hypothesis, decides what ships |
| Researcher or analyst | Reads analytics, watches session recordings, runs surveys and user tests, reads test results |
| Designer | Mocks up each test version so it fits your brand and works on mobile |
| Developer | Builds test versions in the testing tool, then builds winners into the live theme |
| QA (quality checks) | Checks every version on main devices and browsers, and checks that tracking still fires |
On a smaller retainer one person can cover two or three of these. On a strategy-only retainer the agency covers the first two rows and your own developer covers the rest. Ask which rows the agency staffs before you sign, because the missing rows become your work.
Worth knowing
Research doesn’t have to be expensive. Session recording and heatmap tools can be free: Microsoft Clarity describes itself as “Free forever.” User tests need few participants. Nielsen Norman Group’s widely cited model estimates that a study with five participants finds about 85% of usability problems, and it recommends several small rounds rather than one big one.
Book a call
A free 30-minute call about your store. Shopify Plus Partner, rated 5.0 from 8 reviews on Shopify’s Partner Directory.
What does the weekly loop look like?
Tests overlap, so in any given week some ideas are being researched, some built and some read. A steady month looks roughly like this:
| Week | Research and planning | Building and checking | Live testing and analysis | Shipping |
|---|---|---|---|---|
| Week 1 | Review last month’s results with you; add new findings to the backlog | Build the next one or two tests | Read results of tests that reached their planned sample size | Schedule last month’s winners for the theme |
| Week 2 | Re-rank the backlog with new data | QA new tests on mobile, desktop and main browsers | Launch checked tests | Build winners into the live theme |
| Week 3 | New research round: surveys, recordings, support tickets | Fix anything QA found | Watch live tests for errors and broken tracking | QA shipped winners on the live site |
| Week 4 | Write the monthly report and next month’s plan | Prepare designs for next month’s tests | Close finished tests and write up what was learned | Record what shipped and when |
The first weeks of an engagement look different: access, tracking repair and the first research round come before any test, so expect few tests early on. Tracking comes first because every later decision depends on it. Google’s GA4 ecommerce documentation lists the events a store should send, from view_item and add_to_cart to begin_checkout and purchase, and marks the transaction ID as “Required for purchases and refunds.” If purchases double-count or go missing, test results are wrong before they start.
How is a single test built, checked and launched?
Each test moves through the same steps, over several weeks, depending on traffic:
- Evidence. A finding from research: for example, recordings show mobile shoppers scrolling past the size guide.
- Hypothesis. A written statement of what will change, for whom, and which number should move: “Showing the size guide next to the size picker on mobile will raise add-to-cart rate on product pages.”
- Ranking. The idea is scored against others on likely impact, confidence and effort, and placed in the backlog.
- Design. A mockup of the new version, approved by you.
- Build. The developer builds the version in the testing tool, so the live theme is untouched.
- QA. Checks on main devices and browsers, plus a check that purchase and add-to-cart tracking still fire in both versions.
- Launch and watch. The test runs until it reaches the planned sample size. The team watches for errors but does not stop the test early because one version looks ahead.
- Read and record. Win, lose or no clear result, the test goes in the log with what was learned.
Most tests won’t win. Optimizely’s analysis of experiments on its platform found that only 12% produce a statistically significant improvement on the primary metric, and that experiments combining three or more types of change delivered the strongest gains. That is why the log of losing tests matters: it stops the same weak idea being tried twice.
How do winners get onto the live site?
Watch out
A winning test is still running inside a testing tool, on top of your theme. Until a developer builds the change into the theme itself, you are paying for the tool to show it, and the change can break when the theme or an app updates.
A good agency treats shipping as part of the month, not an afterthought:
- Winners are built into the theme within a set time, agreed at the start (for example, within two weeks of a result).
- The test is switched off once the permanent version is live, so the tool isn’t left serving an old version.
- The shipped change gets its own QA pass on the live site.
- Each shipped change is dated in the log, so later revenue changes can be matched to it.
If your own developer ships winners, agree on turnaround with them too. A winner waiting a month in a developer queue is a month of lost lift.
When does an agency fix something directly instead of testing it?
Testing costs time and traffic, so it is saved for real questions. Clear problems get fixed without a test: broken links, a checkout error, a missing shipping estimate, a forced account sign-up. Baymard Institute found that 40% of shoppers who abandon checkout, excluding those just browsing, leave because extra costs such as shipping, tax and fees were too high, and 18% because the site wanted them to create an account. Nobody needs a four-week test to confirm those are worth fixing.
Tip
A simple rule: fix directly when the change removes something clearly broken or clearly hostile to buyers; test when reasonable people could disagree about whether the change helps, or when it could cost money (a new price, a new offer, a big layout change). For the checks an agency runs to find those fix-now items, see the CRO audit checklist for ecommerce stores.
How many tests should run each month?
There is no universal right number. It depends on your traffic, how big each change is and who builds the tests. As a market guide, one agency’s published 2026 pricing guide ties entry retainers to two to four tests a month and mid-tier retainers to six to ten. The same Optimizely analysis says the median company runs 34 experiments a year. In practice, a few well-researched tests with meaningful changes usually teach you more than many small button-color tests.
What should you receive, and what does the agency need from you?
Each month you should get, at minimum:
- a test log listing every test, losers included, with its result and decision;
- a revenue report showing revenue per visitor, not conversion rate alone;
- a list of what shipped to the live theme, with dates;
- next month’s plan and the reasons for it.
In return, plan a few hours a month on your side: a monthly review, quick approvals on copy and design, advance notice of promotions and price changes so they don’t collide with tests, and help reaching customers for surveys or user tests. For running your own user tests alongside an agency, see the expert guide for running successful user tests, and for what to check before you hire, how to choose a CRO agency for Shopify.
How Build Grow Scale runs a CRO month
Build Grow Scale’s conversion rate optimization agency service starts at the checkout and cart, then moves testing outward to the rest of the store. Every engagement includes a data-accuracy audit, checkout and cart optimization, and reporting built around the store’s P&L. In the ongoing partnership, the team is a lead growth strategist, front-end and back-end developers, a data scientist and a QA specialist, working from one backlog scored on reach, impact, confidence and effort. The client sees the 90-day plan before any code is written, and the first six to eight weeks go to clean data and core best practices before week-by-week testing.
FAQ
What is the difference between a CRO agency and a web design agency?
A web design agency builds or redesigns pages to a brief, usually as a project with an end date. A CRO agency measures how visitors behave, decides what to change from that evidence, and tests changes against the current site before keeping them. Some agencies do both. The difference shows in the deliverables: designs and launches versus test logs and revenue reports.
Do CRO agencies write code and build the tests?
Some do and some don’t. Full-service agencies design, code and quality-check each test, then help ship winners into your theme. Strategy-only agencies hand recommendations to your developer. Strategy-only is cheaper, but you carry the build work and the delays. Ask who builds tests, who checks them on mobile, and who makes winners permanent.
How long before a CRO agency launches its first test?
In many programs, between week six and week twelve. The first weeks go to access, tracking repair and research, because tests built on broken data give wrong answers. An agency can fix obvious problems in the first weeks without testing them. If no test is live by the end of month three, ask why.
Can a CRO agency work with my in-house developer?
Yes, and many do. The usual split is that the agency researches, designs and builds tests in a testing tool, and your developer makes winning changes permanent in the theme. Agree on turnaround times before you start: a winner that waits a month for a developer is a month of lost lift.
Why should a CRO agency show me tests that lost?
Because most tests don’t win, and the losers carry most of the learning. A log that shows only winners hides how many tests ran, makes the program look better than it is, and lets weak ideas come back later. Ask for every test with its hypothesis, result and decision, so you can judge the work, not just the highlights.
Book a call
A free 30-minute call about your store. Shopify Plus Partner, rated 5.0 from 8 reviews on Shopify’s Partner Directory.
Sources
- Google, GA4 ecommerce measurement guide, https://developers.google.com/analytics/devguides/collection/ga4/ecommerce (fetched 2026-10-23)
- Nielsen Norman Group, “Why You Only Need to Test with 5 Users,” https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ (fetched 2026-10-23)
- Microsoft Clarity, product page, https://clarity.microsoft.com/ (fetched 2026-10-23)
- Baymard Institute, Cart Abandonment Rate statistics, https://baymard.com/lists/cart-abandonment-rate (fetched 2026-10-23)
- Optimizely, “Top 10 takeaways from running 173,000 experiments,” https://www.optimizely.com/insights/top-10-takeaways-from-running-127000-experiments/ (fetched 2026-10-23)
- An agency’s published 2026 CRO retainer pricing guide, https://clicksgeek.com/cro-agency-retainer-costs/ (fetched 2026-10-23)