How to tell whether a marketing spend actually brings anything

You find out whether a marketing spend brings anything only by comparing it with what would have happened without it, in an area or a period where you switched it off. The report in the advertising platform does not make that comparison.

The platform counts the sales that passed through a click on an ad, including those of customers who were already searching for you by name. Some of them would have bought anyway. The sentence I use for this topic: “A marketing belief becomes fact only after you have tried to disprove it.”

What eBay measured

In 2015, the journal Econometrica published a study carried out at eBay by three economists, Thomas Blake, Chris Nosko and Steven Tadelis. They tested paid ads on search engines. Many companies considered them a profitable channel, and the authors wanted to see how much they actually bring in.

The first test looked at brand keywords, meaning searches that contain the company’s name, such as “ebay shoes”. eBay stopped these ads on Yahoo and MSN and left them running on Google.

Google acted as the control group. The control group is the part where you change nothing, so you can see what would have happened anyway. Without it, any drop during the test period could be blamed on the season, a promotion or a bad day.

The working paper version of the study, published in 2014 by the US National Bureau of Economic Research (NBER), gives the figure: 99.5% of the clicks lost on ads came back through natural results, the unpaid ones. A person typing “ebay” found eBay without an ad. The authors’ conclusion is that ads on brand keywords have no measurable short-term benefits.

The second test looked at the remaining keywords. eBay stopped these ads for 60 days in about 30% of US market areas, that is 68 test areas against 142 control areas. The control areas were chosen by a simple criterion: their sales had moved in the past the same way as in the test areas.

New or infrequent customers bought more because of the ads. Frequent customers, however, whom the ads did not influence, accounted for most of the budget. The average return came out negative.

P&G: two statements, two different things

Procter & Gamble, one of the biggest advertising buyers in the world, cut digital spending in 2017. Two statements about this often circulate together, although they say different things.

The first comes from Jon Moeller, the chief financial officer, in the 27 July 2017 call with analysts about quarterly results. He said the company had cut more than $100 million from digital advertising in the last quarter of the fiscal year and had not seen a slowdown in growth. From this he concluded that the money cut had been “largely ineffective”.

The second comes from Marc Pritchard, P&G’s chief brand officer, in March 2018, at a conference of the ANA, the American association of companies that buy advertising. He said P&G had cut $200 million from digital in 2017 and that reach, meaning the number of people the communication gets to, had grown by 10%. His claim is about reach and says nothing about sales.

Neither statement describes a test with a control group. I take them as serious signals, coming from the people who cut the budget, that part of the money was bringing nothing.

Airbnb and the atypical year

On 25 February 2021, in the call with investors about the 2020 results, Brian Chesky, Airbnb’s co-founder and chief executive, described what happened with marketing. In 2020, the business fell by 80% in eight weeks, and the company stopped all marketing. Before it resumed spending, traffic had returned to 95% of its 2019 level, he said.

Chesky also said Airbnb does not intend ever again to spend the same proportion of revenue on marketing as before. The 95% figure has to be read with care, because 2020 was an atypical year.

The pandemic changed the way people travel, and the pause had no control area. The case remains a natural experiment, meaning a situation in which circumstances stopped the spending in the company’s place. It shows you a good question without giving you a clean comparison.

How to check your own spend

1. Ask what would have happened anyway

For each spend, write down who would have bought even without it. Customers who search for you by name, those who come back monthly and those who arrive through recommendations are the first candidates. If the ad mostly reaches them, the report looks good while the money pays for sales you already had.

2. Choose a control area or period

The cleanest comparison is between areas: you switch the ad off in a few cities or counties and keep it in others. Choose the control areas by how their sales have moved over recent months, as eBay did. If you sell in a single place, compare periods with the ad and without it, alternated a few times, and allow for the season.

3. Switch off only part of it

To learn something, you do not have to stop the whole budget. eBay limited the test to part of the areas to reduce the loss, in case the ads really were bringing in profit. For a small company, a third of the areas or of the budget is a prudent starting point.

4. Decide in advance what you measure and how long you wait

Choose one figure that matters for the company, such as orders, sales or quote requests, counted outside the advertising platform. Clicks and reach do not answer the question. Write down in advance which result would make you stop the spend and which would make you keep it.

Set the duration before you start and do not stop at the first good or bad week. For a product that is bought rarely, the test lasts at least as long as a usual buying cycle.

5. Read the result as a clue

A test tells you what happened in a specific period, in a specific market. The eBay study, for example, speaks about the short-term effect and leaves the long-term effect as an open question. Repeat the test after a few months before you move large budgets.

What to take away

A marketing spend brings something only if sales rise where it runs, compared with a comparable area or period without it. The platform report also counts the customers who would have come anyway. A partial test, with a control group and a figure chosen in advance, answers you better than any report.