Correcting Mischaracterizations of Online Platforms A/B Testing in Marketing Research

Written by Veronica Burbulea, Ph.D. candidate at the University of Groningen (The Netherlands)

Over the past decade, online platforms have transformed how marketing researchers study consumer purchase behavior. With a few clicks, scholars can launch advertising campaigns on Meta or Google, expose thousands of consumers to different ad versions, and observe actual consumer responses. But are these studies the perfect field experiments they appear to be?

The paper “On the persistent mischaracterization of Google and Facebook A/B tests: How to conduct and report online platform studies,” published in the September 2025 issue of IJRM, addresses this question.

I had the opportunity to speak with one of the paper’s authors, Johannes Boegershausen, about a growing concern in marketing research: the widespread treatment of platform A/B tests as randomized experiments, even when true randomization is absent in these studies. A/B tests run via the functionality provided by platforms like Meta or TikTok to compare two versions of an ad by showing each version to different groups of users and measuring which performs better. However, without true randomization, the groups exposed to different ads may differ systematically, making it difficult to determine whether observed differences are caused by variations across the ads or by the platform’s delivery algorithm. The resulting discussion, therefore, challenges not only the use of A/B tests as experimental evidence but also raises broader questions about how the field balances realism, rigor, and responsibility.

The image was generated using AI

An Emergent Problem

Johannes’ concern emerged gradually. While reviewing manuscripts over several years, he repeatedly encountered studies describing online platform studies as experiments. A pattern became impossible to ignore.

“What surprised me most was how widespread these practices were once we systematically looked at the literature.”

- Johannes Boegershausen

During the review process, the IJRM review team encouraged the authors to take a descriptive view of the literature. The authors examined how frequently platform A/B studies appear and how they are framed. The resulting overview was less about critiquing individual papers and more about revealing a broader pattern: considerable ambiguity about what these studies can demonstrate.

Why A/B Tests Feel Like Experiments?

Part of the problem lies in intuition. Designing an A/B test on a social media platform closely resembles designing an experiment: researchers create two ad versions, assign budgets, and compare outcomes like ad clicks. But there is a crucial distinction from a conventional lab or online experiment: researchers relinquish control over randomization to the platforms and their targeting algorithms. Researchers are therefore not comparing ad version A versus ad version B, but ad version A + algorithm + user reactions versus ad version B + algorithm + user reactions.

The algorithm continuously learns, adapts, and optimizes exposure. As a result, the groups receiving each treatment may differ systematically. This undermines internal validity: the confidence with which a researcher can conclude that a cause-and-effect relationship exists between the independent and dependent variables. The paradox is clear: the studies look realistic because they operate in real markets, but internal validity is compromised.

The Field’s Love Affair with Real-World Data

Why have reviewers and editors often accepted these studies as experiments? According to Johannes, field-level incentives may play a major role. Experimental marketing research has long sought evidence that theories hold outside the laboratory. Online platform studies are high in ecological validity: real consumers making real choices in commercial environments.

“They seemed like a silver bullet. A way to show causal effects directly in the field.”

- Johannes Boegershausen

However, ecological validity only has value if it does not come at the expense of internal validity. When algorithmic targeting shapes exposure, researchers may unknowingly test something different than their theoretical prediction.

Stop Fighting the Algorithm—Study It

Rather than abandoning platform data altogether, the paper proposes a shift in perspective: if algorithms determine exposure, then the algorithm itself becomes an object of study.

Online platforms can reveal how advertising delivery varies across gender, age, or other characteristics. Prior research Johannes reviewed shows that subtle wording changes, such as masculine versus feminine language, can influence which audiences receive an ad. Consumers then respond differently, reinforcing algorithmic patterns over time.

This raises an important possibility: algorithms may not accurately reflect consumer preferences; they may amplify social biases through feedback loops. Understanding these mechanisms opens a new research frontier at the intersection of marketing, technology, and society.

Ethical Questions

Platform studies also introduce ethical challenges rarely encountered in lab experiments. Participants do not sign consent forms; they simply encounter advertisements while browsing online. One practical guideline proposed during our discussion was simple: always consider the worst experimental condition. Would it be acceptable to expose unsuspecting users to that treatment without explicit consent?

Researchers can also mitigate risks by providing transparency after engagement—for example, explaining the study on landing pages or directing interested participants toward real organizations when simulated campaigns are used.

A Field Searching for Consensus

Perhaps the most striking observation concerns peer review. Identical platform studies can receive radically different evaluations: some reviewers dismiss them entirely, while others praise them as exemplary field experiments. Such inconsistency suggests the need for shared standards for interpreting platform evidence.

Johannes does not advocate banning these studies. Instead, the goal is consensus on what online platform experiments can and cannot establish. At present, they may be less suited for testing classic consumer behavior theories but highly valuable for studying algorithmic processes.

Will Marketing Research Drift Away from Practice?

A natural concern follows: if social media platforms do not offer true randomized experiments, will academic research become disconnected from industry practice? Not necessarily.

Researchers can collaborate with firms, conduct truly randomized A/B tests on company websites or mailing lists, or even build custom digital environments. Recent projects have gone so far as creating entire e-commerce platforms to enable proper randomized controlled trials. The message is cautiously optimistic: meaningful field experimentation remains possible; it just may require more effort.

Read the paper

Interested in reading all the details about the mischaracterization of online platform studies?  Read the full paper here.

Want to cite the paper?

Boegershausen, J., Cornil, Y., Yi, S., & Hardisty, D. J. (2025). On the persistent mischaracterization of Google and Facebook A/B tests: How to conduct and report online platform studies. International Journal of Research in Marketing, 42 (3), 886-903.

---

Meet Johannes Boegershausen

Associate Professor at Rotterdam School of Management, Erasmus University Rotterdam, the Netherlands

What is the number one question you want to answer in your career?

“I typically don’t have one. I am primarily guided by what excites me at a given moment. Currently, two things stand out. First, we are building a mock short-form video platform to explore what drives people to consume short-form video feeds like TikTok or YouTube Shorts. Second, I’m also very interested in consumers and workers with disabilities and thinking about accessibility. What can we do to bring more people with disabilities into the primary workforce? How do consumers react to that? What can companies do to make marketplaces more inclusive?”

What is the best advice you have ever received, and how has it influenced your career or life?

“You miss 100% of the shots you don’t take. That is, you can’t score if you don’t shoot. At some point, you need to let go and try (e.g., submit your paper). You don’t know how the reviewer draw will turn out. Sometimes it’s sheer luck.”

---

The article was written by

Veronica Burbulea

Ph.D. candidate at the University of Groningen (The Netherlands)