ARF Winning Paper Upends Industry Norms About Digital Environments

The following is an excerpt from the ARF AudiencexScience 2023 award-winning paper, Decode Digital Video Attention by Environments in the Wild. 
 
In recent years, the introduction of feeds, stories, and other new formats has changed how people interact with ads and content. This has increased the complexity of the landscape exponentially yet advertising mental models and categories have not kept up with the innovation on the product side.  
 
As a pioneer in formats and advertising measurement, Meta wanted to better understand these issues and look at the problem in a rigorous, data-driven way (Sophie MacIntyre, Meta Ads Research Lead pictured). 

It tapped an industry-leading team of Realeyes, EyeSquare, VidMob, and Bill Harvey Consulting (pictured), each providing their own expertise, to execute a large-scale experimental study. Details of the study are located at the end of this blog post.

• There are different profiles of consumer behavior across environments 
• Despite this, brand outcomes are comparable - suggesting each environment achieves value in a different way 
• Attention has a different relationship with outcomes across environments and users, which should inform how we use these metrics 
  
  

For this campaign, Realeyes provided attention measurement through participants’ forward-facing cameras, classifying whether the user was paying attention in each frame using neural network processing. EyeSquare provided simulated environments with interactivity and tracking which participants accessed via browser, and VidMob adapted creative where necessary for individual placements. 
 
We used three common classifications of video placements: 

Feeds (I.e., Facebook, Instagram, and Twitter feed) 
Short form (I.e., TikTok TopView, Instagram and Facebook Stories) 
Stream (I.e., YouTube skippable and non-skippable, Hulu pre- and mid-roll) 
 
We looked at whether there are distinct consumer behaviors in each hypothesis that we need more granular categories. The goal was to identify how consumer behaviors varied across media environments and glean insights into how each creates value. 

 
What We Learned 

• The feed environment had the highest engaged audience, but lowest visibility. The stream audience had the opposite (high visibility, low engagement). 
• Average attentive times are longest for stream environments and shortest for feed environments, but overall levels of brand recognition are comparable
• Brand recallers generated per attentive seconds were much higher for feeds and short form versus streams.  
  
  

Each Digital Environment Type Has Its Own Characteristic Behaviors 

Source: Analysis of data from Meta-commissioned lab study by Realeyes of 16,835 ad exposures from 6,076 mobile users ages 18+ in USA, December 9th-24th 2022. Study included placements from the following apps: Facebook, Hulu, Instagram, Snapchat, TikTok, Twitter, YouTube.

Across environments, the feed environment had the lowest level of visibility and attention, while stream had the highest levels of visibility and attention. This makes sense especially given that all the stream formats had some or all forced time, whereas feeds are completely skippable. However, the feed environment showed the highest levels of engagement across the environments. 
 
It is likely that users are more actively sorting in feed environments, producing short and intense bursts of attention - whereas stream environments produce longer and more passive times with the viewer.   

While Attention Norms Vary Significantly Across Environments, Brand Recognition is Comparable 

Taking a closer look at brand recognition, we see robust, but comparable performance across all environments, despite significant differences in consumer behavior. It suggests each environment achieves value for advertisers in a different way.  

The Effect of Attention on Brand Outcomes Differs Across Environments

All things equal, more attention is correlated with higher ad recall and better brand recognition. However, the effectiveness of attentive time varies depending on environment. For instance, a second of attention is worth more in Feed environments than in Stream environments. 

The latter yields more attention mechanically but does not result in higher brand recall rates. The effectiveness is twice as high for feed and short form ads compared to stream ads.

What You Need to Do

• Tailor creative to take advantage of different second-by-second attentive profiles 
  Track how attentive behavior changes over time  
• Understand that the effectiveness of attention varies across environment, and some outcomes are better predicted than others 

Study Design

We partnered with five brands in tech, financial and CPG verticals who provided ads to test. We wanted to hold creative and audience as constant as possible so we can look at the effect of different environments - since we do know that creative and audience affect all the metrics we’re looking at.  

• Panel randomized to platform-placement test cells
• Key factors held constant: creative, audience, frequency 

1 min	2 min	1 min
Pre-Exposure Survey	In-Context View	Post-Exposure Survey
Platform Usage Platform Mindset	Ad Visibility Skips, Scrolls On-Screen Attention Reactions	Brand Recognition Ad Recall Brand Trust Ad Liking Persuasion

Participants were randomized to a platform-placement test cell - creating a huge split test across placements. First, they completed a survey about platform usage, then spent 120 seconds interacting with a simulated media environment. After the task, they were asked standard brand polling questions. 
 
Sample sizes were >300 participants per tested placement which makes this one of the largest studies of attention in the media landscape.