March 6, 2018

What is Advertising Quality Science?

In this blog post, Skyscanner Data Scientist Ruth Garcia gives her take on Advertising Quality Science and explains why Skyscanner have it at the top of their priority list. 

What is Advertising Quality Science?

As mobile usage grows, ad serving systems continue to explore new ways of satisfying users’ needs according to two aspects: relevance and quality.

Relevance is the extent to which an ad matches a user’s interest, and is often addressed by the targeting criteria of campaigns (e.g. show ads related to flights from NY to LDN for users who search flights in that route). Quality, on the other hand, is a characteristic of the ad itself and is independent of the users targeted by the platform. Quality also involves the user experience with the ad.  Advertising Quality Science is thus the process of assessing online advertising from a deeper and more comprehensive perspective than just relevance or clicks. The purpose of advertising quality science is to maximize revenue guaranteeing ROI to advertisers without negatively impacting user experience and being able to “help” advertisers improve the quality of their ads.  

Advertising quality is very much on Skyscanner’s radar as we increase our advertising inventory and improve our app. In mobile especially, we thrive in making the landscape both compelling and competitive and ads are not the exception. We support our efforts with research done on the field of computational advertising.

Why is advertising quality assessment important?

Assessing the quality of online advertising and measuring user experience with ads is important for making decisions that will avoid driving users away and to provide feedback to advertisers on the quality of their ads.

Promoting not only relevant, but also quality ads to users is crucial for maximizing long-term user engagement with the platform [1,2]. In particular, low-quality advertising can have a detrimental effect on long-term user engagement [2,3,4]. Disturbing ads, for example, can cause various issues beyond mere annoyance; users might get distracted, or be unable to consume the actual content of the page where the ad is displayed [2,3]. Low-quality advertising can have even more severe consequences in the context of native advertising, since native advertisement forms an integrated part of the user experience of the product.

One misleading way of thinking about ad quality is to assume that click-through rate (CTR) is a natural metric to predict quality. However, CTR only reflects short-term user engagement. Research shows that although CTR is somehow related to the ad quality, high CTR may not imply good ad quality [2] perhaps due to the high level of accidental clicks.

Native advertising was added on the first position of Skyscanner’s search result pages in 2016 and currently we are testing how to incorporate these formats in the mobile app. For this reason, it is on our agenda to assess the quality of ads in Skyscanner with reliable metrics that go beyond only “click through rates”.  

How to measure advertising quality?

everal research papers measure advertising quality by analysing what happens before (pre-click) and after an ad has been clicked (post-click) [2,5,6].

Pre-click experience refers to the user experience induced by the ad before the user decides (or not) to click. One way of assessing pre-click experience is through the collection of feedback from users when ads are exposed to them. For example, in a study [2] offensiveness annotations were collected from users who chose to hide ads and further select one option motivating the reason of their choice. With this feedback, a set of ad features were designed that allowed the offensiveness of the ads to be predicted, allowing the system to avoid showing ads with high probability of being offensive.

Post-click experience is the way the user experiences the landing page after an ad click. In general, the post-click experience can be measured by means of two well-known metrics: dwell time and bounce rate. Dwell time is the time between users clicking on an ad creative until returning to the stream; bounce rate is the percentage of “short clicks” (clicks with dwell time less than a given threshold). These two metrics allow ad systems to understand when clicks have been accidental, when they have been intentional and when they have been particularly useful for the user. The quality of the landing page will affect the ad post-click experience: “the longer the time on the landing page, the more likely the experience was positive”. Research [5,6] has shown that a good post-click experience, and the structure and organisation of the landing pages, leads users to click on ads in the future and positively affects their long-term engagement.


Understanding pre- and post-click user experience with ads through measurable techniques is important for the stability of ad systems. Ads do not need to be annoying all the time, they can inform people of new products and ideas. They can entertain, communicate feelings and values and even make users feel good about themselves building a positive relationship with the firm. Nevertheless, the path towards advertising quality is not easy because new metrics need to be built and added to prediction models and systems. As a result, not many companies prioritise “ad quality”.

However, in an evolving economy where users interact more with mobile, showing low-quality ads and being unable to detect it can bring terrible consequences for ad systems. Adding the tracking of ad quality is a way of caring not only about the sheer number of clicks on ads but also about the user experience. This could become more important in the future.

At Skyscanner, we are aware of the importance of ad quality and we are taking steps towards showing advertisements that do not discourage users from continuing to use the platform and hopefully to even engage user to click more on our ads.

By Ruth Garcia, Data Scientist, Skyscanner

Interested in finding out more about Advertising, and how you can work with Skyscanner? Get in touch. 


[1] K. Dave and V. Varma. Computational Advertising: Techniques for Targeting Relevant Ads. now Publishers, 2014.

[2]K. Zhou, M. Redi, A. Haines and M. Lalmas. Predicting Pre-click Quality for Native Advertisements, 25th International World Wide Web Conference (WWW 2016), Montreal, Canada, April 11 to 15, 2016.

[3] D. G. Goldstein, R. P. McAfee, and S. Suri. The cost of annoying ads. In Proceedings of the 22nd international conference on World Wide Web, pages 459–470. International World Wide Web Conferences Steering Committee, 2013

[4] C. Yun Yoo and K. Kim. Processing of animation in online banner advertising: The roles of cognitive and emotional responses. Journal of Interactive Marketing, 19(4):18–34, 2005.

[5] M. Lalmas, J. Lehmann, G. Shaked, F. Silvestri and G. Tolomei. Promoting Positive Post-click Experience for In-Stream Yahoo Gemini Users, 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Indutsry Track), Sydney, Australia, 10-13 August 2015.

[6] N. Barbieri, F. Silvestri and M. Lalmas. Improving Post-Click User’s Engagement on Native Ads via Survival Analysis, 25th International World Wide Web Conference (WWW 2016), Montreal, Canada, 11-15 April 2016.