In collaboration with GoPractice, Letyshops CMO Zakhar Stashevsky continues to discuss product growth through effective advertising channel management.
Table of contents for this series of essays
- Errors in calculating ROI and unit economics. Impact of attribution models and incrementality on the ROI calculation of marketing channels. In this column, we discuss why, when calculating the unit economics, it is impossible to ignore the influence of the used attribution models and advertising incrementality.
- Traffic attribution models. Why attribution models should change along with growth channels, product, business challenge and external environment [you are here]
In this piece, we discuss how to select attribution models to assess the effectiveness of advertising channels based on the specifics of the product, marketing mix, business objective, and environmental conditions. We will also explain why it is necessary to revise and adapt the attribution model in the event of changes in these factors.
From here on, Zakhar tells the story.
In the previous essay, we discussed how the incorrect calculation of the unit economics and ROI can lead to an underestimation or overestimation of the advertising channel and, as a result, erroneous marketing decisions. Excessive scaling or channel shutdown can lead to direct or indirect financial loss, which leads to missed growth opportunities on the market.
Such errors can happen for a variety of reasons. One of the most common is marketing and analytics teams not paying enough attention to the attribution models based on which they make their decisions. They simply use the default attribution model available in the analytics system, or select a model when they start working with a new channel but don’t change the model as the company evolves and the marketing mix, the length of the product sales cycle, and external factors change.
→ Test your product management and data skills with this free Growth Skills Assessment Test.
→ Learn data-driven product management in Simulator by GoPractice.
→ Learn growth and realize the maximum potential of your product in Product Growth Simulator.
→ Learn to apply generative AI to create products and automate processes in Generative AI for Product Managers – Mini Simulator.
→ Learn AI/ML through practice by completing four projects around the most common AI problems in AI/ML Simulator for Product Managers.
The role of the attribution model in the marketing team
What problem does the marketing team solve?
The task of the marketing team is to spend its budget as efficiently as possible to maximize the target growth metric.
Like many marketers, for a long time I was convinced that to solve this problem, you need to open a report on advertising campaigns and, based on it, make the following decisions:
- Which channels (or campaigns) are worth scaling
- Which ones can continue without modifications
- Which ones must better be stopped
It seemed to me that the main opportunities for improving the result lay precisely in the following dimensions: searching and testing new channels; managing the existing ones; optimizing campaigns; creation and testing of creatives.
I could explain all my decisions based on the data from the report, and I was completely confident in them. But as the number of channels, the budget volume, and the complexity of the product grew, I began to face increasing anomalies. The differences between our own reports and those of ad networks increased, organics grew abnormally, and the effectiveness of the main growth channels dropped, even though our intuition was that they were working well.
All of this led me to believe that there is a hidden problem. This problem seemed to be hiding in a blind spot, which I had never deeply cared about. How accurately do the reports that guide me describe reality?
Why the attribution model often becomes a blind spot
The blind spot was at the junction of the responsibilities of two teams: marketing and analytics.
The analytics team built an infrastructure a long time ago and equipped it with a specific traffic attribution model. Based on this model, reports were generated for the marketing team, which they used to make decisions.
These reports worked well early on. But the product, channels, team goals, and environment were constantly changing. And at some point, the old model began to describe reality poorly. In the value chain, the team had a blind spot, a space that no one monitored nor understood.
Note that the attribution model is not the only blind spot at the intersection of marketing and analytics in the context of advertising channel management. There are also tasks of forecasting LTV, correctly determining the payback period and others. But we will leave these topics for another article.
What is an attribution model and what does it accomplish
First, a classic definition: An attribution model is a set of rules and technical solutions, based on which you determine the channel to assign an attracted user or sale to. You can also use the attribution model to decide how to evaluate the contribution of a channel in case user interacted with several channels before making a target action.
Ideally, attribution models should measure the contribution of each channel to user acquisition or sale with perfect accuracy. But in practice this is impossible for two reasons:
- It is extremely rare that we have a complete history of a user’s interaction with a product ads in advertising channels (for example, if a person sees an ad but doesn’t click on it, then this interaction will be invisible to us).
- If the user interacted with several channels, then it is difficult to determine which channel played the key role in the conversion (in this case, even the user himself often doesn’t know the real impact of each of the advertising touches).
We might be able to solve the second problem to some degree by distributing value between the channels in different ways based on our understanding of their importance. But the first problem is impossible to solve because we can’t read users’ minds nor see the world through their eyes.
But the good news is that, in practice, you don’t need perfect fidelity from an attribution model. The marketer doesn’t have many choices to make:
- Add budgets and scale the channel;
- Leave it as it is;
- Optimize the budget or stop the channel if it is unprofitable.
Therefore, in most situations, a more appropriate definition of an attribution model would be the following:
An attribution model is a set of rules and technical decisions that allow you to make the most effective marketing decisions in a short time based on available data.
In the context of this definition, the attribution model begins to play the role of a compass instead of a tool that provides accurate coordinates. And for decision-making, reliable benchmarks will be more effective than accurate numbers obtained in an unreliable way (for example, using only one specific attribution model).
To assess the effectiveness of advertising channels, the attribution model must evolve with the channels and product
A common mistake that teams make (including our own team at Letyshops in our early stages) is refraining from developing a process to choose and change the attribution model as the product and growth channels evolve.
Most often, companies take an off-the-shelf analytics product and use the default attribution model it provides. As a result, they don’t understand how well this model suits the specifics of their product, channels, and business objectives.
A more subtle mistake happens when the team adopts an attribution model in the analytics system that works well early on. In many cases, the mode’s accuracy in describing reality gradually decreases as the product and the marketing mix changes, but the team continues to use it, oblivious to the fact that they are making decisions based on a faulty model.
The attribution model used to assess the effectiveness of growth channels should evolve, adjusting to the specifics and conditions, such as marketing mix, product development stage, business objective, and external influences.
An example of the evolution of the traffic attribution model and methods for assessing the effectiveness of advertising channels in a constantly changing situation
Let’s start with an example that clearly illustrates the key message of this essay: The traffic attribution model must evolve in tandem with other key elements of the growth model. If this doesn’t happen, then the quality of marketing decisions decreases.
Example # 1. Short cycle product; one acquisition channel
Company N launched a subscription coffee service and began attracting users through Facebook ads.
↓
For attribution, the team used Google Analytics and the standard built-in Last Non Direct Click attribution model.
To understand this model, note that when a user directly enters the url of a website in their browser, then they are entering the website “directly.” Also, when the system can’t determine the source from which the user came to the website, it will again register the entrance as “direct.”
In the Last Non Direct Click attribution model, if the user enters the website through a non-direct channel (e.g., social media, ad, newsletter, referral, etc.), then that channel is assigned to the user. If the user came through a direct channel, then the system will try to find the last non-direct channel that brought the user to the website in the last six month, and it will assign that channel as the source of acquisition for the user.
↓
The picture obtained using this attribution model described reality well, since most users registered and made a purchase immediately after clicking on the advertising link.
↓
Based on the data obtained, the team began to scale the Facebook channel.
The chosen attribution model was correctly selected for the product, channel and task, which allowed the team to understand the effectiveness of the channel and make the right decision on its scaling.
Example # 2. Organic growth by brand queries
As the Facebook paid channel scaled, the team began to notice an increase in organic search. Analysis showed that these users were coming from branded searches (searches with keywords about your product’s name) that were a direct or indirect consequence of Facebook ads.
↓
The company’s goal was to grow as quickly as possible. The old attribution model was beginning to distort the situation and underestimate the effectiveness of advertising on Facebook. It was now attributing many people who had been exposed to the product through Facebook Ads to the organic search channel, because that was the final method they used to find the website.
↓
The team decided to analyze the data using other attribution models.
First, they tried to measure the effectiveness of Facebook ads by attributing organic traffic to FB ad campaigns because it was an indirect consequence of this ad. There were no other channels in the marketing mix at this point.
They also tried using the ad channel’s internal attribution model as it had a more complete history of user interactions with the ad. Facebook’s data tracked when a user clicked or saw the ad and attributed it to the ad campaign, even if he later made a purchase from a different browser or device.
↓
The two new attribution models gave the team a new perspective and understanding of the range of effectiveness of ad campaigns.
The first model that attributed all organic traffic to the Facebook channel gave the highest possible effectiveness of advertising campaigns (top score). Facebook’s internal attribution model score is an intermediate result based on the data available to the social network (which is more than the data available to the analytics system used by the team).
The old model was also used to get a lower bound of the campaign’s performance.
Now the team had three tools at its disposal to measure the range of effectiveness of different campaigns and monitor the degree of their discrepancy, and understand how effective they are in the current situation.
↓
These decisions allowed for a different assessment of the effectiveness of advertising campaigns on Facebook and the decision to continue scaling the channel, which led to further growth of the company.
It is often a good idea to use multiple attribution models. This approach will allow you to build upper and lower bounds for the effectiveness of a channel or campaign, and therefore make an informed decision about their optimization or scaling in the context of your business objectives.
Example No. 3. Adding a Google AdWords Channel
In response to the growth of the product and the increase in traffic for branded search queries, competitors began buying branded traffic on Google (buying ads for queries where people search for your product).
↓
The team took countermeasures and also began to buy branded queries at prices that allowed them to move competitors off their places in the search results. The team was now using two channels, one fueling the other (brand searches were largely triggered by users being exposed to Facebook ads, but they could also be the result of word of mouth or other mechanisms that made the product more visible as its active user base grew).
↓
Google branded ads were significantly more effective than Facebook ads, according to the attribution model that was originally used. Some team members began to promote the idea that some of the Facebook ads budget should be allocated to Google Adwords because the ROI is much higher there.
↓
But this would be a mistake because the factor of advertising incrementality in search wasn’t taken into account.
The problem with incrementality is that we don’t know if a user attributed to an ad in a channel would come to us even if the ad wasn’t there. It’s possible that a person who searched for a product name on Google would have clicked on the first organic search result even if there were no ads. Or maybe not, because in the absence of our ads, competitors’ ads would appear on the page and the user would have clicked on those instead of the organic result.
Moreover, since search advertising collects demand for a product, and does not generate it, at least one other channel (Facebook or, for example, word of mouth from another satisfied customer) was definitely involved in the user acquisition chain. When analyzing a Google channel based on Last Click Non Direct attribution, this factor is not taken into account.
↓
Previous attribution models did not solve the problem of search ad incrementality. In other words, they did not take into account that ads are only part of the acquisition funnel.
↓
The team added another model for assessing the effectiveness of search advertising: “checkerboard.” (We discussed checkerboard in the first essay of the series. Basically, checkerboard helps evaluate the effectiveness of a channel by intermittently turning it on and off over a period of time and comparing the difference of the outcomes.)
↓
This approach made it possible to assess the incrementality of the channel, measure its objective efficiency, and optimize the costs. As a result, some of the paid search traffic for the brand was re-attributed to Facebook. With the addition of the checkerboard, search ads became less attractive than they initially appeared.
As the marketing mix grows, the problem of channel incrementality arises. For example, a user might have interacted with multiple channels, but value was attributed to the most recent. Meanwhile, the channel that generated demand for the product was underestimated. Incrementality tests are a great addition to attribution models to optimize your channel.
Example No. 4. Complication of the product; adding email marketing and retargeting
The product team redesigned the product and added new features. With the new changes, users could not only buy coffee by subscription, but also learn how to roast and brew it in different ways in offline events and master classes from the company.
↓
The decision to buy a master class or a full course usually takes longer than buying a subscription coffee (the original product).
The change in the product line increased the length of the deal cycle, which in turn changed the process of acquiring users. To avoid users forgetting the product, the team set up remarketing and email newsletters for people who visited the website. These channels optimized purchase conversions and helped close the purchase funnel for more complex and expensive products.
↓
The lengthening of the deal cycle and the addition of funnel optimization channels made it difficult to evaluate the effectiveness of the channels. Previously used attribution models gave all the value to the channels at the end of the funnel.
The attribution of organic traffic also became more complicated, since it was influenced by not one (Facebook) but several channels. At the same time, it was difficult to determine what kind of influence each of the channels had.
↓
In this situation, when using old attribution models, several problems arose at once: overestimating the channels that were at the end of the funnel (the problem of incrementality), and underestimating those that were at the beginning.
To solve this problem, the team decided to study other attribution models available in Google Analytics and select the appropriate ones for their particular case.
Let’s discuss the different attribution models that Google Analytics has and see how using different models will affect the performance picture of different channels.
We will use Google Analytics as an example of one of the most popular analytics systems. Most GA alternatives have similar attribution models
How Different Click-Based Attribution Models Affect The Effectiveness of Channel Evaluation
Below is a list of the most common attribution models used in Google Analytics (GA). There are more complex algorithmic models too, but we will not discuss them at this stage.
- Last Non-Direct Click. The default indirect referral attribution that GA uses;
- First Interaction. First click attribution;
- Last Interaction. Last click attribution;
- Linear. Linear attribution;
- Time Decay. Attribution taking into account the time of interaction;
- Position Based (U-Shaped). With reference to the position.
Let’s clarify the difference between direct and indirect visits to website:
- A direct visit happens when the user directly types in the site’s address or clicks on a bookmark in the browser;
- An indirect visit is a visit to a website from any channel (search, social media, video advertising), including via a link on a third-party website.
Let’s take a look at these models using the example of attracting one specific user. We will visualize a specific user’s path and apply different attribution models to study how they work.
Above is a chain of interactions of a specific user with the advertising materials of our product. The first visit to the website was a result of advertising on Facebook, the second as a result of emails, the third was Google Ads remarketing, and the last interaction before the purchase was a visit from organic search.
Last Non-Direct Click
GA uses this model by default. The entire sale is attributed to the last indirect channel. In our case, this is an organic search.
Why might this be correct? This is the last channel. It makes sense that this channel gets value.
Why might this be wrong? The visualization shows that other channels were involved in user acquisition. Facebook generated demand, email and remarketing reminded about the product, but it’s hard to say which channel was the most important one.
Applying this model will lead to an underestimation of channels at the beginning of the funnel and an overestimation of organic search. In multichannel interaction, this model greatly distorts reality, so it should be used in combination with other models.
First Interaction
Here, 100% of the value is given to the targeted advertising channel on Facebook.
Why might this be correct? This channel brought the user first, and the others just retained this user. If there was no first channel, then there would probably be no sale. The user would not visit the site, subscribe to the newsletter, not get into the remarketing lists, search for the store on the web, and make a purchase. As a result, the user would not know about the existence of our product.
Why might this be wrong? Considering that the sales cycle is long, it is highly likely that the user could have forgotten about our product after the initial Facebook ad. In such a case, the FB ad would not be the cause of the purchase. Just because Facebook has done important work doesn’t mean that other channels didn’t contribute to user acquisition. This means that this model can greatly overestimate effectiveness of Facebook ads.
Let’s continue exploring other models.
Linear
The linear model evenly distributes the value across all channels.
Why might this be correct? We assume that all channels were involved in attracting users, so this distribution is fair. The model is easy to use and minimizes damage caused from underestimating or overestimating channels.
Why might this be wrong? The problem of underestimating and overestimating traffic remains. There is a risk of underestimating the channels that played a decisive role in attracting the user, which, in turn, can lead to stopping a channel or applying improper scaling. And this will already lead to direct or indirect loss of money.
Time Decay
This model factors in the age of users’ interactions with channels and distributes the value progressively. The earlier a user interacts with a channel, the less value that channel receives.
Why might this be correct? People tend to remember recent interactions and forget what they saw a long time ago, so it’s fair to give more value to the channels involved at the end of the funnel and played a decisive role.
Why might this be wrong? It was these long-standing channels that could have been the most significant contribution to building the user’s trust and relationship that led to the purchase. Underestimating early channels can lead us to unnecessarily degrade the channels that generated demand for the product and helped us get through the difficult stages of the acquisition funnel. This will lead to a loss of synergy of the channel combination and a deterioration in the overall results.
Position Based
The U-shaped model distributes value in favor of those channels that introduce the user to the product and close the sales funnel.
Why might this be correct? The first and last channels truly played an important role in the user’s story, introduced them to the product and closed the funnel at the right time.
Why might this be wrong? We underestimate the channels in the middle, although they can play an important role in a complex funnel. Reducing their costs can lead to users dropping out at the most difficult stage of the funnel. At the same time, there will be no channels that can return the user and push further.
Using multiple attribution models will help define the boundaries of channel effectiveness
We’ve looked at a number of options for click-through attribution models. Each of them has its own pros and cons. In different situations, different options may be more objective.
But it’s important to remember that the goal of the attribution model is to help you make an effective decision, not to accurately distribute user value across channels (this is not possible).
Different attribution models reflect reality in different ways. When dealing with tasks of medium difficulty and higher (for example, with a long transaction cycle or several advertising channels), using only one model can lead to false conclusions and erroneous decisions.
Using multiple attribution models allows you to see the situation from different angles, build the upper and lower boundaries of the effectiveness of channels in the marketing mix, and therefore make an informed decision about their optimization or scaling in the context of your business objectives.
In reality, teams gradually develop the most effective decision-making methods by experimenting and observing the impact of different decisions on the overall results of the company. Such observations are made through the use of different attribution models.
The human brain is quite good at learning and mapping input data to the final result. Just as you don’t need to calculate the exact angle you need to turn the steering wheel of the car to avoid an obstacle, you don’t need to know the exact performance of each channel in order to grasp their relationships and make the right decision (as we discussed earlier, there aren’t many options).
Factors influencing the choice of attribution models
In order to determine the list of attribution models suitable for solving your business problem, try to answer the following questions:
- What is your product’s sales cycle?
- What are the specifics of your advertising channel?
- How many channels are you using at the same time?
- What marketing challenge are you solving?
Sales cycle
Understanding the cycle of a deal helps you evaluate how long you need to interact with users from the moment you meet them to the moment they purchase your product.
If you have a short sales cycle, then the Last Click attribution model might work for you, as this interaction will almost always end in a purchase.
But with a long sales cycle and a mix of several acquisition channels, the Last
Click attribution model will not work. At the very least, it should be used in combination with other models.
Here are a few of examples:
Short transaction cycle: You are selling “wow goods” at a low price. The client sees the ad, clicks on the link, and makes a purchase.
You are promoting a casual mobile game on Facebook and people install it right after they click on the ad.
Long transaction cycle: You are selling a car. The purchase is expensive, so the client is making a long-term choice, comparing different brands and models. You have to be constantly visible and provide added value to nudge the consumer’s choice in your favor. Such a cycle involves the use of different advertising channels and multiple interactions with the user from the moment of acquaintance to the final purchase. This complicated cycle makes it difficult to assess the impact of a particular channel.
The specifics of the channel
Each channel influences the user in its own way. Some channels directly encourage the user to click on the link and proceed to the landing page (e.g., search ads or targeted ads on Facebook), and some channels are embedded in the content and natively convey messages about the brand and product value.
Channels that have an explicit CTA (call to action) with a link are easier to work with. The interaction is recorded as a GET parameter in the user’s logs, and the interaction history is saved. Most Click Attribution models are suitable for evaluating these channels.
Channels that don’t provide a clear link to follow are difficult to measure because they leave no visible digital footprint. Classic click-based attribution models won’t work for these channels. You will have to use other methods such as spike analysis, post-purchase polls, regional tests, tests with channel on and off. This is a topic for one of the future articles in the series.
Number of channels
If you are using one advertising channel, it is quite easy for you to track its impact. All sales are likely to be attributed to this channel (even if not all interactions were recorded). There are exceptions when a lot of traffic comes through referral models or word of mouth, but evaluating one paid channel won’t be difficult.
If you use many channels, then even with a short transaction cycle, several channels may be involved in attracting a user to a purchase, which complicates the assessment.
Often channels in combination reinforce each other’s results, which is also difficult to measure. For these cases, it’s best to use multiple attribution models to understand a realistic distribution of value over time.
Business objective
Marketing teams solve different problems: selling products, building brand awareness, retaining or returning users, attracting leads, and many others. Each of these tasks has its own purpose and evaluation criteria.
Here’s an example: Imagine that your task is to bring a new product to market. You need to generate demand and then collect it with performance channels. Your (wrong) actions:
- You set up display ads, but you evaluate their effectiveness not by the number of users who learned about and showed interest in the product, but by the number of purchases generated by the channel using the last click attribution model;
- Optimization of a display ads for a purchase immediately made whole acquisition funnel ineffective. You are trying to optimize display ads for the action that happens after you generate the demand. But it has not yet been generated.
- Consequence – there is no awareness built. The demand has not been formed. Performance channels have nothing to collect. The task has not been completed.
A better approach would be to optimize ads for user familiarity with the product using the First Click attribution model. In this way, you would optimize your campaign for people who saw the ad and showed interest in the product by going to the website. At the same time, sharpen performance channels to work with those users who showed the maximum interest, and convert this interest into targeted actions.
The effectiveness of these channels should be considered together, since they worked in combination.
The same advertising channels can solve different marketing problems, so you need to select attribution models based on the combination of channels and business objectives.
When to rethink the attribution model
Effective management of advertising channels is possible when you assess their effectiveness well. Above, we reviewed the key factors that affect the assessment of the channel’s effectiveness:
- What is the sales cycle of your product?
- What are the specifics of the work of your advertising channel?
- How many channels are you using at the same time?
- What is your business objective?
Understanding these parameters allows you to roughly define the requirements for the attribution model or models. When these parameters change, it makes sense to reassess the attribution model and change it if necessary.
It is useful to separately keep an eye on a number of metrics and indicators that may signal that the current attribution model is becoming less effective.
Here are some of them:
- Comparison of the sum of of new users or revenue attributed to different channel with the overall performance of the company. If the channel overlap exceeds 15%, then it is worth re-checking the attribution models. As a reminder, overlap is an effect that occurs when you attribute one sale to multiple channels, resulting in a difference between the actual profit you have and the amount of profit attributed to different sources. If overlap exceeds 10-15%, this is a signal that your channels have begun to overlap strongly, problems with incrementality have appeared, and it’s time to think about changing the attribution model.
- Sharp changes and jumps in the number of organic users (it is highly likely that some of the users are not attributed to their channels).
- Differences between internal analytics attribution models and ad network models.
- Deterioration of some channels after the shutdown or strong optimization of another channel.
- Stopping the growth of organics when scaling the channel that affects organics.
- Monitoring the correlation of channels without taking into account their mutual influence.
The process of selecting the optimal attribution model for the current situation is an experimental process. In order to get an understanding of what is happening, you need to conduct experiments. For example, use multiple attribution models for each marketing mix to get a range of values from their application. This will help build the upper and lower bounds for the effectiveness of the channels.
But don’t get too carried away with the work of creating the perfect attribution model. A healthy balance between model accuracy, efficiency, and decision-making speed is critical.
- If you need to spend a lot of time and effort to slightly improve the attribution model, which in turn postpones the work of scaling a promising channel, this is a bad trade;
- If your model doesn’t reflect reality and you are confused about what works and what does not, then investing effort in building an attribution model will be a good solution.
Real case of the evolution of the attribution model
As a real example, let’s look at how our company evolved its approach to using various models based on the needs for traffic assessment, which are constantly changing.
As a reminder, Letyshops is a cashback service. We return to users a portion of the money they spend on each online purchase. We also act as a reliable partner for stores, helping them attract and retain price-sensitive users.
The first version of the attribution model. Build your own attribution model
At the time of product launch, we weren’t very good at attribution models. Since we were more afraid of underestimating our channels than overestimating them, we made the decision to attribute 100% value to each channel that took part in user acquisition.
At the start of the development of the project, we scaled through one growth channel, while we were just testing other channels. The crossing of the channels was small. Scenarios involving multiple channels were usually test scenarios. At this stage it was better for us to overestimate the channel than to underestimate it.
This model helped to reflect the real situation and scale business. For a long time, we used this model as our main solution. At the same time, we monitored overlap to avoid channel incrementality.
Now I can say that this attribution model was not perfect, but its simplicity allowed us to make quick decisions and scale over time. At the same time, we had an indicator (Overlap), which signaled when something needs to change.
The second version of the attribution model. Added an additional model
As the marketing mix scaled, the crossover grew, and so did overlap. It quickly exceeded 15%, so we decided to add another attribution model.
We modified the base GA Last Non-Direct Click model to Last Paid Non-Direct Click. Why? All of our organic traffic was branded, and that, in turn, was the result of paid channels. Therefore, we decided to give all the value to the last paid channel in the chain.
We added this model to the existing one. This made it possible to estimate the range of values by channel efficiency. The first attribution model gave an estimate from the top, the second model gave a certain minimum extremum of the channel efficiency.
Based on the obtained ranges of values, we were able to adjust our decisions and reduce the problem of overestimating some channels, which allowed us to free up budget for channels with greater efficiency.
The third version of the attribution model. We spent a lot of time on algorithmic attribution models but got no results
For a long time, the main driver of Letyshops’ growth was influencer marketing on YouTube. We did native integrations and provided links for bloggers without obvious signs of advertising. At first (in 2016), this tracking method worked well: the blogger gave an organic recommendation on a useful product from personal experience, and people followed the links and registered. Back then, the proportion of people who watched YouTube from the desktop was much higher.
Organics grew, but for a long time, the old attribution models were enough for us to evaluate the advertising channel and scale it.
But then YouTube’s audience underwent some serious changes. The share of mobile traffic grew, and viewers’ trust in YouTubers diminished. At some point, referrals from links embedded in the description of YouTube videos dropped significantly. But at the same time, after coverage by major bloggers, we have always seen a sharp increase in organic traffic.
We soon realized that the picture we saw through attribution models was far from reality. We understood that organic is a consequence of the work of the YouTube advertising channel. But we couldn’t accurately measure its share. The marketing mix was already very large, and we didn’t know what kind of organic contribution a particular YouTube campaign was making.
We went into building complex solutions, starting to explore algorithmic attribution models (Markov chains and Shapley vectors). But we forgot about one important condition for them to work: we must have the entire history of user interaction with the channel. We didn’t have it.
As I explained above, people saw ads on YouTube and then installed the app or went to the site without using the link in the video description. This made click-based models useless, even very complex and advanced ones.
Ultimately, we sorted out the problem by running tests of the channel in new regions and assessing the ratio between those who come from the links under the video and those who come in other ways. I mentioned this case in the previous essay of the series.
The next versions. Incrementality, Ad Impact on Old Users, and Other Effects
I will not dwell on future changes in our attribution model, since they are specific to how our product works (four platforms: iOS, Android, desktop, browser extension) and a complex marketing mix on a budget of millions of dollars per year.
But changes at different levels forced us to change attribution models on a regular basis—it’s an ongoing process.
For example, at a certain point, we noticed that YouTube ads not only bring new users, but also revive old ones (this manifested itself in the effect on old cohorts). The question arose: how to correctly reflect this value when assessing investment in the channel?
Then large platforms began to limit the possibilities for adequate tracking, which again forced us to revise the attribution models we used.
Another example: with the growth of the product and its popularity, as well as the share of channels without direct click, the problem of incrementality became even more urgent. And to solve it, we had to resort to using attribution models of advertising systems, conducting special experiments.
Conclusion, key thoughts
Evaluating the effectiveness of advertising channels is a task that is at the intersection of the areas of responsibility of marketing and analytics, so they should solve it together and put the business problem to be solved at the forefront.
The attribution model must evolve to accommodate the changing environment. Answers to the following questions will help in choosing a model:
- What is your product’s sales cycle?
- What are the specifics of your advertising channels?
- How many channels are you using at the same time?
- What marketing challenge are you solving?
Here’s what you need to do to recheck and revise your attribution model in time:
- Pay attention to organic traffic: Acquisitions from paid channels are often disguised as organic. Organics can be an indicator that your attribution models are no longer objectively assessing what’s going on.
- Pay attention to Overlap. If it exceeds 10-15%, it’s a sign that your channels have begun to overlap strongly and you’re having incrementality problems, which means it’s time to consider changing the attribution model and conducting incremental tests.
- Consider changes in the external environment. In the cookieless world, or in terms of IDFA opt-out on iOS, the rules change. The history of user interaction with advertising channels will become thinner and thinner, and classic models will become less and less accurate.
- Pay attention to changes and spikes in the number of organic users (it is highly likely that some users are not attributed to their proper channels).
Pay attention to the differences between internal analytics attribution models and ad network models. - Pay attention to degradation in channel performance after disconnecting or heavily optimizing another channel.
Different attribution models reflect reality to one degree or another and allow looking at the situation from new perspectives, which increases the chances of making the right decision. You need to try different attribution models. Different channel combinations will give different synergies. Finding the optimal model can only be done experimentally.
An important condition for the adequate assessment of your attribution model is to strive for a complete picture about users and the history of their interaction with the advertised brand or product. At the same time, we do not have the absolute completeness of the data, which makes it impossible to accurately solve the problem of assessing the channel’s contribution to user acquisition.
Consider incrementality when working with attribution models. Without this, there is a great risk of overestimating the channel and losing money.
Remember, the attribution model doesn’t need to be seen as a means of getting accurate data on the distribution of value across channels – it is often not possible. The attribution model is a compass, not a means of obtaining accurate coordinates. For solving a business problem, reliable benchmarks are more effective than accurate numbers obtained in an unreliable way.
