Paul manages Churney's relationships with performance marketing agencies. He previously worked in several digital marketing positions, with the last one as a marketing science partner at Meta. Paul holds a higher diploma in Psychology from Trinity College Dublin. 

Paul Fagan

Agency Partnership Manager

Paul

Noy Rotbart is a co-founder of Churney. He previously served as a data science manager at LiveIntent, architect at the innovation department of Nykredit, and lead data engineer at Save the Children. Noy holds a PhD in Computer Science from the University of Copenhagen.

Noy Rotbart

Co-founder & CEO

Anna serves as the Head of Product at Churney. In her previous roles, she has built product teams and managed product roadmaps across various SaaS companies. Her expertise stems from her experiences at Nykredit, Topdanmark, and several startups.

Anna Bigaard

Head of Product

Anna

A/B testing is a widely used method to evaluate the effectiveness of different solutions. It is often employed in a continuous workflow consisting of three main steps: identifying a target segment together with potential treatments to drive a desired outcome, conducting an A/B test where the different treatments are given to randomly assigned users, and lastly, analyzing the results to identify the most effective treatment. If a new treatment is successful, it is implemented. If it isn't, the process is repeated, and the treatment designs are refined.

 to demonstrate a significant improvement. An understudied cause of many failures is that A/B tests bundle harmful effects with desirable ones and provide the average treatment effect. These potentially harmful effects for a subset of the targeted users can mask or dilute the desirable effects in the final analysis, leading to good treatments being abandoned just because they aren’t the right choice for everyone. Often, even when this problem is understood, marketers attempt to identify sub-segments for whom a given treatment will work based on intuition and gut feel. 

Even when successful, A/B tests tend to optimize for the short rather than the long term. Marketers may be tempted to implement strategies that yield immediate results, even though their real objective is to ensure long-term customer retention and loyalty. A focus on short-term gains can sometimes lead to strategies that are detrimental in the long run.

Causal AI offers a new approach to overcoming these limitations. This method makes it possible to personalize A/B testing, where the goal is not just to discover the best-performing variant on average but to identify the optimal variant for each user.

It works iteratively and starts first by running what looks like a traditional A/B test. As the first results roll in, the causal machine learning algorithm identifies patterns and trends in user responses to the various treatments and answers the question, “What would have happened if the user got treatment X instead of Y?". From this, we can create a 

 of the individual treatment for each user and change the 

With each iteration, the algorithm learns the optimal user assignments. This allows for reliance on a continuous AI system for making better decisions about who should receive which treatment instead of relying on a one-size-fits-all mindset and intuition-based segmentations. 

Three conditions are vital for the success of automated, personalized A/B testing.

To summarize, a successful implementation requires a substantial number of individuals to treat, data demonstrating a good conversion rate, and significant differences in treatment effects across various user options. Lastly, we recommend choosing an outcome revealed within a relatively short timeframe and performing a mediation analysis to ensure that improving the short-term outcome also improves the ultimate long-term objective.

Churney's specialization in causal AI provides companies such as Podimo, Beer52, and Mindvalley with double-digit increases in their return on investment. It also  helps them achieve their business goals without wasting time and money. A/B testing is suboptimal, intuition-based.

We highlight the issues of A/B testing for marketing purposes and give some of our experience on using causal AI to improve this process dramatically. 

The Limitations of A/B Testing and the Rise of Causal AI

The issue with A/B testing for marketers

Is there anything an advertiser might be more protective about than sharing its data?  Data is rightly considered one of the most valuable strategic assets. Companies want to maintain control over it to preserve competitive advantage, meet legal obligations, and minimize risks. By leveraging their rich data stores, they can achieve significant growth, as evidenced by 

 from Google and The Boston Consulting Group, which found that data-driven companies can double their revenue and reduce costs.  

On the other hand, advertisers don’t always have the right capabilities to capitalize on this earning potential, leading them to turn to third parties for support. The percentage of organizations that entrust critical and sensitive assets to third parties can be as high as 

.  For some, it might be to process payments and order deliveries. For others, it’s the potential to use data to advertise more effectively, such as better ad targeting, personalization, and measurement & optimization. These activations are made possible with access to the right

and advanced skill sets, e.g., such as statisticians, data scientists, and experts in specific activations. But it's not without its risks.

Worryingly, many third-party data breaches can go underreported. A 

 from Ponemon Institute and Mastercard’s RiskRecon found that 59% of participants acknowledge that their organizations have encountered a data breach originating from one of their third parties. Notably, 54% of these incidents occurred within the last 12 months. The stakes couldn't be higher; a privacy beach can disrupt processes, share sensitive data with competitors, and result in reputational damage and significant 

, the most severe reaching up to €20 million or 4% of the firm's worldwide annual revenue.  

Therein lies the advertiser’s dilemma: leveraging data-driven strategies for revenue growth while safeguarding against potential breaches. The potential for a mutually beneficial relationship brings parties together, but the potential for harm drives them apart. The dilemma shows companies cannot be open without the risk of harm, leading to cautious and tentative partnerships.

To proceed with cautious pragmatism, then, is to scrutinize and reduce risk in search of something that works. A business must see a third-party risk assessment as an essential activity whose opportunity cost is far less than the potential cost or reputational damage. This article explores questions of how we should weigh the benefits and risks of sharing data with third parties and what are the essential criteria companies should evaluate when selecting third-party partners.

Customers are more likely to share their data with companies they trust and where there is a fair exchange of value, a principle which equally applies between companies. Here are five factors to consider when weighing the potential benefits and data security risks of sharing data with a third party. 

1. A clear advertising use case and goal for sharing data  

The advertiser must establish a clearly defined use case for sharing data to determine if it's worth the risk.  

Consider an advertiser grappling with optimizing ad spending to acquire high-value users on Meta. They need help finding the most relevant customers because they're getting too many low-value customers which are incredibly costly for the business. In short, they need a better, high-value signal to optimize ads. What steps can they take to do better?   

First, they could build a predictive model to learn the characteristics of their most profitable customers. Second, to use that signal in advertising, they would need to send the high-value predictive signal to an ad platform in a language the platform can understand (e.g., a PLTV threshold in a Custom Event) and within a short time frame (ideally within 24 hours). Third,  the platform needs optimization for that signal (e.g., Custom Event Optimisation).

Finally, you would need to play around between the prediction quality, delay, and how Meta uses the predictions. There's a high time, risk, and spending cost from the inevitable trial and error you must go through when finding the optimal setup. This is all to say that even if your company has incredible technical talent, it could benefit from the expertise and experience of a partner with a successful track record of going through this many times before.

In the case of our advertiser, they’ve identified a clear business use case and goal for sharing data: the effective use of advertising and analytics. Their business, like any other, is only as valuable as its future cash flows. However, here they've identified a skill gap where third-party expertise could play a substantial role in optimising for higher-value customers on crucial ad platforms thus securing these future cash flows.

2. Only share the data assets necessary for the task at hand

Once the need and use case is identified, the question will turn to the kind of data involved in the processing, its value, and how sensitive it is. 

A machine learning model, whether Regression, Buy Till You Die, Random Forest, Neural Networks, Markov Chains, or Gradient Boosting, will automatically learn and quantify the importance of inputs to make educated guesses on what to expect from new data.  It’s typical for a machine learning model involved in predicting lifetime value to include the following input data from a client’s data warehouse, information as advertiser sensitive and confidential as it gets:  

Given the hyper-sensitivity of this data and the risk it poses, so too should the level of scrutiny of third parties with access to it.  It begins with adhering to the principle of data minimization, which means collecting and processing the minimum amount of data required to deliver the service used for the purposes explicitly stated. Data minimization helps limit the scope of potential damage and protects all parties in the event of a data breach. 

Check regulatory compliance and safeguards

Advertisers now operate in a privacy-centric environment with increasing legal and technical constraints on data, changing how identity is processed and used. 

Historically, the success of digital marketing is due to the ability to identify people across sites and apps. With consumers active on more media platforms, devices, and channels than ever, the ability to unify this data into a single holistic view of the customer could help determine marketing efforts including where and how to invest a limited budget. 

However, the ability to resolve identity is now increasingly restricted. New policies are giving people more options to limit how their data is shared with advertisers, people are choosing to opt out of receiving ads on websites and platforms, removing identity and grouping the data shared with advertisers. To put it into perspective, approximately 

of nations worldwide now have either enacted protection laws or drafted legislation, e.g., Europe's General Data Protection Regulation (GDPR), 

Brazil's General Data Protection Law (LGPD)

, Canada’s Consumer Privacy Protection Act (CPPA), 

the US's Multi-State Privacy Agreement (MSPA),

The tasks for your organization are to know how all parties interact with your data, establish regulatory compliance and safeguards, and honour customers' privacy choices. Only then will you be able to successfully manage the risk. 

Regulations require organizations to implement technical and organizational measures based on the user data's risk level to meet all requirements and assuage security concerns. The following rules are just the ante required to play the game: 

The implications of these policies also require each party to understand their obligations, responsibilities and liabilities. For instance,  working with a predicted lifetime value provider in the EU or UK will begin with a 

The agreement codifies that if the third party is at fault, they will be liable for any damages caused by their data processing.  This framework ensures a transparent and accountable relationship, while also emphasising the need for due diligence in data handling to safeguard the interests of all parties. 

We recommend an isolated instance with your cloud service provider, e.g., Google Cloud Platform (GCP). A third party should manage each client in a separate data warehouse instance in an area of their choosing.

Sharing information stored in web cookies with a third party, known as cookie exchange, used to be a widespread practice among Demand-Side Platforms (DSPs), advertisers, and publishers. While not illegal yet, this common practice is disappearing from the ad tech world. New initiatives starting from Apple and moving to Google have stripped cookie harvesting abilities even from the browser side. This upcoming omission makes the ability to summarize and share high-quality first-party data much more powerful in the long term. 

Regulations tell a company what to do but not always how to do it, which is why frameworks such as 

 (SOC 2) exist. SOC 2 is a voluntary standard whose certification signals a company's adherence to strict standards for safeguarding their systems and data's security, confidentiality, integrity, privacy and availability. Particularly crucial for entities handling sensitive information like customer data or financial records.

SOC 2 certification is good for everyone involved:  

Overall, SOC 2 sets the gold standard for data security, compliance and advertisers’ peace of mind when working with third parties.  

More organizations than ever are sharing data with third parties for access to advanced capabilities. However, given the risk of a potential breach or misuse of customer data from third parties, it necessitates constantly weighing the benefits and risks of such partnerships and assessing whether a partner is a good steward of our customer’s data. Remember the following key questions to ask of any prospective partnership:

Marketing leaders grapple with the challenge of optimizing revenue through data-driven strategies while protecting against data misuse and privacy breaches. To navigate this, this article outlines four principles to handle this complex trade-off. 

The advertiser’s data sharing dilemma

The advertiser’s dilemma: essential criteria advertisers should evaluate when selecting third-party partners

As third-party cookie depreciation looms, 

businesses are turning to first-party data to understand and engage with their customers in a privacy-first way.  A recent 

Gartner survey of marketing leaders found that 82% of marketing leaders are prioritising first-party data to create immediate value for customers, such as strengthening signal resilience and future proofing their data sources (1). B

ut first-party data is more than just a defensive strategy. Businesses are also using it to drive better business results through a variety of use cases such as improved ad targeting, personalisation, and measurement and optimisation (2, 3). Those that learn to harness first-party data will have a significant advantage, because understanding and increasing customer value leads to increased profitability.

What does a strong first-party data strategy look like?

Identifying the most profitable customers early on and understanding what makes a customer loyal allows businesses to maximise profits and minimise losses by avoiding customers who are unlikely to be profitable in the long run. This can be done by using first-party data to 'bolster connections with existing customers and identify new audiences to reach' e.g.  Audience Exclusions, Deterministic Customer Segmentation, Model-Based Targeting. 

 With that in mind, think about about the following questions: 

According to Deloitte, customers are '69% more likely to purchase form a brand that provides personalised, frictionless experience throughout their purchase journey' (3).  Businesses can use

 past behaviour from first-party data to build tailored omni-channel experience across digital and offline touch points via Dynamic Creative Optimisation, Omni-channel Personalisation, Custom Product Recommendations. 

Marketers face the constant challenge of proving how their marketing activities drive incremental business outcomes. They can incorporate first-party data into their measurement strategies to provide accurate visibility into the performance of their marketing efforts. e.g. Server-to-Server Connections, Granular Marketing Mix Modelling, Cloud Environment for Data Collaboration. Deloitte found the measurement front-runners to be 44% more likely to beat their revenue goals.  

Here are the paper's recommended actions:

Marketers who are new to first-party data may find the initial steps to be daunting. While using first-party data can provide significant benefits, it requires a number of enablers to be in place, such as a customer-centric philosophy, data unification, technology infrastructure, and sophisticated analytic resources. These enablers can be costly, but they are essential for reaping the full benefits of first-party data 

Business will find themselves better equipped with some enablers than others. Some lack the infrastructure (unique customer identifiers, Customer Data Platforms etc.) and others might lack the technical and analytics skills to activate a use case like modelling user data (statisticians, data scientists, data engineers). Should you want to evaluate the incremental value of your new high-value based audiences, then you’ll also need to integrate with, and experiment on, advertising platforms - which in our opinion is often the hardest part of the process. 

Starting out with first-party data can be challenging, but there is a growing ecosystem of partners who can help you bring your first-party data use cases to life. A good partner will have the experience and expertise to help you avoid common pitfalls and achieve your goals.

To give one example from the paper, we see a fashion brand with a ‘niche target audience who wants to expand to new customers and drive them to their online and brick-and-mortar stores’ (3). The in-house data science team worked alongside an analytic partner to:

As a result, the experiment demonstrated how the ‘new first-party audiences drove higher engagement and return visits compared to standard audiences, paving the path forward for the brand to build relationships with new customers’.  If this kind of lifetime value use case excites then check out a similar paper: 

‘Predicting lifetime value: How adopting customer lifetime value strategies leads to profitable growth

Collecting, managing, and using first-party data in a privacy-centric way can lead to better business outcomes. It can be used for a variety of purposes, such as improving ad targeting and personalisation, and better measurement and optimisation. However, activating these use cases may require technical capabilities that your organisation does not currently have. In these cases, it is worth reaching out to a partner for support.

No matter where you are in your journey, there are actions you can take to improve. Read the 

Inspired by a recent paper by Meta and Deloitte, this article discusses the importance of first-party data in a privacy-first world and how marketers are using it to improve ad targeting, personalisation, and measurement. It also provides recommendations on how to enable these use cases and a case study of a fashion brand that successfully used first-party data with the help of analytic partner. 

How are leading marketers using first-party data to drive business results? 

Rohit works with developing data visualizations and data-driven applications. He has a background in Interaction Design from CIID and IIT Guwahati. Before Churney, he worked at Datarobot and Leapcraft.

Rohit Sharma

Lead Front-end Engineer

Rohit (off)

With the phasing out of third-party data, many companies use the various conversions APIs to share meaningful data to improve their ad spend. Presently, they target conversions for deterministic events such as subscription made. This way, all conversions are treated equally and are used as target for the ad platform. The more sophisticated companies would differentiate between these conversion events to differentiate between higher value subscriptions or even previous indicators for high-LTV users. However, early signs for profitability can be quite misleading in the long run. To illustrate this situation, we made an info-viz tool for our clients to show the hidden potential in their audience.

We first order clients according to three parameters:

We now take the top ½ of every such sorting of the population and plot their revenue below.

The marketer with a crystal ball would have chosen to focus on C and have to settle for determining based on what they know, which is A.

Setting the time bar to one month reveals that A starts off much more promising with an average 1-month LTV 35% greater than both B and C. As we increase the time passed, the truth of the gap in performance reveals itself, as all three roughly equate within three months and arrive at a pretty dramatic difference of +25% between C and A, and +10% between B and A.

As we onboard our clients, it is helpful for them to be aware of the comparison between A and C in their case to determine if there is a point in producing the pLTV signal to help win this performance gap. It also demonstrates that in the setting of recurring revenue, one should be careful not to optimize for the wrong metric, thinking it is a good neighbor to actual revenue, and that computing and using a more sophisticated metric is highly valuable.

Here's how you can determine if your business case lends itself well for predictive LTV to improve your ad spend.

The case for pLTV in user acquisition

Brian has a PhD in Computer Science from the University of Copenhagen, where his research focused on the intersection between applied machine learning and causal inference. Before Churney, he worked as a research scientist at Spotify, where he initiated Spotify's efforts to maximize the impact of promotional campaigns using causal machine learning.

Brian Brost

Co-founder & CTO

Brian

Roughly speaking, growth teams have two major responsibilities: (i) to acquire the best possible customers at the lowest possible costs, and (ii) to maximize the LTV of their existing customer base. In this blog post, we focus on the second problem, and why predicting churn is a useful starting point, but inadequate to solve the problem on its own.

 demonstrated that a 5% increase in retention can be expected to lead to a 25%-95% increase in profit. Reducing churn has such an out-sized impact on profit because the effects of an increase in retention compound over time. As a result, a natural starting point of any LTV maximization strategy is to focus on reducing churn.

If you work at a subscription company, one of the simplest things you can do to stop your customer from churning is to offer them a discount or some other incentive when they attempt to cancel their subscription. This form of reactive churn prevention often comes too late to change the customer’s decision to stop using your service. Indeed, our A/B testing has shown that the same discount offered after customers try to leave can be less than half as effective as providing that same discount proactively, before the customer has tried to cancel.

Of course, such a proactive approach isn’t possible if you can’t effectively predict who’s going to churn, at least not if you don’t want to offer unnecessary discounts to customers that were happily using your service. Strong predictive models are therefore a prerequisite for proactive churn prevention. They aren’t enough however, and continuing with our previous example, providing a discount to the users deemed most likely to churn can also be counterproductive.

Customers who are very likely to churn may not be possible to save. In the worst case, as illustrated above in the quadrant labelled 'harmful', a user that appears very likely to churn might be reminded to cancel their subscription by a discount offer, earlier than they would have done otherwise. For a customer that’s going to leave regardless of whether you offer them the discount, you will have wasted whatever resources you spend trying to get them to stay. Ideally, you would target only the customers from the 'desirable' quadrant, the ones that would leave, were they not offered the discount, but who would stay if provided the discount.

Unfortunately, predicting which users belong to that 'desirable' quadrant is a difficult problem, requiring expertise in both machine learning and causal inference. The problem boils down to predicting so-called heterogeneous treatment effects. While it may seem like magic to claim to know who will respond to a given treatment, it is possible with a bit of intelligent exploration.

The basic solution is to run an adaptive, targeted A/B test. Starting with an initially randomized A/B split, you need to iteratively estimate for which users the treatment worked using double machine learning. These treatment effect estimates are then used to more intelligently target the treatment. This treatment effect estimation has only become feasible for complicated problem settings thanks to recent advancements in double machine learning based causal machine learning, which enable modern machine learning methods to be combined with rigorous and unbiased causal inference.

As an example of how well this can work, we worked with a client that had unsuccessfully tested a discounting strategy for users that they believed were likely to churn. Their initial targeting resulted in only a negligible increase in retention, and actually reduced revenue. Within less than 4 weeks of adopting our strategy, they had identified the users for whom the discounting actually worked, yielding a 30% increase in retention, and a 17% increase in revenue per user.

Another advantage is that the framework of treatment effect prediction is more general than churn prediction. If you focus on the predicted treatment effect on lifetime value, you can optimally target many treatments which wouldn’t make sense to think of if viewed through the narrow lens of churn prediction. Thus, rather than just trying to prevent churn, treatment effect predictions can be used to maximize the return on investment for upsell and cross sell campaigns, and more generally, to maximize the impact of your retention marketing efforts.

We hope this article will convince you that causal machine learning is the right framework for optimizing your retention marketing. In the coming weeks and months we’ll be sharing more material on how to execute in this challenging, but rewarding domain.

Excessive churn is perhaps the greatest threat to the healthy growth and profitability of any recurring revenue company. More precisely, the real pain point occurs when your customer acquisition cost is greater than their lifetime value (LTV).

Predicting churn - or predicting what you can do about it?

Christian has a PhD in Computer Science from the University of Copenhagen, focused on machine learning for sequential data modeling applied to search and recommender systems, with dozens of publications and more than 600 citations. He has done data science consulting and worked in multiple startups, where he was a co-inventor of different patents on machine learning for recommender systems.

Christian Hansen

Co-founder

Christian

The task of early LTV prediction can be phrased as:

The first part is easily understood; the better the estimates, the better your ad platform can focus the ads on high LTV customers. But for the second part, why is it essential to use as few hours as possible? By reducing the time between a user first appearing and you sending an optimization event back to the ad platform, the platform can learn to identify and target valuable customers earlier; in addition, if the event comes too late, the ad platform may be unable to use it all because of restrictions in their optimization algorithms and limitations on their ability to match the user correctly. Thus, it is not only essential to be able to provide accurate estimates of a given user’s LTV, but to do so quickly. This in turn means you need to be able to model the certainty of your predictions, since you'll want to hold off on sending an uncertain prediction too early for ad platforms that don't allow you to send updated predictions. If you send your predictions when your certainty is low, you may risk optimizing towards low LTV users!

At its core, LTV prediction is a classical time series regression task. At a given prediction time, the user has performed a number of interactions on your platform or service, ideally they have also completed an onboarding workflow that provides descriptive user information, and you want to predict all future value the user may generate.

This seems relatively straightforward for most data scientists. You simply construct a time-based dataset, and use your favorite choice of either classical time series models like ARIMA, or get fancy and employ a recurrent neural network or transformer-based architecture, feed it the data and try to predict the user's future value. What could go wrong? Below we will go over some of the problems that can crop up.

Sending the predictions to the ad platform too slowly

So you have trained and validated your favorite model and it performs well. The loss on the test set is low, and you even took the time to look at some ranking metrics to make sure you're actually able to differentiate between the best and the worst users, a sanity check which can be necessary because often the vast majority of users will have a lifetime value of 0, and naively applying only regression metrics can hide the fact that you've simply trained a model that predicts every user to have a value of 0. But now you are facing a big problem, for some users you may be able to confidently predict their LTV after a few hours, while others may require multiple days before the model is sufficiently certain. How do you control the trade-off between sending the event after 4 or 24 hours of data? There is no simple answer to this dilemma, and the optimal choice depends both on the ad platform, since some tolerate delays better than others, and on the exact improvement in prediction quality resulting from an additional delay. If you don't have past experience with the ad platforms to fall back on, rigorous experimentation is a necessity. 

Deployed model performance differs from your offline evaluation

You have deployed a model trained on a large variety of time horizons for users, and offline evaluation convinced you that the model was ready to be deployed. Still, at inference time after deployment, the predictions vary significantly from those you saw in your offline evaluation. Either of the following reasons normally causes this:

When you trained the model, you did the straightforward thing of looking at a bunch of users from e.g. January to June, found their events in the first 12 hours, and computed the total value of purchases each user made in the future. On the outside, this looks correct, but if we are assuming it is currently September, some of the users have had over twice as long time to accumulate purchases compared to the users at the end of the period! What we see here is the classic problem of right censored data, which means that not all users have data that is observed for the same amount of time. Of course, we could naively fix this by only using data from January, so each user would have had eight months to accumulate value, but then we throw away a lot of useful information. A preferred training strategy would be to use techniques from survival modeling which allow us to utilize users of varying observation periods, and to get the most accurate predictions.

You have made sure you have a model that works at several time horizons from hours to days, you have secured that your data looks the same at inference time as during training, and your training strategy takes into account each user's individual observation period length. What can now go wrong? The distribution of new users suddenly differs due to a change in marketing strategy, or a major update went live, which changes the behavior of the users greatly, and suddenly the trained model is just plain wrong in its predictions. It is important to detect these data drifts so we don’t send out wrong predictions, but we also need to handle it. As the data just drifted, we will only have a few users to train a new model on, and none of these will have a long observation period for accumulating value. Here it is paramount to be able to assess if the existing model can be updated to reflect the new users (e.g. by removing part of the feature space that is no longer relevant due to changes on the platform, but hope the rest stays the same), or if a new model have to be trained. If a new model has to be trained, it is important to be able to relate short-time metrics (e.g., initial activity or general product interest) to LTV, so the models can switch targets to instead of predicting LTV, predict valid proxies for LTV that correlates significantly with LTV, so these can be used for a time period while new data is collected.

Adversarial attacks (less susceptible than standard conversion events)

We have a great model, and we have evaluated it using all the best regression and ranking metrics we can come up with. In addition, we tested it in a lot of scenarios from suddenly losing access to certain data sources or changes in distribution. We now feel very confident that we can reliably send events to the ad platforms. One test we may have forgotten about is how susceptible our model is to adversarial attacks from competitors, which aim to mess up the type of new users found through an ad campaign based on the sent events to the ad platform. If the model learns to pick up on behavior that may be reproduced by an adversarial bot. For example, a TV streaming service gives a free trial, and a user's initial consumption is greatly correlated to their future LTV. In this case, a bot could simply sign up with a free trial and watch a lot of TV shows, which the model would mistakenly believe is a high value user and send an event to the ad platform to find more of this type of user. Here we have two aspects to consider, one from a data point of view and one from a modeling point of view:

Note that even though predicted LTV is subject to adversarial attacks, it is significantly more robust than using a standard conversion event, such as a single free trial event. To prevent and detect such attacks, robust outlier detection methods are necessary, which we will dive into in a future post.

This article is about some of the pitfalls of early lifetime value (LTV) prediction that should be avoided to get the most out of value-based event optimization or value-based-bidding strategies provided by companies such as Meta, Google, or TikTok.

Early LTV predictions - 5 pitfalls you want to avoid

We begin with the table in which each row contains the subscription ID, start, and end date. If the subscription is still ongoing, the end date is blank. For convenience, we bound each subscription ID with some properties such as frequency, age, inbound channel, country, and marketing allowed.

The first column we need to define for each row is the number of payments made, a function of the current time, subscription start date, and end date. We call 

. We also need to compute the number of payments possible with a similar parameter without the end date, which we call 

. If no pauses are allowed, and the frequency is identical across the subscription (assume monthly), this we can compute it as

Num_potential_payments=(datediff(date(today),start_date),months)

The churn curve is a function computing the percentile of the sum out of the whole population (0-100%, y-axis) over all months made across time (say five years, which are 60 payments possible, on the x-axis). We create the x-axis external using:

CyclesToAttrition = GENERATESERIES(0, 60, 1)

Attrition = [Made X payments]/[Population  size]

 are the total number of agreements that made at least x payments computed as follows.

Made X payments = var CurrentNumber = MIN ( CyclesToAttrition[Value] )
return  CALCULATE(COUNTROWS(data),data[num_payments]>=CurrentNumber)

Population size is the number of agreements that could have made at least x payments, and we compute it as follows.

Population  size =
var CurrentNumber = MIN (CyclesToAttrition[Value])
return  CALCULATE(
                COUNTROWS( data),data[num_potential_payments]>=CurrentNumber)

At this point, the churn curve is complete, and as we filter the audience based on the country, age, etc. the churn curve will be updated accordingly. Furthermore, we can maintain the filters for one curve, keep another curve without filters, and then cross-evaluate populations against the general churn curve.

A churn curve is a handy tool to evaluate your churn in different cohorts/periods. BI tools are not great at collecting time-based data, such as subscription periods, and unifying them. This tutorial will demonstrate how to use a BI tool to arrive at a fully filterable churn curve. We choose PowerBI in this demo.

Blog