Our current model calculates similarity scores between campaigns and user profile vectors. 

The similarity score is the cosine similarity between the content+title of the page and the aggregated content read by the user, as stored in their user profile. 

In the current setup, based on an experiment on a small number of users, we found p(threshold) = 0.7 to be an appealing value for filtering the relevant notifications.

Future work involves further tuning of this parameter based on experiments on a more significant number of users. 

When we run A/B tests, we track many different parameters of the models and business metrics. 

But we generally test the model based on Click-through Rate(CTR) wrt clicks vs. sent and clicks vs. received. 

The test usually runs on millions of users, exploring variations of base ideas. 

We typically have scores of A/B tests running in parallel daily, but the key advantage is that they allow our decisions to be data-driven. 

According to Client data, comparing other push services against Oriel.io, the Client's CTR improved by approximately 4.19X, and the absolute average growth in CTR is around 6.32%(based on data between 01-06 -2021 and 01-07-21).

We invite you to test multiple variations of push notifications for the same subscriber segment and monitor the improvements as you go along.

Happy Testing!