The Product Manager's Guide To Building Notification Systems: Omni-channel Analytics

• Part 1: Product Notification Experience
• Part 2: Decoupling Templates From Code
• Part 3: User Preferences Management
• Part 5: Optimized Automation Logic

Web analytics are a standard for online businesses, and product managers rely heavily on web analytics data when making product decisions. Stats like website traffic, conversions, in-app events, or unique users are frequently used as indicators of business health, because they help identify if the product is moving in the right direction or not.

With notifications, however, analytics are a different story. While the notification experience tends to be one of the key parts of product functionality, notification analytics are rarely available — and even when they are, it’s usually not so straightforward to get relevant insights out of the data. But without notification analytics, most companies are “flying blind” on their notification experience and don’t know how to improve it over time.

In this article, we talk about omni-channel analytics, why analytics are hard for notifications, and what we believe is needed to empower PMs. We’ll also share some specific tips on making the best of analytics data to design a more informed notification experience for your users.

What is omni-channel analytics?

Single data points on their own are not very informative, but in aggregate, they can tell a powerful story. Analytics is the art of assembling raw data from your product in a meaningful way. For instance, you could aggregate event data collected in an app, and infer from it which of the app’s features are most popular among your users. You can then use those insights to propel your product further by expanding the features that your users like, and retiring those they don’t. In short, analytics helps you forecast and shape the future by understanding the past and present.

These days, most businesses communicate with their customers through multiple channels: email, push notifications, SMS, and messages on social media complement more traditional means of communication like phone calls and letters. With that many options for reaching out to your customers, it’s insufficient to look at only one channel if you want to gain insights about how a specific communication strategy is working.

If you use a multi-channel strategy to communicate with your customers, it’s typically not helpful to look at those channels in isolation. Your customers might ignore your newsletters in their email inbox, but perhaps that’s because they have already seen and liked your announcements on a social media platform? So the most sensible way of doing analytics is to group messages by content, rather than by channel. So in omnichannel analytics, you look at your users’ interactions with your product holistically, across channels.

Analytics for notifications

Notification data means specifically how your users interact with your notifications. Do they click on an email as soon as they receive it? Do they swipe your push notifications away from their home screens? By doing analytics on notification data, you can understand which communication strategies work well, and which don’t.

The notification experience that your users have with your product can have a huge influence on whether or not they will continue to use it. Sending too many notifications through unsuitable channels or at the wrong time can cause such a bad experience for users that they start looking for better alternatives. A good experience, on the other hand, will go a long way towards a happy and sustained user base.

A central metric in notification analytics is the open-rate: it quantifies how many users opened a notification within a certain time span of sending it. A low open-rate means that you’ll have to think about changing your strategy — perhaps by sending fewer notifications, using different channels, or updating your content. As outlined earlier, an analytics strategy that takes multiple channels into account can deliver more insights. But before we talk about that, let’s quickly have a look at the specifics of notification data.

The difficulties of obtaining notification data

Data is the basis for all analytics — but it can be hard to get our hands on the kind of data we need to perform analytics on notifications. Apple’s iOS, for example, doesn't allow developers to precisely track impressions of push notifications on their users’ devices. With SMS messages, there is no analytics at all: you can know what messages you sent, but you don’t have any guarantee that users received them.

What we really need for PM empowerment: omnichannel analytics

To get a holistic picture of their customers’ notification experience, PMs need to know how different notifications fare across channels.

This is especially true given that the future will bring more notification preferences and flexibility for each user. As everyone’s notification experience becomes more personal, PMs will need to move to analytics that are grouped by notification content, rather than by channel. For example, all notifications about new mentions in a document app should be presented together in an analytics dashboard, regardless of whether the notifications were sent via email, Slack, or mobile push depending on each user’s preferences. But in order to enable omnichannel analytics for notifications, PMs have to overcome a few hurdles.

For channels where notification data is hard to obtain, developers can use reliable proxy data points, which help them approximate the data they’re really looking for. Read on to learn more.

How to get omnichannel analytics in your application

To be able to get omnichannel analytics, you will need to have the right data in place and be able to pull reports in the right format. Here are the steps that we recommend following to get there.

Implement analytics for each channel — as much as possible

Of course, the first step is to collect whatever data you can reasonably collect around notification usage, while respecting relevant privacy regulations and creating as little inconvenience to the user as possible. For example, when sending emails via a third-party email platform like SendGrid or AWS SES, the following are a few of the data points that you should consider collecting:

• For each email notification sent: recipient, date, subject, attachments
• Open and click metrics
• Statistics on unsubscribes
• Statistics on complaints

Note that not all of this data will be available from every platform, and different platforms might use different terms to describe the same metrics.

For some channels, it will not be possible to collect direct data for notification interactions. In such cases, you will need to come up with proxy data points: that is, data that may not be directly what you need but that can be used to infer the metrics that you care about. For example, in the case of SMS messages that have no analytics at all, you can assume a specific open rate based on industry benchmarks, or insert a link which you can track clicks on. You can then calculate an approximate open rate throughout your customer base.

Some channels have well-established workarounds for tracking. In emails, for example, it’s common to use web beacons, also known as tracking pixels. These are usually implemented as tiny images hosted on a third-party web server, and when a user opens the email, the email client loads that resource from the third-party server, which indicates to the sender that the recipient has opened the email. Such solutions don’t always work, however: users can disable images in emails, and iOS’s built-in mail application has a setting that prevents such tracking from working. There are other tracking techniques, like adding user-specific information to all links in the email, so if a link is clicked, it is straightforward to know which user the interaction came from.

Ensure data quality

To generate trustworthy reports about notifications, you will need to be confident in the data the reports rely on. Even though some data might be based on proxy metrics, as we discussed above, you need to be convinced that those are the best proxy metrics to use in your specific situation.

In addition, spend some time to understand possible limitations of source data — maybe there is a delay in the stats that some systems give you, or maybe there are sometimes empty rows in the data. You need to account for that ahead of time so you don’t run into issues later.

Identify how you’ll cross-reference stats between channels

The implementations of notification channels like email, chat, or SMS don’t know anything about the content of the notification you’re sending.

To be able to cross-reference the same kinds of notifications across channels you’ll need to create an identifier — for example, a unique ID number — for each notification. Such IDs need to be the same across multiple channels so that you can group them together later when generating reports.

Transform all data to a common format

In order to look at multiple channels together, you must have all the data available in a single report, dashboard, or spreadsheet, but that’s not that straightforward because different providers present data differently in their statistics. To be able to look at all data points across channels together, you will need to transform all data into a standardized format, for example, through an ETL pipeline.

Once the data is transformed, you can use any reporting or business analytics system that you already use — for example, Looker, Tableau, or Excel — to look at all the data points together.

Group by notification identifier — or eventually topics or notification types

Now that you have all the data in a single place and in the same format, and you also have a way to cross-reference notifications across channels, you can group data points by notification ID (or a similar identifier) across channels. You can then group this data across topics or types of notifications. For example, you might want to see all lifecycle notifications on one dashboard, and all transactional notifications on another dashboard.

Next step: use analytics to make better notifications

Having access to analytics for notifications is an achievement in itself, but remember that you’re doing it with an end in mind: to make your notifications better.

Now that you have data on notifications across channels, here are a few ways you can use the omnichannel data to identify opportunities to fix or improve your notification experience.

Identify opportunities for improvement

Consider which existing notifications have a lot of engagement. What can you learn from this? Is there any way you can replicate the success of such notifications in other parts of your notification experience?

In contrast, which notifications currently have little engagement from users? Maybe that’s a sign that these notifications need to be reworked or removed, or that you’re sending too many notifications.

Another question that can help identify opportunities is which notifications do your best customers receive the most? The “best” part here is up for interpretation according to your business goals — maybe it’s top customers by revenue, or by their level of engagement, or by another metric. Is there a connection between receiving these notifications and the value they get from using your application? If yes, how does that translate to the rest of your user base?

What trends can you monitor?

In addition to looking for specific answers in your notification data, sometimes it may be helpful to monitor trends so that you proactively see if things start to change. A few trends that you can look out for are:

• The relationship between notification interactions and user growth. If you add more users but they interact less with notifications, it might suggest that the users are less engaged, or that the notification experience is no longer a good fit for your newer customer base.
• Cohort analysis. Are newer users responding to notifications better or worse than users that have been around for a while? By looking at notification stats in the context of user cohorts, you can quantify the effect that notification changes have on new user onboarding.

Don’t have time to build notification infrastructure yourself? Try Courier

Omnichannel analytics are an enabler for doing great things with notifications, and we believe getting such analytics to work in your product is key to long-term improvements in the notification experience.

It can be a lot of work to build all the required infrastructure for omnichannel analytics, though. And if you don’t have the required resources to commit to implementing it all yourself, consider using Courier.

One of the key features Courier offers is an Analytics dashboard which helps you see traction across notification channels out of the box. Need something more custom? With the Logging feature, Courier can export the details of all notifications sent, across channels, into a system of your choice so that you can run more detailed reports.