Bot Analytics

15.12.2016 |

Episode #7 of the course Building your own Facebook Messenger bot by Carylyne Chan


Yesterday, we went over the best practices in testing and launching your bot. Today, we’ll pick the right metrics to track and measure your bot and find out how to use this data.

The metrics that you choose should help you to make actionable decisions, such as whether you should continue your current user acquisition efforts, if you can validate your initial hypotheses, and if there are new possibilities that you can now explore.


Defining the right KPIs

Going back to your objective (defined at the start of your bot-building process), work out which KPIs matter to you and figure out which metrics most support these key performance indicators. For example, if you set up a bot to support customers, then you should measure the number of successfully supported customers who didn’t have to file a support ticket in order to get their questions answered (also known as “deflection rate” in some circles).

Set two or three meaningful KPIs and stick to them at launch. Now, you need to choose which metrics are most useful to you.


Engagement metrics

Examples: number of users, daily active users, total conversations daily, number of conversations per user daily, most active users, actions per user, most used buttons, etc.

Tracking engagement metrics can be done using the native Facebook Analytics that comes with your page. With this tool, you can easily track actions and events, including all the metrics listed above. These metrics give you an overview of how users are using your bot and are indicators of how much people like it.



Dialog metrics

Examples: average and median bot confidence scores, top queries and categories, most unanswered question, conversation abandons, escalation queries, etc.

Dialog metrics are more specialized metrics that require some knowledge of text analysis. When using one of the NLP services, they usually show you the confidence scores of their replies, which is the first baseline for figuring out how likely it is that your bot will answer questions correctly. Following that, the other metrics may require you to use more focused text analytics tools/vendors or run your own experiments and visualizations to really get meaningful results. For bigger-scale projects and brands, this tool or vendor is important, since the time needed to manually look through every conversation can be astronomical.


Use case metrics

Examples: number of desired actions taken, number of actual completed outcomes, ratio of attempted actions vs. completed actions.

Depending on your use case, you can define custom metrics that help you to track outcomes significant to your objectives. For example, setting goals for the number of items you sell on the bot or number of support tickets raised can be important for tracking the feasibility of the bot in the long run.


Using the results

Like analytics for any other software, simply collecting information is not enough; you need to accurately interpret and use these results for a specific purpose. Ideally, you should let your bot run for at least a few weeks to get a sizeable enough data set for analysis. Another source of results is qualitative—solicit feedback from users and ask for their opinion on your bot.

There are two phases to this: optimizing and taking (drastic) action. First, you may tune your bot’s functionality, actions, or dialog based on the insights that you have gathered from the metrics. To illustrate, you may find that users often say that the content recommended to them was not what they asked for (such as when they are asked “Is this what you’re looking for?”). You have to figure out if this was due to your ML model parameters or if the bot didn’t explain what it was supposed to do and people were asking something out of scope.

Next, if you realize that despite your best efforts, the bot doesn’t respond accurately after three to five cycles of this process, you may have to consider changing the use case or reexamining whether you want to keep the bot running. (Yes, sad, but it happens!)

Tomorrow, let’s talk about the common pitfalls that beset bots and what we can do to prevent them from happening as much as possible.


Share with friends