Optimizing latency with the Bot Framework

Responsiveness in a conversational bot is critical for a good user experience, as users want responses to be fast. With maximizing user satisfaction as a goal in mind, there are some basic operations which every bot must still do, such as getting an authorization token, and getting/setting conversation state. In this post we’d like to go over some ways to optimize latency with bots registered with the Bot Framework.

  1. Enable periodic token refreshing in your bot
  2. Creating a custom state service

Periodic token refreshing

Within the Bot Framework, before a bot sends a message back to a user via the connector service, it must have a valid token. Currently, a valid token lasts an hour before they expire and the bot needs to retrieve a new one. Occasionally, this can cause latency issues – to a user, this may appear as if the bot is taking longer to reply than usual. At worst, time outs with the Bot Connector Service can occur. One solution to fix this is to ensure that a bot always has a valid token.

Below are two samples, for C# and Node.js respectively, which can be added to your bot’s code to periodically get a new token –

Please note that this optimization is not required, but is recommended to include in order to keep your bot responsive. In addition, this optimization will not help if the bot is stopped by the hosting provider.

C#

using System;
using System.Threading.Tasks;
using Microsoft.Bot.Builder.Dialogs;
using Microsoft.Bot.Connector;
using System.Threading;
using System.Web.Http;
using System.Configuration;
using System.Diagnostics;

public class WebApiApplication : System.Web.HttpApplication
{
    CancellationTokenSource _getTokenAsyncCancellation = new CancellationTokenSource();
    
    protected void Application_Start()
    {
        GlobalConfiguration.Configure(WebApiConfig.Register);

        var appID = ConfigurationManager.AppSettings["MicrosoftAppId"];
        var appPassword = ConfigurationManager.AppSettings["MicrosoftAppPassword"];

        if(!string.IsNullOrEmpty(appID) && !string.IsNullOrEmpty(appPassword))
        {            
            var credentials = new MicrosoftAppCredentials(appID, appPassword);

            Task.Factory.StartNew(async () => 
            {
                while (!_getTokenAsyncCancellation.IsCancellationRequested)
                {
                    try{
                        var token = await credentials.GetTokenAsync().ConfigureAwait(false);
                    }
                    catch(MicrosoftAppCredentials.OAuthException ex)
                    {
                        Trace.TraceError(ex.message);
                    }
                    await Task.Delay(TimeSpan.FromMinutes(30), _getTokenAsyncCancellation.Token).ConfigureAwait(false);
                }
            }).ConfigureAwait(false);
        }        
    }

    protected void Application_End()
    {
        _getTokenAsyncCancellation.Cancel();
    }        
}

Above, we create a new instance of MicrosoftAppCredentials.cs which is part of the BotBuilder SDK, create a new Task,  and within that task create a loop which periodically calls the GetTokenAsync method, which will also call RefreshAndStoreToken, meaning that your token will automatically be cached at this point and you don’t need to do anything else – the bot will have a valid token. You can specify the interval in which to do this, above we have it set to every 30 minutes.

Node.js

let chatConnector = ...;

... 
setInterval(() => {
    chatConnector.getAccessToken((error) => {
            console.log(JSON.stringify(error));
        }, (token) => {
            console.log(`token refreshed: ${token}`); 
        });
}, 30 * 60 * 1000 /* 30 minutes in milliseconds*/ );

Similarly for bots using the Node.js SDK, we periodically call the getAccessToken method from chatConnector.js, and then the new valid token is automatically stored for your bot.

Credential provider to support multiple bots

If your application is hosting multiple bots, you can periodically refresh the tokens for all of the bots from one single thread, instead of creating a new thread for each bot. This can be done by using using the ICredentialProvider as follows:

public class MultiCredentialProvider : ICredentialProvider
{
    public Dictionary<string, string> Credentials = new Dictionary<string, string>
    {
        { "YOUR_MSAPP_ID_1", "YOUR_MSAPP_PASSWORD_1" },
        { "YOUR_MSAPP_ID_2", "YOUR_MSAPP_PASSWORD_2" }
    };

    public Task<bool> IsValidAppIdAsync(string appId)
    {
        return Task.FromResult(this.Credentials.ContainsKey(appId));
    }

    public Task<string> GetAppPasswordAsync(string appId)
    {
        return Task.FromResult(this.Credentials.ContainsKey(appId) ? this.Credentials[appId] : null);
    }

    public Task<bool> IsAuthenticationDisabledAsync()
    {
        return Task.FromResult(!this.Credentials.Any());
    }
}
Since ICredentialProvider itself does not include methods to retrieve App ID and Passwords, we created a new class MultiCredentialProvider, which includes a dictionary storing the App ID and Password for each of the bots the application hosts, and created some getter functions for the properties which we need to refresh the tokens.
Back in the Task where we refresh our tokens, our updated code will look similar to before, only we added a loop to iterate through each bot credential listed in the MultiCredentialProvider from above.
Task.Factory.StartNew(async () =>
{
    while (!_getTokenAsyncCancellation.IsCancellationRequested)
    {
        foreach (var botCreds in new MultiCredentialProvider().Credentials)
        {
            var credentials = new MicrosoftAppCredentials(botCreds.Key, botCreds.Value);
            try
            {
                var token = await credentials.GetTokenAsync(true).ConfigureAwait(false);
            }
            catch (Exception err)
            {
                //log error
                Trace.TraceError(ex.Message); 
            }
        }

        await Task.Delay(TimeSpan.FromSeconds(30), _getTokenAsyncCancellation.Token).ConfigureAwait(false);
    }
}).ConfigureAwait(false);

Custom state service

Getting and setting bot state is another operation which takes time, and happens on every invocation of your bot. By default, the the BotBuilder SDK connects to the default Bot State Service hosted by the Bot Connector, but there’s several reasons why you should implement a custom state store:
  • Improve your bot’s latency as you won’t be relying on the connection of the default connector service.
  •  As mentioned before, the default connector state service is not intended for production applications. The state service was for maintaining conversation state (user data, conversation data, etc) while keeping the bot itself stateless.
  • You gain greater control over your conversation state, and customer data
Because of these benefits, we greatly encourage bot developers to implement their own custom state stores. Luckily, we’ve already released how-to’s for how to achieve this.

From the Bot Framework documentation:

Manage state data – C#

Manage state data – Node.js 

From the blog:

Saving State data with BotBuilder-Azure – C#

Saving State data with BotBuilder-Azure – Node.js 

Happy Making!

The Bot Framework team