Implementing LUIS Action Binding on the Client

In the earlier stages of LUIS, binding Intents to Actions was something that was automatically handled by the LUIS service behind the scenes. Because of this, developers could only influence action binding via the LUIS app portal. Action Binding has since been deprecated and removed from the LUIS.ai service, and abstracted to the client side, which allows the client (developer) to have direct control over action binding, and ultimately visible control over conversational flow.

With control over action binding being given to the client, we can now also specify requirements for an action to be triggered when bound to an intent. These requirements are defined as action members, and will match recognizable entities for the intent that the action maps to.

Client side action binding supports only one action per intent. Each action includes members (properties) mapping to entities. These action members can be optional or required, the client framework will provide you the tools to validate the action’s state in order to see if it can be fulfilled or not.

Defining an Action in your Application

The new Action Binding library provides a different implementation to bind Intents to Actions, however certain concepts such as defining action parameters by using properties (that would map LUIS recognized entities) remain the same. Because LUIS will no longer determine an action based on a user’s query, actions need to be constructed to accommodate intents and entities. Although the same binding concept is apparent in both C# and NodeJS, the difference in languages as you’d expect, necessitates different implementations.

Moving onto the code, we begin with the C# implementation before moving on to the NodeJS.

C# Implementation

Source Code: C# Action Binding Samples

If you are currently working with the Bot Builder Framework for C#, many concepts will be familiar while others will be new. The new Action Binding library defines an ILuisAction interface, which provides the contract shown below:

public interface ILuisAction
{
    Task<object> FulfillAsync();

    bool IsValid(out ICollection<ValidationResult> results);
}

This interface includes two methods, which you define from the client:

  • FulfillAsync: Provide custom logic needed to fulfill the user’s intent. The result type can be whatever you like.
  • IsValid: Here you validate that the action is ready (ie. all mandatory members are resolved) and the fulfillment method can be called.

IMPORTANT: A new mandatory attribute is introduced in order to decorate your actions (classes) per the following:

LuisActionBindingAttribute.cs

[AttributeUsage(AttributeTargets.Class, AllowMultiple = true)]
public class LuisActionBindingAttribute : Attribute
{
    public LuisActionBindingAttribute(string intentName)
    {
        if (string.IsNullOrWhiteSpace(intentName))
        {
            throw new ArgumentException(nameof(intentName));
        }

        this.IntentName = intentName;

        // setting defaults
        this.CanExecuteWithNoContext = true;
        this.ConfirmOnSwitchingContext = true;
        this.FriendlyName = this.IntentName;
    }

    public bool CanExecuteWithNoContext { get; set; }

    public bool ConfirmOnSwitchingContext { get; set; }

    public string FriendlyName { get; set; }

    public string IntentName { get; private set; }
}

This new attribute is mandatory because it defines which intent the action is for (NoteintentName must match the intent name within your LUIS model), as the framework will search for types within your application which use this attribute (LuisActionBindingAttribute) and implement the ILuisAction interface when a call to resolve an action for an intent is made.

But what about the action members mentioned previously? In the beginning of this post, we defined action members as requirements for a specific action to be triggered, when bound to an intent. The action members will be situated at the class implementation which uses the ILuisAction interface.

Adding Validation to your Actions (C#)

The framework provides an abstract class BaseLuisAction which by default includes validations from the .NET Data Annotations as shown below:

To implement common validation logic for an action, the ILuisAction interface provides a default FulfillAsync method you may implement at your class.

BaseLuisAction.cs

[Serializable]
public abstract class BaseLuisAction : ILuisAction
{
    public abstract Task<object> FulfillAsync();

    public virtual bool IsValid(out ICollection<ValidationResult> validationResults)
    {
        var context = new ValidationContext(this, null, null);

        validationResults = new List<ValidationResult>();
        var result = Validator.TryValidateObject(this, context, validationResults, true);

        // Order results
        validationResults = validationResults
            .OrderBy(r =>
            {
                var paramAttrib = LuisActionResolver.GetParameterDefinition(this, r.MemberNames.First());
                return paramAttrib != null ? paramAttrib.Order : int.MaxValue;
            })
            .ThenBy(r => r.MemberNames.First())
            .ToList();

        return result;
    }
}

This allows for two flexible options for action validation:

1) From your action, inherit this class (BaseLuisAction), and you are only required to implement the default FullfillAsync method at your action class. This relies on the .NET Data Annotations for built-in common validations, implemented at the IsValid method.

2) You may implement the ILuisAction interface and include your own custom validation logic. Note that the IsValid method is virtual, and can be extended to your actions as needed.

Implementing and Binding your Actions (C#)

In the following example, an intent of GetWeatherInPlace is bound to an action of GetWeatherInPlaceAction using a LuisActionBinding attribute at the class level. In this Action, the BaseLuisAction is also inherited.

NOTE: By convention, the name of an attribute class ends with the word Attribute, however when applied to a class, the inclusion of the word Attribute is optional. See MSDN Custom Attributes.

[LuisActionBinding("GetWeatherInPlace", FriendlyName = "Get the Weather in a location")]
public class GetWeatherInPlaceAction : BaseLuisAction
{
    [Required(ErrorMessage = "Please provide a location")]
    public string Place { get; set; }

    public override Task<object> FulfillAsync()
    {
    	// TODO: perform necessary calls to get weather in Place
    }
}

From the example above the Place property in the Action includes a Required attribute. When this new action object is created, the IsValid method inherited from BaseLuisAction will check if the Place contains a non-empty value. Assuming no validation errors, IsValid will return true, and then the action GetWeatherInPlaceAction will be bound to the intent of GetWeatherInPlace.

Determining an Action based on the LUIS response (C#)

As mentioned, in the past Actions were bound to intents at the LUIS service. Under the old LUIS service, after a user submitted a request to LUIS, the service would determine an action, and return a reference to that action in the response. As Actions are no longer determined by LUIS, the client (developer) can determine the action to bind based on the returned Intent and recognized entities. In short, LUIS will now only determine intents and entities, and the client will determine the actions the intents and entities determined by LUIS.

To determine the action for a given intent, the framework provides a LuisActionResolver class which receives a list of assemblies when instanced. This resolver will go through the list of assemblies provided and search for all the actions defined in the application.

LuisActionResolver.cs

var luisResult = await luisService.QueryAsync(query, CancellationToken.None);

var resolver = new LuisActionResolver(this.GetType().Assembly);

var intentAction = resolver.ResolveActionFromLuisIntent(luisResult);

if (intentAction != null) {
  while (!intentAction.IsValid()) {
    // TODO: While the action is not ready yet (ie. valid) to call the
    // fulfillment execute appropiate logic in order to try make it valid
  }

  var result = await intentAction.FulfillAsync();
}

A summary of the steps in the example above:

1) An async request is made to the LUIS service and the return object is referenced in luisResult.

2) Create a new instance of the LuisActionResolver class, providing it with a list of assemblies from the application. This class includes a method ResolveActionFromLuisIntent which will be used to determine which action the intend will bind.

3) Invoke the ResolveActionFromLuisIntent method, passing in the LUIS response (luisResult), which returns the proper action to bind, referenced in intentAction.

So.. you might be wondering, why do I need to implement ILuisAction if I need to provide the luisResult (where the user’s intent is provided along with the recognized entities)? Up to now, there is no good reason to do all this as you need to call the LUIS service by yourself, create an action and map it to an intent via the binding attribute and finally implement the code for the fulfillment. You could perfectly have done all this with no need to implement ILuisAction and use the binding attribute, nor even use the resolver. You could simply inspect luisResultinstance and execute the logic to resolve it.

Why all this ‘overhead’ then? Be patient, you will see how all this helps when you implement a Bot app that integrates all these concepts and resolves all the code shown above transparently (in particular the code to resolve missing entities until the action is valid and can be fulfilled).

Now we are ready to examine how we can incorporate Action Binding to our Bots.

Using LUIS Action Bindings in a Bot (C#)

Reference the Github Repo

Now that we are able to bind our Actions to Intents given the response from the LUIS service, the last step is to integrate the binded actions to the application Dialog.

In order to integrate all the features described before seamlessly the framework provides a LuisActionDialog that you can inherit at your bot app. This dialog integrates LuisIntentAttribute attributes decorating your intent handler methods, with the ILuisAction implementations you might have (mapping those intents), getting the appropriate action fulfillment result for a particular user’s request and sending it back to the intent handler after being resolved.

For this, the LuisActionDialog defines new valid signatures for the intent handlers that you can use at your custom dialog:

public delegate Task LuisActionHandler(IDialogContext context, object actionResult);

public delegate Task LuisActionActivityHandler(IDialogContext context, IAwaitable<IMessageActivity> message, object actionResult);

As said, the dialog receives the action binded to an intent by using the LuisActionBindingAttribute.IntentName property within the ILuisAction classes, and validates if the action is valid to be executed. If the action is not valid yet, the user will be prompted for non-valid member values until the IsValid method from your action returns true. Finally, it calls the intent handler with the object you return at the FulfillAsync method of your action.

At the client level for our Bot, we begin by importing the new module we’ve been discussing.

using Microsoft.Cognitive.LUIS.ActionBinding;

Reference: C# Samples Bot

So, how can a custom dialog be implemented which handles a user’s requests? Quite simply – your dialogs should only inherit from LuisActionDialog as shown below, and decorate your handler methods using the LuisIntentAttribute.

LuisActionDialog.cs

[Serializable]
public class RootDialog : LuisActionDialog<object>
{
    public RootDialog() : base(new Assembly[] { typeof(RootDialog).Assembly }, luisService)
    {
    }

    //By convention, the word 'Attribute' is optional, LuisIntent/LuisIntentAttribute are equivalent

    [LuisIntent("GetWeatherInPlace")]
    public async Task GetWeatherInPlaceAsync(IDialogContext context, IAwaitable<IMessageActivity> message, object actionResult)
    {
        var reply = context.MakeMessage();

        // TODO: Use actionResult to provide a meaningful response to the user within the reply message
        // For example if the result is just a string you could simply do reply.Text = actionResult.ToString()

        await context.PostAsync(reply);
    }
}

As you might have seen the constructor for the LuisActionDialog receives a list of assemblies where it will find ILuisAction implementations decorated with the LuisActionBindingAttribute, as well as several ILuisService instances that will be used to resolve user’s messages.

The only requirement in order to use your actions with the LuisActionDialog is that they should be serializable (include the [Serializable] Attribute as shown above).

Node.js Implementation

Source Code: Node.js Action Binding Samples

From our app level we begin with requiring the new module which allows for Action Binding:

var LuisActions = require('botbuilder-luis-actionbinding');

Action Binding in Node.js

First we examine the main file within the module (index.js). Here let’s look at several of the core elements the framework uses to implement LUIS Action Binding and bind LUIS Intents within your own bot. Each LUIS Action is implemented as a JavaScript object with four primary fields, and other optional fields:

  • intentName (string)

    Determines which LUIS intent name the action should respond to.

  • friendlyName (string)

    The friendly name is used to be displayed in some cases, as for example when you are switching from one action to another a prompt might ask you to confirm the switch and the friendly names of the actions are being shown in the confirmation message

  • schema (object)

    Defines the list of object properties the Action requires, their types and message to display to request the parameter. It is defined using schema-inspector. More information here.

    Each parameter is defined using the following fields, plus any schema-inspector validation property:

    • type (string)

      Defines the expected type. See schema-inspector types for valid options.

    • message (string)

      Defines the message displayed to the user when the parameter is missing or validation does not pass.

    • builtInType (string) – (Optional)

      This is optional and is used to help identify the entity that should match to this parameter. See LuisActions.BuiltInTypes for valid options.

  • fulfill (function)

    This function fulfills the user action. It receives the parameter values as a keyed-object and a callback function that should be invoked with the fulfillment result. Here you do whatever is required to fulfill the intent and provide a meaningful result.

Let’s examine an example of how we may incorporate these action members into an Action.

var WeatherInPlaceAction = {
    intentName: 'WeatherInPlace',
    friendlyName: 'What\'s the weather?',
    // Property validation based on schema-inspector -
    // https://github.com/Atinux/schema-inspector#v_properties
    schema: {
        Place: {
            type: 'string',
            builtInType: LuisActions.BuiltInTypes.Geography.City,
            message: 'Please provide a location'
        }
    },
    // Action fulfillment method, recieves parameters as keyed-object (parameters argument)
    // and a callback function to invoke with the fulfillment result.
    fulfill: function (parameters, callback) {
        rp({
            url: util.format('http://api.apixu.com/v1/current.json?key=%s&q=%s', ApixuApiKey, encodeURIComponent(parameters.Place)),
            json: true
        }).then(function (data) {
            if (data.error) {
                callback(data.error.message + ': "' + parameters.Place + '"');
            } else {
                callback(util.format('The current weather in %s, %s is %s (humidity %s %%)',
                    data.location.name, data.location.country, data.current.condition.text, data.current.humidity));
            }
        }).catch(console.error);
    }
};

In the above Action binding example, only one parameteter Place, is defined. You may include any number of parameters to your application as you need, by default they are required, however can be made optional by providing the parameter a key-value of optional : true.

See the Github Repository for more Node.js samples to implement LUIS Action Binding.

Using LUIS Action Binding – Action Models and Usage (Node.js)

The framework provides an easy hook for integrating Action Binding within a Bot built using BotBuilder (as you’ll see in the next section), but a low-level construct is offered that can be used programatically with any application type. The Console and Web application samples makes use of this building block.

This option is provided by the LuisAction.evaluate method and has the following signature:

function evaluate(
    modelUrl: string,
    actions: Array<IAction>,
    currentActionModel?: IActionModel,
    userInput?: string,
    onContextCreationHandler?: onContextCreationHandler)
    : PromiseLike<IActionModel>;

NOTE: You can check the TypeScript bindings as reference.

The input parameters are:

  • modelUrl is the LUIS.ai application url.
  • actions is an array of Action Binding definitions.
  • currentActionModel is the actionModel returned from a previous call. The first time you invoke this method, it should be null.
  • userInput is the current input string – typically submitted by the user.
  • onContextCreationHandler is an optional callback method to switch contexts when triggering contextual actions. Context handling is used to switch between different intents during conversation, based on the user’s input (ie. the user changed their mind and requests something else). Context switching is an in-depth topic worthy of it’s own dicussion and beyond the scope of this article, it is ok to disregard this for now.

The returned Promise resolves to an IActionModel (just actionModel from now on) that can be used to re-call the evaluate function and, in that way, keep the context of the conversation and the action binding state. This object contains the status if the action binding evaluation and, if fulfilled, the result object.

Using LUIS Action Bindings within a Bot (Node.js)

The key thing here is creating an IntentDialog with a LuisRecognizer, and bind the Actions array to the bot and dialog using the bindToBotDialog helper function. A summarized example is shown below:

var LuisModelUrl = process.env.LUIS_MODEL_URL;

var recognizer = new builder.LuisRecognizer(LuisModelUrl);
var intentDialog = bot.dialog('/', new builder.IntentDialog({ recognizers: [recognizer] })
    .onDefault(function (session) {
        session.endDialog(
            'Sorry, I did not understand "%s". Use sentences like "What is the time in Miami?", "Search for 5 stars hotels in Barcelona", "Tell me the weather in Buenos Aires", "Location of SFO airport")',
            session.message.text);
    }));

// Import the Core Module for Action Binding
var LuisActions = require('botbuilder-luis-actionbinding');

// Import an array with Binding Actions (e.g: actionbinding-samples.js)
var SampleActions = require('./actionbinding-samples');

// Finally, bind the actions to the bot and intentDialog, using the same URL from the LuisRecognizer
LuisActions.bindToBotDialog(bot, intentDialog, LuisModelUrl, SampleActions);

The bindToBotDialog function registers each Action Binding with the IntentDialog, using the action’s intentName. Internally, it uses the dialog’s matches() function.

The function also registers with the bot a new BotBuilder library that contains a single dialog, named Evaluate. Then, it uses the internal createBotAction function to route incomming messages with matching intents to the Evaluate dialog.

The Evaluate dialog extracts the entities from the LUIS Result, maps them to parameters, validates the parameter values and, if possible, fulfills the action function, among other things.

When validation does not pass, prompt messages are presented to the user to provide each of the missing or invalid parameters and continues until validation passes. Then the fulfill function is invoked, passing all validated parameters.

And that’s it! Everything in place to support Action Binding for your Bot in Node.js.

Summary

We’ve showcased how to implement Action Binding on the client end for building applications using Microsoft’s LUIS service in both C# and NodeJS, both supported by the Microsoft Bot Framework. While LUIS Actions on the service side have been removed, this was to provide more control and flexibility to developer, who can now more strictly control how conversational flow works in their Bots!

In the Future, we will provide an overview of how to implement this concept for Web Applications (ASP.NET MVC), as well as Console apps.

Once again, you are welcome to the source code at the Github Repo.

Cheers!

Pablo Constantini, Ezequiel Jadib & Matthew Shim from the Bot Framework team3