Become a Bot Builder with Microsoft Bot Framework

James Mann speaking at dotnetsheff in April, 2017
Great talks, fired to your inbox 👌
No junk, no spam, just great talks. Unsubscribe any time.

About this talk

The way in which customers expect to be able to interact with companies is changing. We'll look at this change and how you can take advantage of it by creating chatbots using the Microsoft Bot Framework. We'll cover the basics, including what tools you can use and where to go in order to get started. Then we'll see how we can use Microsoft Cognitive Services to sprinkle AI magic on your chatbots and turn them from simple command-line pretenders to useful tools that customers will love.


Thank you very much for turning out. It's really good to see people interested in this topic. Can I just do a quick show of hands? Has there anyone actually looked at Bot Framework before? Besides to use it? So you're all completely new to this, which is brilliant for me because what I'm looking to do here is to kind of give you an overview of everything that you need to know to get up and running and creating your own bots. So at the end of this, the idea is, you guys can go off and create your own bots with your own application ideas. Just a quick word about myself. I'm James, I'm a Microsoft MVP. Recently, been awarded an MVP for data platform, but really for my work around Bot Framework, which is something that we're heavily involved with. If anyone wants to follow me on Twitter, I tweet a lot about Bot Framework. You can get some good news, good updates, from there if you're interested. I just wanna quickly talk you through the agenda. It's broken into two bits, with pizza in the middle. The first bit is really just introducing you to some kind of core concepts that you need to be aware of. Just, we'll look at how we can build conversations of using different techniques. We'll look at something called QnA Maker, which is a real sort of low resistance entry point for you guys to create bots. Then we'll break, and then, after pizza, we'll look at how we can add intelligence to your applications. So we'll look at things like, we'll look at building more richness into your bots with Cards and attachments, and then we'll look atProactive Bot, which I'll talk about when we get to that. I've posted the code in the slide channel. So you guys might have already seen this. But I'll put this up at the end, as well, so you can, if you wanna have a look at the code when we're done, it's all pretty straightforward. There's no kind of prerequisites other than copy of Visual Studio. So just wanna jump right into it now and talk about, why are bots interesting? 2016, people are saying it was the year of the bot, and bots were the, kind of the peak of the graph thing that they have. Why was that? Well, one of the main reasons is these devices that we have at home. We have Siri, we have Cortana, we have Google Home, and they're everywhere. They can be in your bedroom, they can be in the kitchen. They're probably with you now. You guys have all got phones with you, and I'm sure you use them to some extent, maybe, some more than others. That's one reason. But the other reason is messenger applications are getting really popular. This diagram here kinda shows you how the messaging apps have actually surpassed social networks, you know, Facebook, that kind of thing. People are way more engaged in messaging than they have been. This is continuing to grow. One thing that I like to talk about, actually, when I'm introducing people to this concept is, if you guys have got your phone with you, just pull it out and have a look at your home screen and just count how many messaging apps you've got. We find that most people have got between five, five and seven, just on the home page. That's, just in terms of distribution, that's a really compelling story. In terms of business cases for these things, there's tonnes of them. You've probably all heard "the customer is a bot," "bot wave," you've got bot which is first-line support for maybe technical supports, or someone got a problem, they can't connect to WiFi, they wanna know how to do that. Well, that's the perfect kind of scenario that a bot can deal with. But it can help with all kinds of things, as well. We'll look at some of those in more detail. But you can see with all these vertical slices down here. What I would say is, not everything is actually appropriate to be implemented as a bot. I was talking to someone today about actually one of the examples on this page is shopping, isn't really the best use case for bots, because if you think about, if you are shopping for something and you don't know precisely what you're looking for, it's good to be able to kind of discover based on context, if that makes sense. Bots are very good for really precise operations, really precise applications. But they're not good for getting a wide visibility of information. So that's just some things to bear in mind. They're not appropriate for everything. But they do solve some problems really well. I think the key thing for you guys to be aware of is, they are really focused applications. They're not general purpose, they're not trying to solve a big wider-ranging problem. They're very focused on a specific problem they've got. And it makes sense for them to be in there, in a conversation experience. Now we've kind of set the scene in terms of why bots and some use cases for bots. I just wanna talk about Bot Framework, because this is kind of what we're gonna be talking about for the rest of the talk. So, might be useful to introduce it. Traditionally, if you wanted to target, let's say you wanted to create a bot for Slack, for example, you guys are all using Slack. You can create bots for Slack really easily. But the problem is, in order for you to create a bot for Slack, if you're connecting to it natively, you need to write your application to consume that particular API, which means distribution across other channels is very difficult. Bot Framework kinda aims to sit in the middle. It acts like a broker between all these different channels. So you can, you define your business logic as a single unit and then you can basically configure it to be exposed over whatever channels you see. One of the benefits of it is, at the moment, maybe we've got 10 channels that we can expose our bot across. But that's not to say that that number of channels is fixed, 'cause of course these things change, social messaging applications come on board. We've seen Teams recently is a new one which is trying to compete with Slack. When those things come into play, Bot Framework will basically consume those as a channel. What that means to you guys is, you can basically provide one implementation and you're almost future-proof from these additional channels that come on board. So it's quite a compelling story from that point. I'll just briefly talk about the architecture that Bot Framework gives you. It gives you a set of tools and services, something it called the Bot Connector. It gives you an SDK in C#, so .NET. Can I just do a quick show of hands? Who's using .NET in their day-to-day work? Who's using Node.js? Right. For you guys, I mean, this talk is focused on C#. But the concepts are, you know, they apply to both. The only reason I don't present on Node.js is just because I'm not a Node.js developer. So it's difficult for me to articular myself in Node.js terms. But just be aware of this, too. Then we have our application, our brains of our bot, which is basically a web service. If you guys have all used ASP.NET Web API, that's all it is. You just create a Web API that the Bot Connector connects to, and that implements your business logic. It takes care of things like state management and scalability, all that kind of good stuff. Then, we talked about the channels. Once you've defined your bot application, your implementation, you can use a configuration in Bot Framework to distribute it to whatever users you have. So that's kind of the high-level architecture. Now, what I like to do is, talk you through the three steps that you guys need to do in order to get up and running with this. We'll break it up in three. The first step is, we register. We do this through the Bot Framework portal. We just type in a bit of metadata, we give it a name, we upload an icon, we give it a description. We set the messaging endpoint, which is this guy here. That's literally just the address of our Web API. That's it. We then code our application using the SDK of our choosing. As I say, I'm gonna focus on C#. But there's nothing limiting you from doing the same things in Node.js, as well. Then, finally, we distribute across these channels. So if we've deployed our application out, we literally just turn it on. So we turn Facebook on, we turn Skype on. As I say, we get this future proofing. You know, whenever new channels come on board, we can make use of them. All right, I've got a demo to show you all this. You guys are gonna wanna, if you do actually, if you're impressed by this and you wanna pick this up, you're gonna wanna go to That's the place basically where you go and you register your bots and you download the whole tooling and everything. As I say, we're gonna be focusing on C#. So I'm just gonna click on Documentation, just kinda show you what we're looking at here. There's a decent level of documentation here to kinda get you up and running. But I'm just gonna jump to the Bot Framework Resources. In here, there's a Downloads section. There's two things. There's this thing called an emulator. That's basically the thing that lets you test it locally. So you develop your bot, you run it as you do any other website, and then you can use the emulator to test it on your local machine before you publish it out. Then we've got a Visual Studio template, as well. The Visual Studio template is literally just a template which has already got the NuGet packages and a bit boilerplate stuff set up for you so it kinda gets you up and running quickly. I've downloaded these previously, but it tells you how to instal everything if you haven't. I'm just gonna go ahead and create a new project. I'm gonna call this one DotNetSheff. That's gonna go off and is gonna create basically a Web API for me. It sets up the authentication, because obviously this is deployed to a public endpoint, so it needs to be secure to stop people basically providing their own implementations of the bot. So it sets all that up for you. Inside the web.config, we've got these two entries, AppId and AppPassword. We get those values from the portal, which I'll show you in a moment. But before we do that, just wanna talk you through the code briefly. We have this MessagesController in here. All this is is, if I just rebuild this just to get rid of those squiggly lines, all this is is a Web API endpoint. It accepts the HTTP post. The bot connector, if you think back to that diagram, the bot connector will send the message to this endpoint. Basically it's sent the activity, which is all the information about what the user's saying to the bot. Then there's these tools that we can use to kinda drive the conversation from that point on. Then we have this dialogue. We'll talk more about dialogues in a bit. But this dialogue is literally just repeating what the user's sent to the bot. So I'm just gonna start that. As I say, this is just a NVC website. So it's gonna start up so I can... There's no HTTP get there, so that will respond. But basically, that's our endpoint we're connecting to. This is the emulator, right? We talked about download the emulator. It's pretty simple to use. We just click on Connector and we just point it to our endpoint. In here, we put our app ID and our app password. Haven't set that up yet, so I just leave them blank. I can connect to that and then I can say, send a message. You can see it's responded. It's just repeated what I've said to it. Nothing massively impressive there. But it'spretty quick. Now I'm just gonna go in and I'm gonna register a bot application so I can publish it and so we can use it through a channel. I'll just log in using my Microsoft account. If we have a look at, there's two things, right? The first one is this activity type, which, if I go into, you can see there's tonnes of solve information about the conversation in there. You can retrieve what's going on in the conversation, when the message was sent, you can retrieve attachments, you know, if people are sending images, that kind of thing. So it gives you a lot of things to help you on the way. It gives you strong types to wrap up. The other thing is the security, as well. If you have a look in this MessagesController, we have this BotAuthentication attribute. That's basically the thing that secures the application. When the Bot Connector connects to it, he uses the credentials to basically do it, do a handshake and make sure everything's authenticated. So it does those two things for ya. I'm gonna come in and I'm gonna register a new bot. I've got a icon I've prepared. Then I'm just gonna call it DotNetSheffBot, right? I give it a handle. Obviously, if I was doing a real application, I would put something a bit more meaningful in here. If I just have a look in my notes here, I've actually created an Azure website I'm gonna publish this out to. You can use whatever web hosting provider you want. You don't need to use Azure. I'm just using that because it's convenient to me. The key thing is, we basically provide the URL to our restful endpoint that we just saw, /apimessages. No, you can host it wherever you want. The key thing is, it's hosted somewhere that is public. But it can be hosted anywhere. There's not a private version of Bot Framework at the moment. Whether that will change or not, possibly. I mean, people ask me quite a lot, so Microsoft may be looking into that. But at the moment, it's just public only. So I've set up some metadata. Now I'm actually defining the security handshake here. I need to take this app ID. Then I'm gonna paste that into my web.config. Then I generate a password, and I'm gonna copy this password. Again, I'm just gonna paste that in there. That's basically the handshake that secures the bot application with the Bot Connector. Then I can add AppInsights if I want. I'm not gonna do that. I'm just gonna click Register. And hopefully, if no one's taken that name, then it should create it. Brilliant. At the moment, there's nothing at that endpoint, obviously. So I need to publish my application now. I'm just gonna make a quick change. Just gonna put DotNetSheff in the front, just so you can see there's not smoke and mirrors. Just rebuild that. Then I'm gonna publish this out. Just wanna publish it to my Azure website. That's gonna go off and deploy the NVC application, Web API application, to Azure. If you've been using Web API, it's exactly the same. There's nothing different. All right, so that's published out. Hopefully, now we've got this test window here, using the Chat channel. So I can hopefully type a message and I'm gonna get a response. Delivered the message to the bot and it's responded. You can see it's got the change I made. The nice thing about this is, you can take this web channel and embed that in your web application. That's the channel that it gives you kinda free. But if you wanna expose it over other channels, it's really easy to do that, as well. It's already set up for Skype so I can add this to Skype. But I don't wanna do that. I actually want to choose one of the new channels. So I'm just gonna use Teams, just 'cause I've got that installed on here. So not much configuration you need to provide for Teams. That's why I'm demoing it. Some of the channels are quite difficult to, well, they're not difficult but they're long-winded. Facebook Messenger takes five minutes, so it doesn't make a very good demo. But it's pretty easy to do. Now our bot is hopefully registered. I can click this button now and hopefully, we'll click Open in Teams, it's gonna add that bot to Teams. It's running the same code. All I've done is configuration. All the channels are exactly the same. You won't get anything. You don't get anything. So you do need, yeah, you need to be aware of that. One thing I would say is, you do get, in the portal, you get this thing here, these issues. You can see we had an issue with the web chat when I published it the first time, because I hadn't published my application. So you can get some level of logging here so you can see what's going on. You can also wire into ApplicationInsights to get more information or whatever analytics we're using. Yes, you can, that's a really good question. Let's say you've got, I don't know, you've got a web chat used in your organisation, you use this thing called Direct Line and basically you can provide your own channel just to connect back into. Yeah. And that's really easy to use, as well. Exactly. It's a rest call with a, there's a streaming endpoint, as well. You get all the updates via the streaming endpoint and then you send updates over that channel. It's quite a nice experience. That's the demo. Hopefully that was useful. I'm going to plough on 'cause I've got quite a lot content to go over here. So we've seen a bot. We've seen how easy it is to create your first bot and get it up and running. I just wanna talk about how we actually build up the conversation now. There's quite a few tools that we can use with Bot Framework. What I've tried to do here is, I've provided a visualisation in terms of flexibility going up and then the effort that it takes for you guys to implement it going across. The very simplest one, but with least flexibility, is QnA. That's something that you can pretty much get up and running. You could almost do this before you leave today. Do it on your phone if you wanted to. Really, really straightforward. But obviously, you're limited in terms of what that can deliver. It's really designed for questions and answers, FAQs, that kind of thing. Then we have this FormFlow tool, which is basically for, imagine you got a form on your website for inquiries or something like that. It's taking those kinds of user experiences and providing them through a bot. Then finally, we've got this Dialogues, which gives you a lot more flexibility but it's a bit more effort to implement. But I think this actually is the most fundamental thing for you guys to understand. So we'll spend a bit of time going through that. The last thing I wanna talk about is, as you say, you can write your own Bespoke implementation, but it's a lot of work. But you can do that. So if you do want something that isn't tailored do by the SDK, you can write your own and make use of scaling and the benefits that the Connector gives you. We're gonna talk about this bit today. But yeah, that's an option for you guys if you're interested. We're gonna talk about Dialogues. I wanna run you through a scenario here to kind of cement what we're gonna talk about. Hopefully, you can see these slides, because there's some quite small text on here. But I'll try and explain what's going on. The idea here is, we've got a banking bot. Through this banking bot, we can either check our balance, our current balance or our savings balance, or we can make a payment. So it's pretty straightforward. Just got a couple of conditional branches. When we're talk about dialogues, we talk about dialogues in a stack, right? We start with one dialogue on the stack. If the bot is saying, "Would you like to do "one of these two things, "make a payment or check balance?" The user's saying "Make payment." At that point, we transition into another dialogue and we add the make payment dialogue to the top of the stack, right? Now we're inside the make payment dialogue, and the conversation continues. So the bot's saying, "Who would you like to pay?" You can give it an account number. "How much should I pay?" "100 pounds." And then the bot's confirming that it's paid it, at which point that dialogue gets popped off the stack and the conversation returns to the dialogue underneath. Then it's saying, it's just going back in a loop, saying, "Anything else I can help with?" This time, the user said, "Check balance," so we're going to check balance dialogue. Bot's saying, "Which account," so again, we go in another level. User's saying "Current," so we go into another dialogue, and so on. Hopefully that makes sense. We're kind of building up these dialogues in a stack as we go through the conversation. I think it is because it gives you a stack, basically, and that stack kind of persist state. You could, yeah, you could implement a bot as a repel. But you'd be limited in terms of managing different parts of the conversation, if that makes sense. Whereas using Dialogues, you can add them to each other and kinda build up a conversation and then backtrack. There's a few advanced techniques I'm not gonna talk about here. But something called the high scorable, which lets you interrupt conversations and do advanced things and add richness, which is really difficult to manually do yourself. Yeah, that's one way to think about it, yeah. In order to do these transitions that we talked about, adding things to the stack and waiting for messages inside the dialogue, we have these three call methods that we use. Context.wait waits for a message in the same dialogue. and context.forward add to the dialogue to the top of the stack and then transition into it. The difference is, context.forward you can pass a value in. And then context.done is the thing that removes the dialogue from the top of the stack. So I've got that demo to show you here. If I just open that. What I've done is, I've built up the diagram that you just saw in terms of, like, a hierarchy of dialogues. I'm not gonna go through all of this in detail, I just wanna kind of show you. Then, if you are interested, you can look at the code later and have a look for yourself. We start with the main dialogue. The main dialogue is type Idialog. Key thing here is couple of things. That can be a specific type. It's basically the type that you're returning. So when you remove something from the stack, let's say you wanted to return the amount as a value to the stack underneath it, that could be decimals, something like that. But a key thing here is the serializable attribute. Everything, all dialogues in Bot Framework are serialised. So when a conversation happens, it doesn't need to be kept in memory all the time. Bot Framework will take care of saving it to persistent storage. Then when the conversation's resumed, it will dehydrate that and continue the conversation in the right place. That could take maybe a month. There might be a month between the bot saying something and the user responding. So the very first thing we get when we come into a dialogue is, we come into the StartAsync. All we do in this example is, we wait for a message from the user, basically to initiate the conversation. Once that message comes in, I'm creating basically a card that the user can interact with and say yes or no. We then post that back to the user, here, context.Wait, post it back. Then we do that transition, context.Wait. That's where we are able to deal with what the user said. Then, when the user has basically responded to that message, we come into this post back, which is MessageReceived. What I'm gonna do to demonstrate this to you is that I set a break point for these two and I'm going to run this. I'm gonna initiate the conversation first, which is gonna fire the first dialogue. At this point, I'm just creating something called a HeroCard, which is just a visual element which we'll talk about later. I'm gonna send that back to the user and then I'm gonna wait, use context.Wait, for the user to respond. So I'm gonna continue here. We've got, you can press a button, basically. So I'm gonna say here, I'm gonna say "Make payment" here. That has fired another break point here. Now we're inside that break point, I can step over and I can see the message that's been sent to Bot Framework, it's kinda hard for you guys, if you'd like to see this. But basically this is all the information about the message that the user sent. So you can see which channel it's coming through, the message text, the time stamp, the locale of the user, loads of information. But the key thing is the message text. All I'm doing here is, I'm just saying, "If it equals check balance, transition into check balance, "otherwise, transition into make payment." Does that make sense? I'm just gonna continue on now. What I was gonna do, again, I'm not gonna labour this because it'll bore you to death. But we're basically adding dialogues to the stack. I'd encourage you to have a look at the code just to kinda see what's going on. That's a really good question. Bot Framework will do, it has these things called channel agnostic cards, which basically available in pretty much all channels, maybe not SMS. SMS is a really good example of where you want to be able to inspect the channel that user sent the message on. And if they're trying to interact with your bot over SMS, then you need to provide some alternative experience to that. And you can deal with it by inspecting that channel information. So here, I just wanna draw your attention to one other thing. We're in make payment at this point. I said before these are serializable, right? You guys have all used serializable if you've worked with .NET before. There's two members on this class, pay and amount. So we can just say, if I just do that, in fact, that's what we do. When the user says, "The payee is this," we save that. When the amount is this, we save that. Then we echo that back to the user, which is quite an important concept because what we're doing there is, we're actually retrieving information from the persistent state. Remember, this conversation could happen over a period of, I don't know, over months. Oh, the character is an emoji. It's a, I don't know what it is, actually. It looks like a dollar bill with wings or something. No, it's an emoji. 'Cause this is really boring. I mean, let's be honest. So I try to spice it up a bit by adding some visual interest. Now I'm just gonna say, "Right, I'm gonna pay "account 12345," or 1234. Then it's echoing back that 1234. Remember, we're now in a different method. So I'm gonna say, "100 pounds." It's retrieved that value from persistent state. Nothing that impressive there, but I just kind of wanted to point that out to you. So I'll leave it at that. Feel free to have a look at this code in your own time. You can, but you have to provide that behaviour yourself. It's doable, we've done it. But it doesn't, because there's no way for it to know without you telling it that this user on this channel is the same user as this user, so you have to kind of provide that. But yeah, you can provide. We've got an example here of a student inquiry. A student might talk to a bot and say, "I've got this question about my course," and the bot needs to capture certain information in order to record that inquiry. Maybe it goes off and saves it in CRM or something. But it's a very sort of linear process that we're going through. It's very, very similar to a web form. Bot Framework gives you this capability called Form Flow, which is very, very easy to work with. But you are limited in terms of what you can deliver. If you've got a problem that it makes sense to use it, then use it because it will really work, get you running quickly. But if not, then don't use it. We've got a demo here just to show you what that looks like. You guys now should be quite familiar with this. We've got a Web API controller here, and inside there, we just create, we basically transition in into our Form Flow down here. We're just basically transitioning into it. Then we've got our, basically this is an example of, we're appointing a problem with a parking metre. There's a bit of information that you wanna capture about whether it's a fault or whether this just got an inquiry. The ID of the machine or the postcode where it's located, and then obviously the problem. We define a class. Again, it's serializable. Then we provide this BuildForm behaviour. That's basically the thing that tells Bot Framework how to build up the conversation. You can see here, we're saying, first of all, capture IsFault, whether it's a fault or not. Then capture the machine ID or the postcode. Then we wanna confirm it. The way I'm confirming it here is, I'm just basically returning the Google Maps image with our postcode built into it. Most of the channels spot like that. Just to make it easy, I'm just embedding it. I'll point it asdown. And I'll show you what that looks like in a minute. Basically, this is confirmation. So the user can say yes or no to this. If they say yes, then we continue. If they say no, then it gives them the option to change it. Not really. That's one of the limitations of it. You have the context in the completion. But you don't have the context actually inside the form. And that's, I mean, that's really astute of you to ask that question, because that's is the big limitation of Form Flow. But as I say, if you got a really specific problem that it works for, it does work quite well. Here, I'm just saying, if it's a fault, then post one message. Otherwise, we just post another informational message to the user. I'm just gonna run this and just show you what it gives you out the box, having done no customization to that. It just say hello to the bot. It's saying, "You're having a problem with a parking machine?" I'm not sure that smiley face would be appreciated. And then it's saying, "Would you like an is fault?" I mean, it kinda makes sense. But yeah, it's not very polished, is it? I mean, probably customers wouldn't appreciate that, I'm sure, if you output it that. But I can say yes, whatever that means. Then its saying, "Please enter machine id or postcode." That's fine. I'm just gonna put in my postcode. Yeah. I'm gonna say yes. Then, "Please enter problem." So it's quite abrupt, isn't it? It's not very conversational. So, okay. I put money... At this point, you probably wanna hand off to a human because there's someone who's basically paid for a ticket and hasn't got one. That's not gonna make him feel any better, is it? So there's techniques that we can use to hand off to a human, which we'll talk about later. You absolutely can. And in fact, that's one of the examples that we talk about in the second session. What I wanna focus on here, though, is actually tidying this up a bit to make it a bit more intuitive. So rather than saying, "Would you like an is fault," you'd say something that's actually English. The way you do that is, you basically provide these attributes here. I've already filled these in. You can have a look at the documentation just to. Yeah, it was reflecting the type, basically. So it does that for you, which is really nice. But you probably wanna refine it a little bit. I've just put those attributes back in. I can start the conversation again. Now it's saying, "Are you reporting a fault with the machine?" I can say yes, and it's actually changed the way it's interacting with me now. Then it's providing some context, to say, "If you don't know the machine name, "then you can put your street address on there, as well." So that's kind of, tidies it up and makes it a bit more usable. If you have a look on the Bot Framework documentation site, there's loads of docs about this. Hopefully that made sense to everyone. Form Flows are really useful in certain scenarios. But there's this other tool that we can use called QnA Maker, which is really, really powerful for just getting something out there that's low barrier to entry. If you've got FAQs in your organisations, whatever organisations you are, whatever industry, you can make use of this and get a bot out there in an hour. It really is that straightforward. How is works is basically, you provide the URL of an FAQ, or you can provide a PDF document, and it will build everything up for you, to the website. I'll demo this in a minute. How it works is, you provide FAQ content. As I say, it could be a URL or it could a PDF or it could be a Word document. That goes into this QnA Maker service. It extracts the questions and the answers from that corpus and then creates a knowledge-based endpoint for you. You can then expose that over whatever channel you want. In this example, it makes sense to talk about Bot Framework. But you could use it for any other bot platforms or you could use it for, if you've got your own application that you wanted to incorporate Q and A in, you could use it for that. You're not limited to Bot Framework for this. I've got a FAQ, which is basically a set of questions and answers on Azure. You can see it's not formatted in question, answer. It's formatted in its own way. So it's intelligent enough to be able to parse this. I'm gonna go to QnA and actually show you how easy it is to create a new service for this. I'm gonna create a new service. I'm gonna give it a name. I'm gonna call it DotNetSheff Azure FAQ. Then we paste in the URL of our website. This is when you use your FAQ URL, if you've got one. But if you haven't, you can upload a file or you can even create a blank one and start building it up yourself. Then I just click Create and that's it. Now it's gonna go off and extract that information and build up a knowledge base for us to be able to moderate and actually expose. There's an API. So in theory, yeah, you can. There's basically a moderation API that you can use. So that's the way you do it. Likewise, if you had your FAQs in some kind of line of business system, you could connect it with the API and automatically pump it into there and then have it retrain automatically. It's quite a nice experience. You can see what it's done here is, it's extracted all the questions and answers. It's basically built up a knowledge base for us. So we can go in and test this now, and we get, like, a Chat Box experience. I can talk to this as it were a chat bot. I can say hi and it'll come back and say hello to me. I can ask it a question about Azure. I'm gonna say, "What is a preview service?" There's a natural language element here. "What is a preview service?" isn't a question in the FAQ. But it's picked the best match based on the intelligence that it has. You can see it's picked the right answer. But if it hasn't picked the right answer, I can say, "Actually, the right answer is this one" or, "Actually, it's none of these." And then I can go back into the knowledge base and add variations and retrain it. So it's really powerful. Once I've done with that, click Publish. Then I get an endpoint. It's just a restful endpoint, right? You just do HTTP post, you add a header, subscription key, a value, that's your JSON payload, question, there's your question. And it will give you a response from that knowledge base and put that natural language element into there. It does. It's backed by basically the corpus of the web. So Microsoft, you're using all the information you have about Bing, using it to build these intelligent models. Then your FAQs are contained within this particular service. But it applies that knowledge to contextualise it. So it's quite, you know, it gives you a lot for free. It's really quick and easy to get up and running. No. At that point, you'd need to provide more contextual NLP, natural language processing element, usingor something like that. But that's a really good example of a way you could maybe use QnA Maker to do the low resistance, 90% of frequently asked questions, and then what, the other 10%, you could maybe put it through and try and understand the intent and entities and that kind of a thing. Yeah, absolutely, absolutely. One technique that we see is actually, we've got a bot that's consuming FAQ information from different FAQs. And each one of these answers is basically ranked like a search rank. So basically how confident it thinks the answer is. So if you send a question out to five FAQs, it can tell you which the most likely answer is out of those five FAQs, and you can present that. So it's quite a nice way to lay it out together. That's a good question. If you use the tool, I'm gonna show something called the QnA Maker Dialogue, which basically you just plug it up to the QnA Maker service and it does everything for you. In that case, you basically get a question and answer back and that's it. But if you wanted more than that in the scenario you talked about, you could consume it as a restful endpoint and then provide that implementation yourself, because you get a full list of answers back. I'm just going to, I don't think I actually trained this. So I'm just gonna go ahead and train it before I do the next demo. The next demo is actually showing you the QnA service as a bot. I'm just gonna train this. Now we've defined our QnA service. Either say, you can consume this with any application. Doesn't need to be a bot. But it makes sense because we're talking about bots for me to show you how you can consume it with a bot. I've got a really simple example. This is using a NuGet package by MVP Gary Pretty, who works quite heavily in Bot Framework. He's basically using this QnA Maker dialogue. So by implementing the QnA Maker dialogue, I'm providing the details to this service. So the key and the ID, that's all we need to provide, basically, to get your bot up and running. I'm just gonna grab those two values. The key is this guy here. Then the ID is this one here. Then just run that. I'm just going to start up the emulator, connect to that, and then I'm gonna ask it a question. I'm gonna say, "What is a preview service?" Hopefully, that's gonna just connect to my QnA service and give me a response. Yeah, so there you go. It's presented that information for me. And obviously, I could just publish this out to Azure, and there you go, you've got bot that's able to answer questions and add that natural language element in 10 minutes. Really straightforward. We've gone through the basics, we've had a look at the tools that you can use to build up the conversations. Now we're gonna focus on how we can enrich them, adding things like natural language processing, adding visual elements to make the conversational experience a bit more interesting. Then we'll look at a scenario that's come up, I think you asked the question about how you could kind of route a human into the conversation. We'll look at a technique called Proactive Bot to deliver that. First of all, we're gonna talk about To cement this idea, I just wanna talk you through the terminology before we look at it. If we think of a scenario where we're basically doing home automation, so we can turn lights on, we can set the temperature, a user might be engaging with a bot in this way. He might be saying something like, "Set the temperature in the kitchen to 21 degrees." We call this, or luis calls this, an utterance. That's the whole spoken message. The intent is what the user's trying to achieve or what the user's trying to do. In this instance, the intent is, set temperature. Then the entities are also pulled out. In this instance, we have an entity for the room, kitchen, and then an entity for the temperature, to set the temperature to. That's also an entity. Let's talk through an example just so you can see this. The user speaks that utterance. That goes into the luis API. It understands the intent is, turn on the lights. It's not turn on the lights. I don't know why I've written turn on the lights. It's actually set the temperature. But hey. It's pulled out the room as being the kitchen and the temperature being 21 degrees. I mean, it's completely wrong but hopefully still makes sense. Oh, God. Luis gives us two things, basically, to deliver this end-to-end experience. It gives us portal, and we use that to define, train, publish our model, we call it a luis model. Then we get an API and we basically consume the model through that API. I've got a demo to show you the portal. I'm just gonna talk you through setting up a new luis model for a home automation scenario. I just go to Our models are in this bit here, under My Apps. So you can see I've got a few set up I've been testing with. I've already defined the Home Automation model that I'm using. So I'm gonna use that one when we come to the bot. But I'm just gonna talk you through creating a new one from scratch. First of all, we need to give it a name. I'm just gonna call it DotNetSheff Home Automation. Then we provide the culture. At the moment, we can provide these languages. So it will recognise more than just English. Give a description. Then we provide a key to use. This basically will use the Azure subscription, because basically this is a pay-per-use model. I think you get two million requests per month for free, and then anything above and beyond that, you have to pay for. It's reasonable, but there is a charge for it. I'm just gonna create that. Then once it's created, I can go in and start to build it up. The first thing I wanna do, if you just think back to our example, we're gonna maybe set the temperature in the kitchen to 21 degrees. I'm gonna define the entities there, 'cause they're the things I wanna pull out of that sentence. There's two different types of entities. The first one I'm gonna define as pre-built entity, and that's the temperature. Luis gives us the ability to use pre-built entities for certain types of data, things like age, daytime, dimensions, email, and so on. This is again using intelligence that Microsoft has built from the corpus of the web. So we can make use of that. So we don't need to train that low-level detail. I'm just gonna say, I wanna use temperature in my model. Then I'm gonna add a custom entity for my room. I'm just gonna give it a name, click Save, and that really basic level, that's all I need to do. I then go in and add the intent. If you remember, the intent is the thing the user's trying to do. In this instance, we'll just have one intent, which is to set temperature. We've defined our intent now. Now we need to tell luis what the phrase is, the utterance is, for set temperature looks like. This is basically where variations of language, speech patterns, that kind of things, comes in. What we normally do is, we will train it with different people. I'll maybe define the model to begin with, and then we'll have someone else come in, a tester, and kind of build up the richness, because you can have five different people say the same thing five different ways. So the key is really getting those variations in when you're training it. I'm just gonna define an utterance here, and I'm gonna say, I don't know, "set the temperature "in the kitchen to 21 degrees C." That's just kind of a simple example of something that a user might say. You can see it's already recognised that, temperature, it's pulled out the entity temperature based on that corpus of the web providing that intelligence. But it's, I've just noticed this, actually hidden the kitchen. So let me just type in another example. Set the kitchen, so I can give you a better example. Sometimes it pulls out stop words and sometimes it pulls out wrong stop words when you're training the model, which is fine except when you're doing a demo. It kind of stumps you. In this instance, I can basically classify this and I can say, kitchen is a room. Then I can build this up so I can say, set kitchen to 21 degrees C. Here, I can build all kinds of variations into this. Once that's done, I can train it. No, I don't wanna, I wanna save it and then I can train it and I can test it. I can test it, it's very similar to QnA Maker. You've got the same kind of interface where you train it and you publish it, and you can consume it with any client you want. So just gonna train this. In fact, what I'm gonna do actually is, jump to my pre-built application because I'll get a better quality result because I've trained this quite heavily. In here, if I just show you what we've got. We've got a number of intents. I've defined typical home automation activity intents, like setting temperature, turn the lights on, turn the lights off, opening and closing the garage doors. If I click Publish, you can see when we publish this out, you get an endpoint, a restful endpoint, which I can just call and I get JSON back. It's hard for you guys to see, but on the top, there's a Q in the query string, and that's where you basically put your query. So I can say something like, "set the kitchen to 21 degrees C," and hopefully, that's pulled out the, yeah, so it's pulled out the top scoring intent. Make it a bit bigger so you guys can see that. It pulled out the intent, which is set the temperature. It said it's 83% confident that that's the best answer. There it's also told me, maybe, actually, if it isn't the best answer, then these are the alternatives. So none is 0.1, turn lights on is less than 0.1. So it's fairly, fairly confident that that's the right answer. But the other thing that it's done is, actually pulled out those entities, as well. It's pulled them out as part of the JSON type. So consuming this as a client, I can basically say, "Give me your entities," pull those out, and then I can use that to make intelligent decisions. So maybe I'd be connecting to, like, a hive, a heating controller or something, and I'd be saying, "Turn the temperature up," or something like that. What I wanna do now is, jump to the slides. Then we'll take this a bit further. Does that make sense to everyone, first off? Yeah, clear? Let us have a look at how we can use this intelligence and make a more intelligent bot. A user comes through the channels, whatever channels we've exposed our bot on. We then intuit the bot application, which is our API controller. We can then provide the integration with luis through something called Luis Dialogue. The luis dialogue is bound to the model. Basically here what we're doing is, we're passing in an utterance and we get an intent and entities back in JSON. That comes back into our dialogue and then we can make some intelligent decision based on top of that. Let's have a look and see how we can do that. It's pretty straightforward. I've gone to the demo. Still this guy. I'm just gonna start at the top. If you remember, back to QnA dialogue, this is very similar. We basically extend Luis Dialogue. We define our luis model. This is the thing that binds the dialogue to our luis model. Then here, we just provide a model ID and a subscription key. I'm just gonna grab these from here. Model ID is that one there. Then the subscription key is that one there. Then actually in the dialogue what we've got is basically a bunch of methods. Each of those methods are mapped to an intent, a luis intent. Where we've got, if we just turn back to luis, you can see we've got settemperature, blah, blah, blah. Then each one of those is mapped to a method, basically. So pretty straightforward. I'll let you look through these in detail, but we've basically got one for each intent. But the interesting one is this settemperature, because here, we're pulling out the room from the list of entities that's passed into the dialogue. We're also pulling the temperature, as well, that's passed in. Then we're just outputting that to the chat bot. But that's, if you're integrating with a piece of physical hardware, that's where you put that integration. I'm just gonna run this and now, hopefully, it should connect up and we should be able to talk to it, to our bot, so I can say, "Set the kitchen to 21 degrees C." So it's pulled out the intent, it's pulled out the room, it's pulled out the temperature. So we can use that to make a decision. But, you know, I'm being quite prescriptive about how I say, "set the kitchen to 21 degrees C." But the idea is, the more variations you train it with, the better it'll get at recognising other variants. So I'm gonna go up, in there, and just try a different one. "Turn up the kitchen to," see if it recognises that. It has. It's pulled out the intent is still setting the temperature. And it's worked out that I'm talking about the kitchen, I'm talking about the temperature. I haven't trained that particular phrase, but it's using the intelligence I built up and doing this. You know, if you put the training in, you can get some good results from that. It's not context-aware at all. So any contextual information you need to provide yourself. You need to basically layer it on top of it. You've got the tools to build that intelligence on top of it. But luis isn't context-aware in its own right. So hopefully that was useful to everyone. We've talked about adding natural language. Now we talk about actually adding some visual interest. Rather than the intelligence, we're looking at how we can make it more engaging and make the users use the bot. We talked about bots so far, we've been talking really about text and conversation. So user says something, the bot responds in text. We've got an example here of a bot that maybe is telling someone that a flight's been delayed. It's not a very compelling user experience. But if you were to use something called an airline card that Facebook Messenger gives you, you can present that information in a more engaging way. It's not just that particular example. There's a whole bunch of different sort of rich elements that you can add to your bot to make it more engaging. You can see we've got some examples. We've got maps. We've already seen examples of a map. We've got lists of products. We've got another airline ticket there. We've got, like, a Hero Card, we've got a graph in this one. We've got buttons built into this one, and we've got, like, a master detail view there, as well. There's quite a lot of richness that we can build in. The way we do that in Bot Framework is, we can use these things called generic cards. These are things that, I mean, within a margin, you'll write them once and they'll basically work on any channel. The caveat to that is basically SMS, because SMS doesn't have any kind of visual element. They'll work on most of the channels, not SMS. The idea is, by implementing a generic card, by using these built-in classes that Bot Framework gives us, we can basically write it once and then just distribute it across whatever channels we want. These are the steps we need to follow. We create the activity, as we've been doing so far. That's just, so far we've been creating messages as activities. But we can attach attachments to those messages. We use these as strongly-typed elements. We create instances of those. Then we just add those to the activities. So it's pretty straightforward. Now, if we've got a channel-specific card, this airline one, for example, is only available on Facebook Messenger. You can target those channel-specific cards, as well. The way you do that is, again you create your activity. You create basically a JSON payload. If you have a look on Facebook Messenger or Slack, and you have a look at the cards it lets you build up as part of a conversation, it'll show you the JSON that you need to create to consume that. If any of you guys have been creating bots using native platforms, you probably have already seen these. But Bot Framework allows us to leverage these and target channel-specific behaviours. Then we just assign it to the activity in a similar way to before. But rather than adding it to the list of attachments, we're assigning it to ChannelData this time. So the approach is very similar. I've got a demo here just to kind of illustrate the generic and the channel-specific. I've deployed this before I came over, just so you're not watching me deploy this. I'm just gonna, got a few examples that we can use. Let me start with carousel, all right? I'll show you the code that creates these in a moment. I'm sending a message to the bot, and the bot's just presenting the card back to me, just to show you, you can build visual interest quite well. You can see here, we've got a carousel which I can navigate. I can put whatever content I want in there. If I wanna put a map in there, I can put a map in there. I can mix it up, I can have different elements on the different parts of the carousel. It's quite rich. I'm just gonna have a look at some of the other ones. Then we got our receipt card. This basically gives a receipt. So we made a purchase and we get a nicely formatted message that shows us our purchase, and we can attach images to that. That's built into Bot Framework. This is on Facebook Messenger, but if you use the same thing on a different channel, you get the channel-specific behaviour. But because you're using a generic card, Bot Framework will take care of formatting it for the appropriate channel. Then I'm just gonna show you the airline one, 'cause it's quite an interesting one. Imagine that we've booked a ticket. When we wanna check in, we can present this information to a user through a Facebook Messenger application. Again, it's just coming back to this idea that you don't need to have a native app experience to deliver quite rich functionality. There's tonnes more of these types of cards. A lot of these are channel-specific. But the key thing for you guys is, you can take advantage of them by using custom channel data. It depends on the channel. Some channelsyou basically a blank message. Other channels, it just won't show anything. It's a good question. You do need to test the channel support. So why it's safe for that is, if we have a look on dev Bot Framework, there's something called a channel inspector. My WiFi has died. Right. So yeah, basically with Bot Framework, we've got this tool called the channel inspector. It's basically an online tool that you can use to inspect channels, basically, so you can see what messages look like in different channels. You don't actually have to get your application out there in order to use it. If you have a look in the code, this is pretty straightforward. We're just using a dialogue. So you guys should be familiar with now. When the message comes in, this is basically the message I'm sending. I'm just doing the switch statement. I'm saying, if I say carousel, then show that. If we just have a look at one of the examples, I'll pick one of the simpler examples, the Hero Card, for example. I'm just building up basically Hero Card down here. This turns it, like, out of the box cards that you can make use of. They're on the Bot Framework website. So you can have a look at that in your own time. The one thing I do wanna show you is the airline one, because that is specific to the channel. So the way we deal with that is slightly different. Here what we do is, we create our object basically using an anonymous type. That gets serialised as JSON, so it's basically a JSON object. Then I'm just attaching it to ChannelData down there. If you go on the Facebook Messenger SDK website, you'll see this example is on there and I've literally just lifted and shifted it from there and removed the JSON, quote, space that turned it into an object. But there's nothing else you need to do. It's quite nice. Actually, before I move on to that, I just wanna talk about different scenario, which is Facebook webview. This is another type of visual element that you can add to your page. It's specific to Facebook. But what it does is, lets you embed a web view inside your chat window, which is quite powerful if you're using Facebook. But it is limited to Facebook. So I'm gonna kill my Internet connexion again and show you what that looks like. The idea is, we can basically ask the pizza delivery bot where our pizza is. Again, this is specific to Facebook Messenger and should pop us as a public figure. You're just sending a message, and I can say, "Where's my pizza?" Then it's gonna respond. It's gonna query its system, internal tracking system, and it's gonna give me a response based on the actual location of the pizza. They're saying, "Okay, it's on its way." So I can click this, and that gives me an embedded web view actually inside the chat window. So it's fully contextual. This is just a Bing map embedded on the page. That's why it's delayed. But it's fully contextual, so I can pass the information from the conversation into this web view, and then, you know, maybe I've got secure checkout, payment or something. I pay with checkout and I can complete that inside a secure website that's maybe hosted on our infrastructure, and then complete the process. And then I'm returned to the chat and I can pass the information to the web view and from the web view. So it's really powerful. It's quite underused at the moment because I just think people are not aware this thing exists. I'm gonna show you the code for this, 'cause it works in a very similar way to the custom cards that we just looked at. Slack does, Skype does, Teams does, Facebook Messenger does. I haven't used that many of the other ones. I think the ones I've tried it on have done, but I haven't used the like, you know, things like Telegram and Kick and things like that I haven't used too much. But the big ones are quite, they have good levels of support. Let me just pull up the code for that one, as well. Again, you guys wanna take this code away with you, you're welcome to have a look. So in here. First of all, this is my dodgy website. It's literally hard coded location, if you hadn't guessed already. There's nothing complicated in there. But in our bot, what we do is, again, we've created our payload, JSON payload. And this is, it's a message. But inside that, we've got a web URL. By basically enabling messenger extensions and setting the how big we want the web view to appear, we can make it a bit contextually inside that chat window and we can pass the information between the bot and the page using the JavaScript It's quite powerful used in the right situations. What I would say is, it's not appropriate for everything because what you don't wanna do is take a website and just embed it in a bot. That doesn't make sense. You need to use it for really discrete activities that make sense in part of a conversation. Yeah, absolutely. I mean, that example is HTTPS, and that's basically running on my SUS hub. That's not going through Facebook. The only things that are going through Facebook is the chat and any information I choose to send from that particular window. So it's fully secured. Now we're gonna talk about proactive bots. The bot communication isn't synchronous all the time. Just to give you an example, "Your flight is delayed" is an example of that. Long running tasks, so maybe we've got build process. You know, someone talked about Who Bot before, that kind of thing. You know, we started building them. When it's done, maybe an hour later, it'll let you know. And then putting a human in the loop as well, which is a really common scenario. If you want a human to be built into the workflow, this is a way you can achieve that. I've come up with this scenario. Hopefully, this diagram is clear to everyone. I'll try and explain it best I can. But basically what we've got here, on the left-hand side, we've got our user. In the middle we've got our bot. Then at the top we've got some kind of service. This is basically a machine service that can provide answers. During the normal course of conversation, we'd hope to direct, like, 90% of our requests to that service to be automatically handled. But maybe there's a question that the user has that can't be answered by the service, at which point we need a human, this one here, to basically perform some kind of manual task. They're gonna maybe do some investigation about a support issue, for example. Let's talk about that example. So user's reported a support issue, a problem with the website, the machine can't work out what the problem is so a human needs to kind of do some investigations. The way we do that is, we use something called the resumption cookie. So that the service basically saves the information inside a resumption cookie. That can be used to continue the conversation and route it back to the user through the same channel. So from the user's point of view, they're still interacting with the same thing. But behind the scenes, there's some kind of, you know, long-running human process there. Yeah, it's fully contextual. So everything's preserved. Maybe they've reported an issue and maybe they've got angry and it's being escalated, that kind of thing. You can kind of build that all into the process. I've got a really simple example of a customer support bot, which kind of demonstrates the previous slide. What I've got here is, I've basically set up two controllers. The first one is pretty straightforward. You guys have all seen this a million times now. It's our messages controller. Whenever a message comes in, this is the world's most useless support bot. Basically what it does, it just says, to anything, "I don't know how to deal with that, "so I've passed in on to a human." And then it saves this cookie, the cookie on the diagram is basically a conversation reference. That has all the information that's needed to be able to continue the conversation at a later time. In this instance, all I'm doing is just saving that to disc. But obviously in a real application, you might want to put that somewhere a bit more scalable. That would be part of the conversation. You could provide information out. But this is literally this metadata about routing. It's not actually the contents of the message itself because that's all preserved as part of the conversation. But what we need to do is basically interrupt the conversation emanating from this process. So now there's data, we need some, basically a way to call back. That make sense? So we save this information. It's got, like, channel information, who the recipient is, the ID of the conversation, that kind of thing. Save that to disc. I'm just gonna run this now. Sorry, before I run it. The next part of this puzzle is the callback controller. This is the thing that I'm gonna send the proactive message to. Imagine you've got a process where a human's maybe done something. They've clicked a button to say, "I'm done now." This controller would then be called with the information needed to complete the process. What we do here is, we post a message back to this HTTP endpoint. We read the cookie, the resumption information from this JSON file that we saved when the user initiated the conversation. We read that, and then we use that to create a reply. That's kind of the magic that glues everything together. We create a reply and we just send the message that's being passed into this controller method back to the user. Does that make sense to everyone? We got two controllers. One is the bot controller, which saves a cookie and basically says, "We'll get someone to sort it out." Then we got a human that's doing something, at which point they click a button, it sends a message back to this controller, which resumes a conversation at the right place, basically. I'll just run this real quick. Let's say I'm reporting a problem with a website. "Your website is showing a 500." Then we've responded with the message, and it's basically said there's a human gonna do some processing. So use your imagination. Maybe you've got some kind of support system in place there. We've saved a cookie out to, let me just track that in here so you can see it. Inside that cookie, we've got all the information we need to resume the conversation. We've got things like the activity ID. That's the place in the conversation. Who the user was, who the bot was, the ID of the conversation, and the channel that's being used to complete the conversation. This process is exactly the same for all channels. It works with all channels. The user's reported a problem. There's a human doing something. When the human's finished, they're gonna press a button. To simulate that, I'm just gonna run a HTTP post. I'm just gonna issue a curl post. I'm just posting to my callback controller. I'm saying, "The problem with the website is fixed." So, you know, the user's filled this information in. Just posting it to that endpoint, up theJSON. If you just watch carefully as I do this. Nope, there's no while on it. I'll just put that over the top. So if I just press enter, the message will get routed back to the conversation and the right recipient. You can use that technique for a load of different scenarios. Any long running process you've got is really powerful, really simple. You just need to make sure you save that information somewhere persistent and do a better job than I have, because I've just saved it in a single file. So if another user comes onboard, they can override that information. Sorry I kinda rushed through that. But I just wanted to show you that one. That kinda wraps everything up. Hopefully this has been useful for you guys. Hopefully you've seen quite a few things that got you thinking. Next steps for you, really, if you are interested, go sign up at, download the emulator and the SDK of your choosing. Apologies that I couldn't show you any Node stuff today. That's really just down to me being a bit rubbish at Node. Then think about how you can apply these to your own applications, because you guys are gonna have some really, really interesting use cases, I'm sure. As I say, if you want the code, it's there. I've posted this is the slide channel, as well. I'll just leave this on as we kinda wrap up, just to, if you've got anything you wanna look into, there's quite a good level of information there. Got myself on Twitter and got my blog, and Gary Pretty, who I work with quite closely on Bot Frame