Brillig Understanding, Inc.

The flaws of Machine Learning

Machine Learning (ML) has demonstrated many useful abilities. But its use for natural language comes with a number of problems and ChatScript is often more effective (as Kore demonstrates).

Machine learning works by giving it a series of sentences of tokens (what we think of as words but in actuality there is no understanding of them as words). Give enough sentences and the system can learn to “generalize” and detect the intent of an input sentence. Intent typically means what command do you want a bot to execute. Correspondingly, you can use ChatScript to script detection of an intent. Both techniques involve matching a pattern. With ChatScript, you know what the pattern is and you can explicitly alter it. With ML you have no clue what its pattern is and changing it involves providing even more sentences as examples of sentences that do and do not match the intent.

Using ML involves almost no skill. Someone skilled picks the ML technique to use, and then anyone can add sentences as examples. If the system makes a mistake, you just add the new sentence with the right or wrong mark, and tell it to relearn. This seems so easy (though relearning takes noticable time for a bot with many intents). Almost every single bot development environment uses this technique.

But there are hidden issues. First, no one warns bot developers that typically at least a thousand sample sentences per intent are needed. ML works with big data, data the typical bot developer does not have. This means the developer will give a bunch of examples, think it works, release it to the public, get back failures one-by-one and then add new sentences. This will take quite some time to actually get the system to work well, time the developer does not expect. Facebook's AI Research team estimated that they would need many millions of queries to optimize a single restaurant booking intent. In contrast, ChatScript uses small data. One can use Fundamental Meaning to define intents by using a verb, its synonyms, and a direct object noun.

The typical ML system requires both intent and entities. Entities are the data detected to go along with the intent, like “what is the weather in Seattle”. Since machine learning does not handle entities, systems provide predefined entity types. One disadvantage of such systems is that entities are not defined by machine learning and are not definable by the bot author (with the exception of enumeration data). So if the system doesn't currently support the entity you need, you are stuck. ChatScript allows you to define any entity you need.

Fine, so what else can't ML systems do? Bot platforms must learn each bot's intents independently. So you need to name the bot in addition to the intent. E.g., “Alexa, use WeatherUnderground and tell me the weather in Seattle.” Having to name the bot on every interaction is tedious. This is not required with ChatScript. One central bot can determine what bot is needed to use an intent.

And because ML has independent bots, they cannot pass along data from one bot to another nor track context of recent conversations, nor detect multiple requests in a sentence. So ML cannot handle these:

Tell me the weather in Seattle. And in Chicago.

Tell me the weather in Seattle and book me a flight there.

Nor can you interrupt a conversation with a bot.

User: Book me a flight to Seattle.

Bot: When would you like to fly?

User: First, schedule a meeting for next week.

… conversation about meeting

… when complete, return to flying request.

ChatScript can handle these things.

By the way, most companies use ML to perform sentiment analysis. Like IBM Watson, for example. Yet it turns out that Kore's sentiment analysis using ChatScript is more effective than IBM's.


Home About Us Technology Projects Testimonials ChatBot Demo Awards/Press Publications Contact