Everything You Need to Know About Natural Language Understanding (NLU) and Beyond

By Botpress

Computers excel in responding to programming instructions and predetermined, plain-language commands, but they continue to struggle with the imprecise nature of normal human language, both in speech and in text. A simple command like “Hang up the phone,” for example, has historical and colloquial contexts that shape its meaning. The human mind understands this phrase quickly, but computers do not.


Fortunately, advances in natural language processing (NLP)—which encompasses both natural language understanding (NLU) and natural language generation (NLG)—give computers a leg up in their comprehension of the ways humans naturally communicate through language. These processes break down the language to identify the intent of the human delivering it, allowing the computer to act upon that intent accordingly.


Success in this area creates countless new business opportunities in customer service, knowledge management, and data capture, among others. Indeed, natural language understanding is at the center of what Botpress seeks to achieve as a company—helping machines to better understand humans is the goal that inspires our development of conversational AI.


Although implementing natural language capabilities has become more accessible, their algorithms remain a “black box” to many developers, preventing those teams from achieving optimal use of these functions. Grasping the basics of how these algorithms work is essential to determining what kind of training data, also called “intent data,” they will use to train these intelligent machines. Selecting and applying the right training data is critical to success.


In this article, we review the basics of natural language algorithms and their capabilities. We also examine several key use cases and provide recommendations on how to get started with your own natural language solutions.


What Is Natural Language Understanding?


Simply put, NLU is a technology that can identify the meaning of “intent” behind phrases either spoken or written in everyday human conversational language. The machine then takes a desirable action based on its understanding of that intent—whether it is providing information, making a recommendation (via text or automated voice), executing a transaction, changing account information, or activating a feature or function, such as canceling a service.


The word “understanding” is critical in this context. This familiarity requires artificial intelligence (AI), trained to develop a deep understanding of language parsing so that the computer can identify “intent” and distinguish it from simply “what was said.”


Like other AI solutions, this technology requires training. Intent detection depends on the training data provided by the chatbot developer and by the platform engineers’ choice of technologies. These specialists must supply training data to ensure the tool understands users within the context of its function—whether that function is servicing external customers or assisting internal users with knowledge management. Even with training, NLU will get lost as conversations steer away from its core functions and become more general.


Fortunately, these technologies can be highly effective in specific use cases. Optimizing and executing training is not out of reach for most developers and even nontechnical users. Recent breakthroughs in AI, emerging in part because of exponential growth in the availability of computing power, make applying these solutions easier, more approachable, and more affordable than ever. 


How Does NLU Work?


“To gain that understanding, machines need to be able to understand and generate parts of speech, extract and understand entities, determine meanings of words, and use much more complicated processing activities to connect together concepts, phrases, concepts, and grammar into the larger picture of intent and meaning.”

Forbes, “Machines That Can Understand Human Speech: The Conversational Pattern Of AI,” June 2020


Language is complex—more so than we may realize—so creating software that accounts for all of its nuances and successfully determines the human intent behind that language is also complex. But as with human intelligence, sufficient training of AI enables a machine to overcome these complexities.


Training AI has specific requirements unique to each AI’s use and context. For example, let’s assume we intend to train a chatbot that employs NLU to work in a customer service function for air travel. The chatbot will process the natural language of customers to help them book flights and adjust their itineraries.


In this case, a chatbot developer must provide the machine’s natural language algorithm with a massive amount of “intent data.” This data consists of common phrases travel customers may use to create or change their bookings. The natural language algorithm—a machine learning function—trains itself on the data so that the conversational assistant can recognize phrases with similar meaning but different words.


Ideally, this training will equip the conversational assistant to handle most customer scenarios, freeing human agents from tedious calls where deeper human capacities are not required. Meanwhile, the conversational assistant can defer more complex scenarios to human agents (e.g., conversations that require human empathy). Even with these capabilities in place, developers must continue to supply the algorithm with large volumes of data so that it can calibrate its internal model to keep pace with changes in customer behavior and business needs.


To this end, a method called word vectorization maps words or phrases to corresponding “vectors”—real numbers that the machines can use to predict outcomes, identify word similarities, and better understand semantics. Word vectorization greatly expands a machine’s capacity to understand natural language, which exemplifies the progressive nature and future potential of these technologies.


Leading Use Cases for Natural Language Understanding


At Botpress, we employ word vectorization for all of our NLU-enabled technologies. In this context of chatbots, word vectorization lends itself to a number of key functionalities critical to future use cases for conversational assistants, including:

  • Intents detection, or understanding the customer’s goals
  • Topic classification, or associating those goals with certain business functions
  • Entities and slots extraction, or extracting actionable information
  • Contextual awareness, for accurate answer prediction
  • Multi-language support, featuring seamless translations without multiple conversational assistants


In this way, humans do not need to conform to a machine’s “ways of doing things”—that is, swiping, clicking, or communicating in code. When computers employ natural language and its corresponding capabilities successfully, a wide variety of use cases across industries emerge, including:


  • Knowledge management for internal teams. Business users can rely on natural language to summon any reference material they need at a moment’s notice, without “digging” or using tedious manual processes to search.


  • Customer-facing chatbots. Customers can interact online or over the phone with a conversational assistant that can answer questions, change account information, adjust or cancel recent transactions, and more—without a human agent getting involved and without the customer waiting in a queue for service.


  • Assisting the elderly. Systems can read emails out loud in a natural-sounding voice, answer questions about important personal information, remind users about appointments, and analyze users’ sentiments when making requests.


These are only a few examples. In truth, we are only beginning to even imagine the applications of natural language understanding, because its functions might be highly useful in any number of tasks where human intelligence can be automated or displaced.


NLU and the Future of Conversational AI


Natural language algorithms make up only a single category within the broader AI ecosystem that enterprise companies are beginning to embrace today. But “Conversational AI,” the business function these algorithms power, is expected to become a $15.7 million market by 2024—more than tripling in size since 2019, The Wall Street Journal predicts. 


Fortunately, natural language functions of all types have become quite easy to launch, and you don’t need to be a developer to set one up. With a deep understanding of your customers and the right partnership, you’ve got everything you need for success. Contact an NLU expert at Botpress today to learn more.


Join 30 000+ developers and subscribe to our newsletter.


Disclaimer: We encourage our blog authors to give their personal opinions.  The opinions expressed in this blog are therefore those of the authors. They do not necessarily reflect the opinions or views of Botpress as a company.