logo

Natural Language Processing & Natural Language Understanding: In-Depth Guide in 2021

This is a comprehensive article about natural language understanding. How it works, and the different applications it can have for businesses.

1 Jun, 2021

Updated 23 Dec, 2021

NLP vs NLU

Share

  1. What Is Natural Language Processing?
  2. What is Natural Language Understanding?
  3. How Does NLU Work?
  4. How does NLP vs NLU differ?
  5. Use Cases for NLU

Computers excel in responding to programming instructions and predetermined plain-language commands, but we are just in the early phases of them understanding natural language.

A simple command like “Hang up the phone,” for example, has historical and colloquial contexts that shape its meaning. The human mind understands this phrase quickly, but computers might not.

Fortunately, advances in natural language processing (NLP) give computers a leg up in their comprehension of the ways humans naturally communicate through language.

Success in this area creates countless new business opportunities in customer service, knowledge management, and data capture, among others. Indeed, natural language understanding is at the center of what Botpress seeks to achieve as a company—helping machines to better understand humans is the goal that inspires our development of conversational AI.

Although implementing natural language capabilities has become more accessible, their algorithms remain a “black box” to many developers, preventing those teams from achieving optimal use of these functions. Grasping the basics of how it works is essential to determine what kind of training data, they will use to train these intelligent machines. Selecting and applying the right training data is critical to success.

In this article, we review the basics of natural language and their capabilities. We also examine several key use cases and provide recommendations on how to get started with your own natural language solutions.

What Is Natural Language Processing?

Natural Language Processing is a subfield of artificial intelligence studying the interactions between a computer and human language. It's a field of study that combines linguistic and computer science. The purpose of NLP is to transform a natural language input into structured data. It uses a multitude of tasks to do that, such as; part-of-speech tagging, named entity recognition, syntactic parsing, and more.

What is Natural Language Understanding (NLU)?

Natural Language Understanding is about the comprehension of the language. Similar to us, the technology can hear or read something without understanding it. The NLU is the technology that powers conversational interfaces. Without the understanding part, the conversation is nearly impossible or at best awkward.

How Does NLU Work?

Like other AI solutions, this technology requires training. Intent detection depends on the training data provided by the chatbot developer and by the platform engineers’ choice of technologies. These specialists must supply training data to ensure the tool understands users within the context of its function—whether that function is servicing external customers or assisting internal users with knowledge management. Even with training, NLU will get lost as conversations steer away from its core functions and become more general.

Fortunately, these technologies can be highly effective in specific use cases. Optimizing and executing training is not out of reach for most developers and even non-technical users. Recent breakthroughs in AI, emerging in part because of exponential growth in the availability of computing power, make applying these solutions easier, more approachable, and more affordable than ever.

“To gain that understanding, machines need to be able to understand and generate parts of speech, extract and understand entities, determine meanings of words, and use much more complicated processing activities to connect together concepts, phrases, concepts, and grammar into the larger picture of intent and meaning.” Forbes, “Machines That Can Understand Human Speech: The Conversational Pattern Of AI,” June 2020

Language is complex—more so than we may realize—so creating software that accounts for all of its nuances and successfully determines the human intent behind that language is also complex. But as with human intelligence, sufficient training of AI enables a machine to overcome these complexities (if the training data is well-shaped enough).

Training AI has specific requirements unique to each AI’s use and context. For example, let’s assume we intend to train a chatbot that employs NLU to work in a customer service function for air travel. The chatbot will process the natural language of customers to help them book flights and adjust their itineraries.

In this case, a chatbot developer must provide the machine’s natural language algorithm with intent data. This data consists of common phrases travel customers may use to create or change their bookings. The natural language algorithm—a machine learning function—trains itself on the data so that the conversational assistant can recognize phrases with similar meanings but different words.

Ideally, this training will equip the conversational assistant to handle most customer scenarios, freeing human agents from tedious calls where deeper human capacities are not required. Meanwhile, the conversational assistant can defer more complex scenarios to human agents (e.g., conversations that require human empathy). Even with these capabilities in place, developers must continue to supply the algorithm with diverse data so that it can calibrate its internal model to keep pace with changes in customer behaviors and business needs.

To this end, a method called word vectorization maps words or phrases to corresponding “vectors”—real numbers that the machines can use to predict outcomes, identify word similarities, and better understand semantics. Word vectorization greatly expands a machine’s capacity to understand natural language, which exemplifies the progressive nature and future potential of these technologies.

Tips to build your dataset

  • Stick to one concept per intent (an intent contains multiple utterances)
  • Try mixing synonyms in the utterances
  • Write your utterances with language your persona would use
  • Use entities
  • Avoid spelling and grammar mistakes

Here is our complete guide to build a training dataset for your chatbot.

How the NLU works
How the NLU works

What NLP technology is not?

NLP technology is not yet a human-level understanding of natural language. There have been tremendous breakthroughs in AI due mostly to the recent abundance of data and computing power. NLP, however, is not generalized intelligence.

While it can be impressive that chatbots understand spoken and written phrases, developers need to be highly cognizant of the context in which these bots are used. They only want to deploy bots in environments where the questions or intent the chatbot is likely to get have been previously learned.

How does NLP and NLU differ?

NLU is a subtopic of NLP, which is why the line between the two seems so thin. NLP includes not only NLU but also natural language generation. Natural Language Understanding is not only about analyzing the free text in a literal sense and creating structured data; it’s also about understanding the meaning and drawing insights. Not all tasks of NLP are not focused on lexical semantics.

Natural language understanding needs to use tasks from NLP to prepare the data for text summarization or question answering. The word “understanding” is critical in this context. This familiarity requires artificial intelligence (AI), trained to develop a deep understanding of language parsing so that the computer can identify “intent” and distinguish it from “what was said.”

List of tasks involved in NLU and NLP
List of tasks involved in NLU and NLP

Leading Use Cases for Natural Language Understanding

At Botpress, we employ word vectorization for all of our NLU-enabled technologies. In this context of chatbots, word vectorization lends itself to a number of key functionalities critical to future use cases for conversational assistants, including:

  • Intents classification, or understanding the customer’s goals
  • Entities extractions and slots filling or extracting actionable information
  • Contextual awareness, for accurate answer prediction
  • Multi-language support, featuring seamless translations without multiple conversational assistants

In this way, humans do not need to conform to a machine’s “ways of doing things”—that is, swiping, clicking, or communicating in code. When computers employ natural language and its corresponding capabilities successfully, a wide variety of use cases across industries emerge, including:

  • Knowledge management for internal teams. Business users can rely on natural language to summon any reference material they need at a moment’s notice, without “digging” or using tedious manual processes to search.
  • Customer-facing chatbots. Customers can interact online or over the phone with a conversational assistant that can answer questions, change account information, adjust or cancel recent transactions, and more—without a human agent getting involved and without the customer waiting in a queue for service.
  • Employee-facing chatbots can support a team in a wide variety of tasks ranging from human resources process automation and question answering to IT service management (ITSM).

These are only a few examples. In truth, we are only beginning to even imagine the applications of natural language understanding, because its functions might be highly useful in any number of tasks where human intelligence can be automated or displaced.

Natural language processing will become the foundation for a new era in software and it always a critical component of any conversational AI dev tool.

NLU and the Future of Conversational AI

Natural language algorithms make up only a single category within the broader AI ecosystem that enterprise companies are beginning to embrace today. But “Conversational AI,” the business function these algorithms power, is expected to become a $25 billions market by 2024—more than tripling in size since 2019, The Wall Street Journal predicts.

Finally, I want to mention that NLP is very easy to set up. You don’t need to be a developer to set up NLP and in fact, it is not the best use of a developers time. It is true that normally a developer is needed to program the action that takes place after an intent it recognized, but anyone can set up the NLP especially the phrases associated with each intent. You need to agree at the start which intents will be set up and what naming convention you will use, but once these are set up, anyone can add phrases to the intents.

Download it and get started in 38 seconds...Download Center

Marc Mercier

Marc was the first marketing hire at Botpress and is now acting as Chief of Staff. He spent the past 7 years working for tech startups in various roles, but his strengths are in operations and marketing. Marc is an avid learner who's always trying to learn more and improve.

Related Articles

20 May, 2022

What's Happening at Botpress: May 2022

Botpress launched its new Learning Center this month! Plus, two new OpenBook webinars.

9 May, 2022

Snippet Components — A New Module for Botpress

Introducing the Snippet Components — a new module created by Botpress. This module will help you create flows faster and gives some basic examples with other skills.

2 May, 2022

Introducing the Botpress Learning Center

A new place to master Botpress, chatbots, and conversational AI.