How to Add Languages to the Botpress NLU

By default, Botpress gives you a list of 12 languages to work with. These languages were selected based on our experience and the experience of developers using Botpress.

In this blog post, I will go over the language extension of the NLU service for Botpress. You will need to have an on-premises or hosted version of the NLU service.

Requirements

You will need the following:

• Internet connection to load the model and word token
• On-premises version of the NLU service
• Linux

‍

Important files

To add a new language to botpress you will need two files:

• Word embedding (.bin) file
• Model (.model) file

‍

Word embedding

You can select from this list of languages (make sure to NOT USE THE WIKI on that list). Download the bin file. Heads up, this file is BIG, so don't be surprised.

This file needs to follow this format: bp.{Language - 2 Char ISO Code}.{Dimensions}.bin

‍

Model

You can find the model here. The model needs to match the same language you have chosen in the word embedding section.

The bigger the vocab size, the longer the NLU will run (find the right balance for you).  

This file needs to follow this format:  bp.{Language - 2 Char ISO Code}.bpe.model

‍

Transport and Load the File

Those two files need to be in the same folder.  Example:

$ git clone https://github.com/botpress/botpress.git$ cd examples/docker-compose$ mkdir -p botpress/language$ mv ~/Download/BIN_FILE botpress/language$ mv ~/Download/MODEL_FILE botpress/language

Word Embedding

The word embedding needs to be gunzipped (german example) and put in the Botpress format:

$ gunzip cc.de.300.bin.gz$ mv cc.de.300.bin bp.de.300.bin

Model

The model needs to be in the botpress format

$ mv de.wiki.bpe.vs5000.model bp.de.bpe.model

Start the docker-compose or your NLU binary

Modify the docker-compose-standalone-nlu.yaml and modify the command: value under the botpress_lang service

 botpress_lang:    build:      context: ./botpress      dockerfile: Dockerfile    image: botpress/server    command: ./bp lang --offline --dim 300 --langDir /botpress/lang --port 3100    expose:      - 3100    ports:      - 3100:3100    volumes:      - ./botpress/language:/botpress/lang

Pull the image and load the service:

$ docker-compose -f docker-compose-standalone-nlu.yaml pull $ docker-compose -f docker-compose-standalone-nlu.yaml up -d

It's a bit annoying, but you need to be patient. The botpress_lang needs to load the model and can take up to five minutes. So you might need to reload your service for a second-time docker-compose -f docker-compose-standalone-nlu.yaml up -d

‍

In Botpress

In Botpress, you need to modify the nlu.json config file to point on your local NLU

And that's it! Congrats on adding a language to the Botpress NLU!

‍