📘

Note

If you are looking to simply understand a response in the emulator, you can use the inspect button to get information about the Autonomous nodes thinking. But for other use cases, Botpress allows you to create a javascript hook that allows you do something with the data programatically.

Using LLMz hooks

Sometimes you want to do something with that information in your production bots, perhaps for analytics, perhaps for figuring out issues. LLMz hooks allow you to do that. You can use them to monitor and analyze responses from LLMz-powered Autonomous Nodes, especially when they retrieve information from the Knowledge Base (KB), and you can send the information to a Botpress Table, or to your preferred analytics tool.

You can follow the Botpress guide for setting up streaming analytics for messages, but if you'd like to get information on LLMz logic, you can follow these steps to adjust for this.

Step 1: Setting Up Your Analytics Tool

Start by configuring your analytics tool to capture data from your Botpress bot. Refer to your tool’s documentation to obtain any necessary API keys or project identifiers.

Likewise, if you'd like to use Botpress Analytics, or Botpress Tables, you can also do so.

Step 2: Configuring the "After LLMz Execution" Hook

The After LLMz Execution hook can be set up to collect data whenever an Autonomous Node retrieves and responds with information from the Knowledge Base. This hook is triggered each time the node generates a response, enabling you to capture KB interactions.

To set up this hook:

1. Navigate to the Hooks tab in Botpress Studio.
2. Click Create Hook and select the After LLMz Execution hook.

This hook function provides access to two key parameters:

• event: The Event object with details about the interaction.
• execution: An object containing data on the LLMz execution, including details of each iteration and response content. More information on the execution object structure is provided below.

Example Code for the After LLMz Execution Hook

The code below demonstrates how to capture analytics data when an Autonomous Node references information from the Knowledge Base.

// Only execute if Knowledge Base requests were made
if (event?.kb?.results?.length) {
  const botAnalyticsData = {
    // Timestamp of the event
    time: event.createdOn.getTime(),
    
    // Unique identifiers
    $insert_id: event.id,          // Prevents duplicate events
    distinct_id: event.userId,     // Identifies unique users
    bot_id: event.botId,
    conversation_id: event.conversationId,
    
    // Knowledge Base interaction data
    query: event.preview,          // User's query to the Knowledge Base
    result: event?.kb?.results[0].content,  // Matching KB content fragment
    fileName: event?.kb?.results[0].title,  // Source document name
    
    // Bot response data - formatted message sent to user
    autonomousNodeResponse: execution.iterations[execution.iterations.length - 1].message,
    
    // Context information
    current_flow: event.state.context.currentFlow,
    current_node: event.state.context.currentNode,
    current_card: event.state.context.currentCard
  }

  // Replace with your analytics tool's API endpoint
  const projectToken = '<your_analytics_token>'
  await axios.post(
    '<analytics_tool_endpoint>',
    [{
      event: 'Outbound Message',
      properties: botAnalyticsData,
    }],
    {
      headers: {
        Authorization: `Basic ${btoa(projectToken + ':')}`,
      },
    }
  )
}

The execution Object

The execution object provides detailed information about the actions taken by the Autonomous Node, specifically capturing each iteration and its outcome. This includes messages sent, code executed, signals received, and more. Here’s an in-depth look at its structure and contents:

Structure of execution Object

• status: String indicating the overall status of the execution, e.g., "success", "partial", "failure".

• iterations: Array of objects, each representing a step or interaction in the Autonomous Node’s execution. Each iteration contains:

  • message: The text response generated at this stage.
  • code: Code executed during this iteration, often in response to specific queries.
  • variables: Object holding variables used within the iteration.
  • status: Status of the individual iteration, e.g., "partial", "success".
  • signal: Information on any signals received, often in the form of structured ThinkSignal or other signals, detailing contextual reasoning.
  • started_ts: Timestamp (in milliseconds) indicating when the iteration started.
  • ended_ts: Timestamp (in milliseconds) when the iteration concluded.
  • llm: Object detailing the large language model used, including its version and parameters.
  • mutations: Array of any mutations or transformations applied to variables.
  • traces: Trace data related to the LLM's decisions and actions.
  • messages: Messages used or processed within this step.
  • notebook: Object representing any generated or stored notebook content for reference.

• context: Contains settings, variables, and tools relevant to the execution:

  • __options: Execution settings such as the model version, number of loops, temperature, and specific instructions.
  • transcript: Transcript of messages exchanged, with each entry detailing user and bot interactions.
  • getMessages: Function for retrieving messages, typically used internally for processing.
  • partialExecutionMessages: Array holding messages generated if the execution is paused or partially completed.
  • tool_names: Names of tools used in the execution, e.g., APIs or functions.
  • iteration: Current iteration count in the execution loop.
  • injectedVariables: Variables introduced during execution, such as results from knowledge base queries.

• location: Provides information about the Autonomous Node’s current position in the workflow:

  • type: Indicates if it's a workflow or standalone operation.
  • workflowId: Unique identifier for the workflow.
  • workflowName: Name of the workflow where the execution takes place.
  • nodeId: Identifier for the specific node within the workflow.
  • nodeName: Name of the node, e.g., "AutonomousNode".