Skip to main content

Flow AI Documentation

As of 9/7/2023 release

Thank you for jumping in and experimenting with us! This first release is a little raw, and things are changing rapidly, but we wanted to share it with you and see if we are on the right track and get your feedback. This is not the full vision of where the app is going, and we’ll keep rolling out functionality over the next weeks and months.

Please provide any and all feedback here! Please don’t hesitate to let us know how this works or could work better for you.

What we have started with are these features

We’ll discuss each of these in more detail below.

  • Get context of any selected dot
  • Build a visual conversation with AI
  • Ask a PDF document for context
  • Manipulate swarm with voice commands

Enabling AI and the AI buttons

First, enable AI for any particular Flow in the Settings, accessed through the menu in the upper right of the AI: 

Once selected, find the “Enable AI features” checkbox.


You may also want to check the “Show AI conversation” in order to see the history of the conversation.

To actually start using AI, you create a new button here, using the “+” icon:


A new button editor will appear:

Click the checkbox to “Enable AI.” This will show the options to control the AI and put the button on the “heads up display” at the bottom of the screen.

There are three options for the Action for response:

  • Ask AI
  • Ask PDF
  • Edit swarm


After you have set up two buttons with labels, and perhaps with additional context for your prompt, here is what it looks like:


Here is a view on the Desktop:

Here is what it looks like in a VR or AR headset:

image.png

GPT prompt:

Using LLM prompts is a whole sub-specialty these days.

The variables flow provides become available with the GPT prompt:
$spokenInput - what did the user just say verbally?
$visiblePopupData - what is the text within the currently visible Popup?

You don’t need to get carried away, but you might find some inspiration from these examples:

Generic example:

$spokenInput| Tell me more about $visiblePopupData| Do not respond with apologies as an AI large language model. Keep the answers to three short concise paragraphs.

Example from a historical biographical Flow::

$visiblePopupData, $spokenInput.
Respond in the voice of the person referenced, including linguistic characteristics of this person's writings.
Don't  apologize about being an AI model, but if there are no extant writings by this person, end your response with a short phrase such as "I've done my best to speak in my voice with limited material."


AI response window

On the desktop, the AI response is shown in a popup window that is scrollable and movable.


Each time a response is provided, the window will open. It can also be opened by clicking on the short response directly above the AI buttons.

In an XR headset, the window appears a bit to the left of the user:



Cancelling an on-going spoken command

Whenever a $spokenInput is present on a visible button, then an accompanying “x” button will appear beside the button to cancel or reset listening. This is great when you misspeak and want to start the query over, or it makes a mistake and recognizes your speech incorrectly.


Edit Swarm

Edit Swarm is used to manipulate the swarm, as shown below where the user asked to filter the data.

The following current prompts are supported, but remember that simple variations on these phrases work as well because it is not an exact match:

  • Filter to a category: “Filter to Vietnam”
  • Filter to a time frame or a numeric range: “Filter from 2016 to 2023”
  • Change swarm color: “Make the color a gradient from green to red”
  • Set swarm column on a specific axis for the scatterplot: “Set the depth axis to population”
  • Toggle axis on and off: “Hide the depth axis”
  • Change dot size: “Make the dots bigger”
  • “Remove filter”

The mechanism currently works with only one filter, so each filter request will replace the prior AI-generated filter with a new one, which removes the ability to complex filters (no AND or OR filtering). In the future, we’ll need a more complex UI visible to the user to help a user indicate when they want to replace or add a filter.

As of this release, there is NOT good error reporting, so it fails silently if it gets confused. It is alos possible for it to create a filter that shows no dots. In this case, as it to “remove filter.”

Ask PDF

Flow has the ability to upload a PDF document (just one for now) and use that as the source material for ChatGPT to query against.

Upload the PDF in the “Manage Data” popup.

GPT model

Currently this release supports OpenAI’s ChatGPT 3.5 and 4.0 models. If using 4.0, you also have the ability to add system context. We can support additional models upon request for enterprise customers.


Conversational AI experiment

We have been experimenting with Conversational AI. 

There is now a checkbox in the Settings in the UI that enables the “conversation” with the AI to be captured and displayed. It actually is displayed as a swarm, off to the right of the origin by 1.5 meters. Currently, this swarm is not very configurable, but we look forward to getting feedback and will be extending the functionality soon.

 

Back to top