Spatial Keyboard

The Spatial Keyboard sample provides base prefabs and scripts to add and customize a virtual keyboard. Currently the keyboard is built to work with input fields as the main use case, but can be expanded or customized to support use cases like buttons or a script based workflow.

Note

The TextMesh Pro TMP Essential Resources should be imported to use this sample. Once TextMesh Pro has been installed, you can install the TMP Essential Resources by going to Window > TextMeshPro > Import TMP Essential Resources and importing all files when prompted.

This sample is installed into the default location for package samples, in the Assets\Samples\XR Interaction Toolkit\[version]\Spatial Keyboard folder. You can move these Assets to a different location.

Asset	Description
Affordances\	Contains affordance themes for the keyboard prefabs.
Audio\	Audio files for the keyboard prefabs.
Editor\	Custom editor scripts for the keyboard scripts.
Key Functions\	Asset folder containing scriptable object instances for specific key functionality.
Layouts\	Scriptable object instances defining keyboard layouts and key mappings.
Materials\	Materials used for the keyboard prefabs.
Models\	Models used for the keyboard prefabs.
Prefabs\	Keyboard prefabs and variants.
Scripts\	Contains scripts for the keyboard prefabs.
Sprites\	Contains keyboard icon sprites.
KeyboardDemo	Scene that demonstrates use of the keyboard prefabs. See Demo scene below.

Demo scene

The KeyboardDemo demo scene contains some demonstrations of global keyboard and world space keyboard use cases in conjunction with a few different configurations of the XRKeyboardDisplay and input fields.

Global Keyboard Example

This example demonstrates input fields which are configured to spawn a global keyboard instance in rig space. This includes a standard input field with no limitations, an input field that imposes a character limit on the keyboard, an input field that only updates when the keyboard submits text, an input field that clears text when it opens the keyboard, and one that always observe the global keyboard.

World Space Keyboard Example

This example demonstrates input fields which are configured to listen to the world space keyboard stationed in front of them. There is one input standard input field with no limitations, an input field that imposes a character limit on the keyboard, one that clears the text on keyboard submit, and one that always observes the world keyboard.

Prefabs

Prefab	Description
Input Field Global Keyboard	Prefab of an input field with a XRKeyboardDisplay component that is configured to utilize the global keyboard.
Input Field World Keyboard	Prefab of an input field with a XRKeyboardDisplay component that is configured to utilize a world space keyboard. This prefab requires a connection to the world space keyboard to function properly.
XRI Global Keyboard Manager	GlobalNonNativeKeyboard prefab configured to spawn a XRI Spatial Keyboard as the global keyboard. Requires connection to the XR Rig in the scene in order to spawn the virtual keyboard in rig space.
XRI Keyboard Key	Virtual keyboard key prefab.
XRI Keyboard	Base virtual keyboard prefab.
XRI Spatial Keyboard	Prefab variant with XRGrabInteractable handles to support positioning and scaling of the keyboard.

Scripts

The following scripts are included to support the prefabs and KeyboardDemo.

Script	Description
GlobalNonNativeKeyboard	Singleton which manages spawning, opening, and positioning of the global keyboard.
KeyboardBatchFollow	This component moves a set of transforms to the same local z-axis position as a poke follow transform. This is useful for batchable objects that need to move together.
KeyboardOptimizer	Optimizes the keyboard rendering performance by updating the canvas hierarchy into separate parent transforms based on UI Grouping. This greatly reduces the number of draw calls. Optimization is only done at runtime to prevent breaking the prefab.
KeyFunction	Abstract class defining callbacks for key functionality. Allows users to extend custom functionality of keys and keyboard. Derived classes are defined within the KeyFunctions\ folder.
XRKeyboard	Keyboard implementation of the virtual keyboard.
XRKeyboardConfig	Scriptable object that defines key mappings to support swapping keyboard layouts.
XRKeyboardDisplay	Class used to display keyboard text and help facilitate input field relationship with the virtual keyboard.
XRKeyboardEvents	UnityEvent types and associated event arg classes for events raised by the keyboard components.
XRKeyboardKey	Key base class that sends key commands to the virtual keyboard.
XRKeyboardLayout	Manages the the layouts for the keyboard. Updates each keyboard key that is a child object with the key mapping data from the XRKeyboardConfig.keyMappings.

Global keyboard

The GlobalNonNativeKeyboard should be added to your scene to support a global keyboard instance. The global keyboard can be opened by calling GlobalNonNativeKeyboard.instance.ShowKeyboard and closed by calling GlobalNonNativeKeyboard.instance.HideKeyboard.

Script	Description
Keyboard Prefab	Keyboard prefab for to spawn for the global keyboard instance.
Player Root	XR Origin of the player to spawn the keyboard in rig space. If this is left empty the keyboard will spawn in world space.
Keyboard Offset	Offset from the camera transform where the keyboard is placed upon opening.
Camera Transform	Transform of the camera. If left empty this will default to Camera.main.
Reposition Out Of View Keyboard On Open	If true, the global keyboard will be repositioned to its opening position if out of view when opened.
Facing Keyboard Threshold	Threshold for the dot product used to determine if the keyboard is out of view.

Keyboard

The XRKeyboard is the base implementation of the virtual keyboard. It is used in conjunction with the key implementation of XRKeyboardKey. The keyboard defines the functionality for commands it receives from the keys. Currently, the following commands are supported:

• Character values (updating keyboard text)
• Shift
• Caps Lock
• Backspace
• Delete
• Clear
• Enter or Return
• Layout change
• Hide

By overriding ProcessKeyCode, TryProcessKeyPress, or adding custom KeyFunctions used by XRKeyboardKeys, you can extend the functionality the base keyboard.

Property	Description
Submit on Enter	If true, the OnTextSubmitted event will be invoked when the keyboard receives an enter or return command. Otherwise this enter or return will be process as a newline.
Close on Submit	If true, the keyboard will close on when firing the submit event.
Double Click Interval	Time (in seconds) which this keyboard considers a double click.
Subset Layout	List of keyboard key configurations supported by this keyboard. Each item in the list consists of the string command to toggle the layout, the root XRKeyboardLayout to apply the configuration to, and the configurations themselves. to For more info see Keyboard layout.

Note

Currently the keyboard tracks its own caret position and does not support updating caret position or selecting text from the input field.

Updating subset layouts

The keyboard contains a list of sets of XRKeyboardLayout, XRKeyboardConfig, and string which act as keys to toggle between keyboard layouts.

Property	Description
Layout String	The string which acts as the key for the layout.
Layout Root	The XRKeyboardLayout at the root of the set of keys to update.
Toggle On Config	The XRKeyboardConfig to set as the active key mapping on the Layout Root when the Layout String is received by the keyboard.
Toggle On Config	The XRKeyboardConfig to set as the active key mapping on the Layout Root when the Layout String is received by the keyboard, but the Toggle On Config is already set as the active key mapping. Typically this will be the default key mapping.

In the image above, when a key sends the string \sym to update the layout, the keyboard will set the Toggle On Config, LayoutSymbol, as the active config on the Layout Root if it is not already active. If it is active, it will revert to the Toggle Off Config, LayoutAlphaNumeric.

In the second image above, when a key sends the string \sym to update the layout, two different keyboard layouts will be updated. The GameObject Keyboard Layout will be updated to LayoutSymbols and the GameObject Keyboard Layout Numpad will be updated with LayoutSymbols_Keypad.

These patterns can be reused with any number of layouts as long as the configuration and the layout is registered with the keyboard here.

Keyboard keys

Keyboard keys interface with the keyboard sending strings, commands, and calling different function on the keyboard. Instead of authoring each key individually, XRKeyboardConfig and XRKeyboardLayout help populate the keys with their appropriate data based predefined configurations. See the Keyboard config and Keyboard layout sections for more info.

The keyboard has default implementations for certain commands coming from the keys, but KeyFunctions enable keys to more easily interact with custom keyboard functionality and define functionality for an individual key.

Note

XRKeyboardKey extends Button to utilize OnPointerDown and OnPointerClick

Property	Description
Character	The character that will be processed by the keyboard when this key is pressed. Primarily used to update the text string of the keyboard, but can also be used to trigger keyboard functionality like updating layouts.
Shift Character	The character that will be processed by the keyboard in the shifted state when this key is pressed. Primarily used to update the text string of the keyboard, but can also be used to trigger keyboard functionality like updating layouts.
Display Character	(Optional) The character that will be displayed on the key label. If empty, the key will use the character property as the display character.
Shift Display Character	(Optional) The character that will be displayed on the key label when in the shifted state. If empty, the key will use the Shift Character property as the display character.
Display Icon	(Optional) Icon that will be displayed on the key. If set, this will sprite will be displayed instead of the Display Character. If left empty, Display Character or Character will be used as display text.
Shift Display Icon	(Optional) Icon that will be displayed on the key in the shifted state. If set, this will sprite will be displayed instead of the Shift Display Character. If empty, Shift Display Character or Shift Character will be used as display text.
Key Function	(Optional) Scriptable object instance of KeyFunction that defines functionality of the key. If left blank, the key will TryProcessKeyPress that will attempt to process the Key Code or the Character to determine the functionality of the key.
Key Code	(Optional) Key code that defines the functionality of the key. Used in Key Code Function or as a fallback for the key when the Key Function field is left empty.
Update on Key Down	If true, the key will fire key pressed events on button down. If false, the key will fire key pressed events on button release.
Text	Text field that is used to display the display characters.
Icon	(Optional) Image used to display icons for key.
Highlight	(Optional) Image used to highlight key indicating an active state.
Audio Source	(Optional) Audio source that is fired when the key is pressed.

Key functions

The keyboard keys utilize some KeyFunction scripts which allow users to implement their own key functionality with key specific overrides for keyboard API calls, pre and post processing functionality when the key is pressed, and the ability to override the key display based on the context of the key and the keyboard. The base functions of PreprocessKey and PostprocessKey call into

These KeyFunctions can be updated dynamically based on layout using XRKeyboardConfig (See Keyboard layout and Keyboard config for more info) allowing the ability change a key's functionality based on the current keyboard configuration and layout.

This pattern is optional, but provides flexibility for custom keyboard implementations.

Key Function	Description
Hide Function	Calls the XRKeyboard.Close function to hide the keyboard.
Key Code Function	Passes a KeyCode for the keyboard to process. This KeyFunction expects a valid KeyCode populated on the XRKeyboardKey.
Layout Function	Updates the layout of the keyboard.
Shift Function	Sends shift and caps lock commands to keyboard. Overrides key icon based on shifted, non-shifted, or caps lock state.
Value Function	Updates keyboard with a new text character. Base key function for most keys which update the keyboard text string.

The following KeyFunction base methods can be overridden to provide additional key functionality.

Method	Description
PreprocessKey	Called before the the key press is processed and sent to keyboard.
ProcessKey	Primary function callback when a key is pressed. Use this function to interface directly with a keyboard and process logic based on the current keyboard and key context.
PostprocessKey	Called after the the key press is processed and sent to keyboard.
OverrideDisplayIcon	Determines if the icon should be overridden by the key function. Used to support special icon cases like caps lock.
GetDisplayIcon	Returns an icon for the key. Used in conjunction with OverrideDisplayIcon.
ProcessRefreshDisplay	Called when the key is refreshing it's display (shift change, caps lock, layout change, etc). Allows the key function to process any display logic necessary when refreshing display.

For example if you wanted to implement a custom Alt key, you could do the following:

public class AltFunction : KeyFunction
{
    public override void PreprocessKey(XRKeyboard keyboardContext, XRKeyboardKey key){
        base.PreprocessKey(keyboardContext, key);
        var customKeyboard = keyboardContext as CustomKeyboard;
        /*
        Check state of key and keyboard and perform some other logic before the key is pressed.
        */
    }

    public override void ProcessKey(XRKeyboard keyboardContext, XRKeyboardKey key)
    {
        var customKeyboard = keyboardContext as CustomKeyboard;
        /*
        Check state of key and keyboard and perform some other logic while the key is being pressed.
        */

        // Call into keyboard to process the key
        customKeyboard.ProcessAltKey();
    }

    public override void PostprocessKey(XRKeyboard keyboardContext, XRKeyboardKey key){
        base.PostprocessKey(keyboardContext, key);
        var customKeyboard = keyboardContext as CustomKeyboard;
        /*
        Check state of key and keyboard and perform some other logic after the key is pressed.
        */
    }
}

Layouts and config

The keyboard uses a list of XRKeyboardConfig objects and the configured XRKeyboardConfig.KeyMapping to manage keyboard layouts.

Keyboard layout

A XRKeyboardLayout manages a subset of keys and updates the keys with the data from the active configs XRKeyboardConfig.KeyMapping.

Property	Description
Default Key Mapping	The default configuration when the keyboard resets.
Active Key Mapping	The active configuration for the keyboard layout. When updated, the keyboard layout will automatically attempt to update the subset of keys.

Keyboard can support multiple layouts. For example if there is a standard keyboard and keypad, each set of keys can have a XRKeyboardLayout at the root to manage the layouts. This allows the data of each layout to change individually or together.

Keyboard config

The XRKeyboardConfig is scriptable object which contains the definition for key mappings. It also contains a default KeyFunction which can be applied to most of the keys. Typically this is the Value Key Function which updates the keyboard text.

The XRKeyboardConfig.KeyMappings define the characters, display properties, and the key functionality of the keys for the layout. The number of items in the key mappings list should match the number of keys under a XRKeyboardLayout. Currently, the XRKeyboardLayout simply iterates through key mappings list and assigns them to the keys in sequential order.

Property	Description
Default Key Function	The default key function that will be assigned to the keys in the key mapping. To override the default key function, check KeyMappings.overrideDefaultKeyFunction and set the KeyMappings.keyFunction property on that key mapping item.
Key Mappings	List of key mapping data that will be applied to a set of keys when this config is set as active.

There should be a XRKeyboardConfig instance for each layout. For example, if there is a keyboard that defaults to a set of alpha-numeric keys, and two keys that change the keyboard to a symbols layout and an emoji layout, there should be one config for each layout for a total of three.

Keyboard display

The XRKeyboardDisplay listens to the keyboard and updates accordingly.

When this display gains focus via the input field, it opens the keyboard. When the keyboard is opened with an input field reference, it in turn observes the keyboard to support mouse and keyboard and entry and stay in sync with display options like XRDisplay.ClearTextOnSubmit and XRDisplay.ClearTextOnOpen.

Property	Description
Input Field	The input input field connected to this display that will trigger the opening of the keyboard and be updated to display the keyboard text.
Use Scene Keyboard	If true, this will enable the keyboard field below to allow connection to a keyboard that is placed in the scene.
Keyboard	The keyboard this display is listening to. If left empty, the display will default to the GlobalNonNativeKeyboard keyboard.
Update on Key Press	If true, the display will update the input field each time a key is pressed and the keyboard text is updated. If false, it will update when the keyboard submits text (typically on the enter or return key).
Always Observe Keyboard	If true, this display will never stop listening to the keyboard when the keyboard closes, changes focus, or is opened by a different display field.
Monitor Input Field Character Limit	If true, this display will use the character limit of the input field as the character limit for the keyboard.
Clear Text on Submit	If true, the display will clear after receiving a the text submit event from the keyboard.
Clear Text on Open	If true, this display will clear when it gains focus and opens the keyboard.
Hide Keyboard On Disable	If true, this display will close the keyboard it is observing when this GameObject is disabled. If this display is not observing a keyboard when disabled, this will have not effect on open keyboards.

There are several events this display invokes regarding its state and relationship with the keyboard.

Property	Description
On Text Submitted	The event fired when this keyboard receives a submit text event from the keyboard.
On Keyboard Opened	The event fired when this display opens the keyboard.
On Keyboard Closed	The event fired when the keyboard this display is observing closes.
On Keyboard Focus Changed	The event fired when the keyboard this display is observing changes focus to a different XRKeyboardDisplay.

Known limitations

• Highlighting text and dragging cursor position is not currently supported.