Introduction to Typing

The Moonsense SDK provides a seamless mechanism to observe typing events when integrated. The overall goal of capturing typing events with the SDK is to model the reality of users interacting with input fields within the application. With this in mind, the SDK provides the following events to represent typing behaviors:

  • TextChange
  • KeyPress
  • FocusChange
  • Pointer

Note that while it is useful to observe all these events to deduce typing behavior, each of these events individually provide a lot of context into patterns exhibited by users of the application.

This document provides context around the information contained within each of these events. It also describes the nuances of how these events are populated per platform (i.e. iOS, Android & Web).

TextChange

This event is triggered every time a text field within the application observes a "text change". Using the TextChange event we can determine how the input field is being interacted with within a given Session. When a Moonsense recording Session is in progress, the SDK collects all the relevant input fields or TargetElements to observe for text changes. In the event of the text change a TextChange event is fired with the TargetElement returned as a part of the target field. The unique id associated with the TargetElement can help users distinguish between two TextChange events that happen on separate input fields. The type of the TargetElement(when available) provides additional context around the values permitted by the input field.

Using the determined_at field returned by TextChange, a user can keep track of how the text changes over time. The text captured using the TargetElement can be found in masked_text. Note that the text returned by the event is masked for privacy reasons - more on that later.

The user can also keep track of the state of a TargetElement using the focus field. This field helps determine if the TargetElement is in focus when a TextChange event happens. For cases where the text is populated programmatically, the focus field returns a false with a valid TargetElement. We have a separate event called FocusChange that tracks how TargetElements gain and lose focus(see below).

Finally, the action field included in the event can be used to determine if multiple characters appearing or disappearing from the masked_text field are due to a cut or paste action. Note that iOS does not provide the cut or paste action due to platform limitations.

KeyPress

These events capture the action of a key being pressed and released - signified by the KEY_DOWN and KEY_UP type events. It is also useful to observe the exact timing of these events using the determined_at field. If the key events are being delivered to a specific TargetElement then that element is returned using the target field.

In order to determine the character returned by the KeyPress event we can keep track of the masked_key value that returns a unicode character. This field again is masked for privacy reasons. The special_key on the other hand is unmasked and returns key presses like Home, Alt, Shift and so on. The other fields like alt_key, control_key, meta_key and shift_key can be used in combination with the other fields to determine combinations.

Note that while this information is useful this event is not always returned by all platforms. iOS and Android provide for the KeyPress events in instances where an external keyboard has been attached to the device. Android may return some KeyPress events associated with the soft keyboard for prominent actions like Enter and Line Feed. Web will in most situations return the KeyPress events when delivered using the keyboard.

FocusChange

We can keep track of TargetElements in focus by following the events of type FocusChange. The SDK sends out an event of type FOCUS_GAINED when a TargetElement comes into focus and FOCUS_LOST when it goes out of focus. While this event is relatively straightforward, it is important to note that a FocusChange event is also delivered if a screen has been initialized with a TargetElement in focus. In this case we can determine if the subsequent FOCUS_LOST can be attributed to the TargetElement that was initialized with focus.

Pointer

The final event in the group of events that cover typing is the Pointer event. While the Pointer event is primarily used for touch actions associated with the SDK. We have added an additional field called is_software_keyboard for all touch events captured with the soft keyboard. With this flag a user can track touch patterns used by the user on the keyboard presented within the application. Note that due to platform restrictions this event is only available on iOS.

TargetElement

An important part of the TextChange and KeyPress event is the TargetElement or input field that is being operated on. The SDK performs a best effort in terms of furnishing the details contained within the TargetElement field based on the information available. Given a Session we can be sure that the target_id associated with a TargetElement does not change and remains constant. While this id is unique it is not uniformly represented across platforms. Below are some recommendations to improve the reliability of the TargetElement if the current values do not meet your needs.

Android

On Android, we determine the target_id using the id associated with the View. If an id is not available, we traverse up the parent tree to determine a View that does have and id and use it to construct the target_id. If the entire path to the root View does not contain an id we default to using the id android_view to construct the target_di. The SDK also adds a unique int suffix to every TargetElement (android_view_0 for eg.). This guarantees that every TargetElement is unique in case two Views share the same id. In this case, the suffix associated to each view can help differentiate one from another.

In order to determine the target_type the SDK uses the inputType associated with the input field. If one is not available it returns "text" or simply "null". For situations where the target_type is too generic, it is recommended you ask the app developer to include an inputType when creating a View. This helps ensure that the associated target_type is constructed correctly.

iOS

On iOS, we determine the target_id based on the class name of the UI Component that is the target of the TextChange, KeyPress or FocusChange event. For instance, the first UITextField being edited will have the target_id of UITextField_0.

When determining the target_type of a UIComponent that conforms to the UITextInput protocol, the iOS SDK will inspect the UITextInputTraits.textContentType property of the target and will report the value if present. The iOS SDK will also inspect other properties provided by the UITextInput protocol and will report any that are not set to their default values. UITextInput properties that are inspected are:

If any of these properties are set to non-default values they will be included in the target_type as a | delimited list.

Web

On Web, an order of precedence is used to determine the target_id of an Element. A custom attribute, target-element-id, is supported to explicitly set the target_id of inputs for use with the Moonsense Web SDK and is the primary value used for the target_id. If target-element-id is not set on the element, the id attribute of the element will be used. If id is not set, the name attribute will be used. If none of these are set, a GUID will be generated for use as the target_id for the bundle.

The target_type is taken from the type attribute set on the element.

Masking

In order to preserve the privacy of the content of the user data we have proceeded to mask the sensitive information contained within the input fields. Each character(s) contained within the TargetElement has been replaced by another character in the UTC space based on a specified algorithm. The algorithm has been implemented in a per TargetElement basis, therefore if two TargetElements where to contain the same text, the final masked text would be different. Note that only the field prefixed with masked_ have been masked.