Skip to content

Tokenized Text Stimulus

A tokenized-text stimulus is similar to the text stimulus, with additional options for manipulating the stimulus on a token-by-token basis (in most cases, a token is a word). A tokenized-text stimulus can be particularly useful for creating a self-paced reading study.

Required Properties

content

  • Definition: The text to be displayed (for example, in a typical self-paced reading study, this should be a single sentence)
  • Possible values: A string of text (strings are any text enclosed in quotation marks " "). If you'd like to add formatting to your string, the following HTML tags are supported: <abbr>, <acronym>, <b>, <blockquote>, <code>, <em>, <i>, <li>, <ol>, <strong>, <ul>, <p>.
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed."
}

type

  • Definition: The type of stimulus (text, tokenized_text, image, etc.)
  • Possible values: For a tokenized text stimulus, type must be specified as tokenized_text.
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed."
}

Optional Properties

alignment

  • Definition: Where the text will appear on the screen
  • Possible values: "left", "right" and "center"
  • Default: "left"
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "alignment": "right"
}

backward_mask

Dependencies

Requires mode to be set to "masked"

  • Definition: Whether tokens preceeding the active token are displayed as the mask character. If set to true, then tokens preceeding the active token are displayed as the mask character; if set to false, then tokens preceeding the active token are displayed normally.
  • Possible values: true or false
  • Default: true
  • See Also: forward_mask, masked_spaces, and mask_char
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "mode": "masked",
  "backward_mask": false
}

bidirectional

Dependencies

Requires mode to be set to "masked"

  • Definition: Allows a self-paced tokenized text stimulus to go in both forward and backward directions. If set to true, participants can press one key to advance the tokens and another key to reverse. Reading/reaction times are recorded for each forward and backward action.
  • Possible values: true or false
  • Default: false (i.e., forward self-paced only)
  • See also: self_paced, key_advance, and key_reverse
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "self_paced": true,
  "bidirectional": true
}

backward_mask

Dependencies

Requires mode to be set to "masked"

  • Definition: Whether tokens preceeding the active token are displayed as the mask character. If set to true, then tokens preceeding the active token are displayed as the mask character; if set to false, then tokens preceeding the active token are displayed normally.
  • Possible values: true or false
  • Default: true
  • See Also: forward_mask, masked_spaces, and mask_char
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "mode": "masked",
  "backward_mask": false
}

color

  • Definition: The color of the stimulus
  • Possible values: Common color names such as "red", "darkgreen", or hex codes like "#FF6600"
  • Default: "black"
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "color": "#FF6600"
}

delay

Info

This property is used for presenting multiple stimuli on a trial in a sequence. To implement inter-trial intervals, see the delay parameter in trial templates instead.

Warning

The delaying of multiple stimuli on a trial does NOT have a chain effect. That is, the delay of any stimulus on a trial is always relative to the onset of that trial or the end of the last stimulus acting as a barrier.

  • Definition: Delay the onset of a stimulus by a certain number of seconds, relative to (1) the onset of a trial if there are no stimuli acting as barriers or (2) the end of the last stimulus acting as a barrier.
  • Possible values: A number in seconds (decimals are supported)
  • Default: 0 (no delay)

{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed",
  "delay": 1.6
}
The above code introduces a delay of 1600 milliseconds.

delimiter

  • Definition: A character or a string that serves as the boundary for splitting content into a list of tokens
  • Possible values: Any character or regex string
  • Default: A single space character (i.e., " ")

Note

If the delimiter should be interpreted as a regular expression (instead of a string), regex must be set to true.

{
  "type": "tokenized_text",
  "content": "apple,orange,lemon,lime,banana",
  "delimiter": ","
}
Because delimiter is set to "," (a comma), "apple", "orange", "lemon", "lime", and "banana" will be displayed as separate tokens.

forward_mask

Dependencies

Requires mode to be set to "masked"

  • Definition: If set to true, tokens following the active token will appear in the mask character; if set to false, then tokens following the active token are invisible.
  • Possible values: true or false
  • Default: false
  • See Also: backward_mask, masked_spaces, and mask_char
{
  "type": "tokenized_text",
  "content": "The tokens following the active token will be masked.",
  "mode": "masked",
  "forward_mask": true
}

hint

Dependencies

Requires self_paced to be set to true.

  • Definition: Whether or not to display an instruction below the tokenized text stimulus on which key to press to advance in self-paced reading. The instruction will read: "Press the "[SPACE]" key to advance." If you have set key_advance to a different key, then [SPACE] will be replaced by the name of that key. In the case of bidirectional self-paced reading, both keys will be listed.
  • Possible values: true or false
  • Default: true
  • See also: self_paced
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "self_paced": true,
  "key_advance": "j",
  "hint": false
}

In this example, because hint* is set to false, no instruction will be displayed with this stimulus. If hint** were set to true, participants would see the following instruction: "Press the "j" key to advance."

keep_delimiter

Info

In most cases, this setting is only useful when you are using a regex-based delimiter.

  • Definition: If set to true, then the tokens are displayed with the delimiter character between tokens; if false, then the tokens are displayed with a space between tokens.
  • Possible values: true or false
  • Default: false
  • See also: regex, delimiter

{
  "type": "tokenized_text",
  "content": "apple,orange,lemon,lime,banana",
  "delimiter": ",",
  "keep_delimiter": true
}
Because delimiter is set to "," (a comma), "apple", "orange", "lemon", "lime", and "banana" will be displayed as separate tokens. Because keep_delimiter is set to true, the comma will appear when the tokens are displayed: "apple,orange,lemon,lime,banana"

If keep_delimiter were set to false, the tokens would be separated by a "[SPACE]": "apple orange lemon lime banana"

keep_last

  • Definition: Whether to keep the last token on display after the stimulus completes.
  • Possible values: true or false
  • Default: false
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "mode": "masked",
  "keep_last": true
}

key_advance

Dependencies

Requires self_paced to be set to true.

Warning

Avoid setting the key to any Shift keys unless your participants are in a controlled lab environment. Some operating systems (e.g., Windows) associate system actions with repeated presses of Shift keys, which may interfere with experiment progress.

  • Definition: The key that a participant should press to advance the presentation of a self-paced tokenized text stimulus
  • Possible values: Any alphanumeric character, and one of the following keys: "ArrowUp", "ArrowDown", "ArrowLeft", "ArrowRight", "ShiftLeft", "ShiftRight", "ControlLeft", "ControlRight", "AltLeft", "AltRight"
  • Default: When bidirectional is false, the default is " " (the space bar); when bidirectional is true, the default is "ArrowRight" (the right arrow key).
  • See also: self_paced, key_reverse

In this example, we change the key for advancing in a self-paced reading stimulus to the letter "J". 

{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "self_paced": true,
  "key_advance": "j" // change the key to the letter J key
}

key_advance_instruction

DEPRECATED

This feature will be removed in a future grammar update. Use hint instead.

  • Definition: Whether or not to display an instruction below the tokenized text stimulus on which key to press to advance. The instruction will read: "Press the "[SPACE]" key to advance." If you have set key_advance to a different key, then "[SPACE]" will be replaced by that key.
  • Possible values: true or false
  • Default: true

{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "self_paced": true,
  "key_advance": "j",
  "key_advance_instruction": false
}
Because key_advance_instruction is set to false, no instruction will be displayed with this stimulus. If key_advance_instruction were set to true, participants would see the following instruction: "Press the "j" key to advance."

key_reverse

Dependencies

Requires self_paced to be set to true. Requires bidirectional to be set to true.

Warning

Avoid setting the key to any Shift keys unless your participants are in a controlled lab environment. Some operating systems (e.g., Windows) associate system actions with repeated presses of Shift keys, which may interfere with experiment progress.

  • Definition: The key that a participant should press to reverse the presentation of a self-paced tokenized text stimulus
  • Possible values: Any alphanumeric character, and one of the following keys: "ArrowUp", "ArrowDown", "ArrowLeft", "ArrowRight", "ShiftLeft", "ShiftRight", "ControlLeft", "ControlRight", "AltLeft", "AltRight"
  • Default: "ArrowLeft" (i.e., the left arrow key)
  • See also: self_paced, bidirectional, key_advance
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "self_paced": true,
  "bidirectional": true,
  "key_advance": "AltRight" // change the key advancing the tokens to the right Alt key
  "key_reverse": "AltLeft" // change the key reversing the tokens to the left Alt key
}

mask_char

Dependencies

Requires mode to be set to "masked"

  • Definition: A character used to mask inactive tokens if the mode is "masked". For example, if the mask character is "#", when the token "TOKEN" is no longer active, it will be displayed as #####.
  • Possible values: Any character
  • Default: "#"
  • See also: mode

In this example, we change the default mask character from # to *. 

{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "mode": "masked",
  "mask_char": "*"
}

masked_spaces

Dependencies

Requires mode to be set to "masked"

  • Definition: If set to true, all spaces in a token will be displayed in the mask character; if false, all spaces in a token will be displayed as a space, revealing word boundaries and lengths.
  • Possible values: true or false
  • Default: true
  • See also: mode, mask_char
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be masked.",
  "mode": "masked",
  "masked_spaces": false
}

When the masked_spaces property is set to false, the content "The text in these quotation marks will be masked" will be masked as follows:

 ### #### ## ##### ######### ##### #### ## #######

When the masked_spaces property is set to true (its default value), it will be masked as this instead:

 #################################################

That is, the spaces are masked as well, making it next to impossible to infer word boundaries.

mode

  • Definition: The mode in which the tokenized text stimulus is displayed. Three options are possible: "plain", where tokens are displayed in sequence, "masked", where only one active token is displayed and other tokens explicitly masked, and "singleton", where only one active token is displayed with all other tokens invisible.
  • Possible values: "plain", "masked", and "singleton"
  • Default: "plain"
  • See also: mask_char, forward_mask, backward_mask, masked_spaces
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "mode": "masked"
}

parent

  • Definition: If specified, then the properties of the specified stimulus will be inherited by this stimulus, unless a property of the same name is defined in this stimulus
  • Possible values: The name of another defined stimulus

Consider a tokenized stimulus named "t_t_1":

"{
  "type": "tokenized_text",
  "content": "This text will be masked",
  "key_advance": "j",
  "mode": "masked",
  "barrier": false
}

and then define a second tokenized stimulus named "t_t_2":

{
  "type": "tokenized_text",
  "content": "This text will not be masked",
  "parent": "t_t_1",
  "mode": "plain"
}
Here, the t_t_2 stimulus inherits the barrier and key_advance properties of t_t_1. t_t_2 does not inherit the mode property of t_t_1 because the stimulus definition of t_t_2 specifies its own mode property.

regex

  • Definition: If set to true, the delimiter is interpreted as a regular expression instead of a string.
  • Possible values: true or false
  • Default: false
  • See also: delimiter

{
  "type": "tokenized_text",
  "content": "We want to display each word AND each half of half-baked separately!",
  "self_paced": true,
  "mode": "plain",
  "delimiter": "[\\s-]",
  "regex": true,
  "keep_delimiter": true
}
In this example code, the delimiter [\\s-] is set to be a regex pattern, and it will recognize either spaces or dashes as boundaries between tokens. In the content "We want to display each word AND each half of half-baked seperately", every space will mark token boundaries, such that "We", "want", "to", etc. are displayed as distinct tokens. Additionally, dashes ("-") will mark boundaries, so "half" and "baked" are recognized as separate tokens.

self_paced

  • Definition: Whether participant interaction (in the form of a key-press) must trigger the presentation of each token. If true, then the display of tokens in a tokenized text stimulus must be triggered by participant interaction, and reaction time for each token is recorded; if false, then each tokenized text stimulus will be automatically presented.
  • Possible values: true, false
  • Default: false
  • See also: bidirectional, key_advance, key_reverse, hint, speed, and token_duration
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "self_paced": true
}

size

  • Definition: The font size of the tokenized text stimulus
  • Possible values: Either 1) HTML font sizes such as "px", "em", or "rem", or 2) a percentage of the default size.
{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "size": "50px"
}

speed

Dependencies

Requires self_paced to be set to false (or omitted). Overriden by token_duration if it is also set.

  • Definition: The speed, in characters per second, at which a tokenized text stimulus is presented
  • Possible values: A number of characters per second (i.e., 6) to be displayed
  • See Also: self_paced, token_duration

{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "self_paced": false,
  "speed": 5
}
In this sample code, 5 characters will be displayed every second.

token_duration

Dependencies

Requires self_paced to be set to false (or omitted). Overrides speed when both are set.

  • Definition: The speed, in seconds per token, at which a tokenized text stimulus is presented
  • Possible values: A number of seconds per token (i.e. 2) to be displayed
  • See Also: self_paced, speed

{
  "type": "tokenized_text",
  "content": "The text in these quotation marks will be displayed.",
  "self_paced": false,
  "token_duration": 2
}
In this sample code, a new token will be displayed every 2 seconds.

Relevant Readings

Design Pattern: Respond at the Right Time

See Also

Text Stimulus, Tokenized Audio