layer_string_lookup {keras3} | R Documentation |
A preprocessing layer that maps strings to (possibly encoded) indices.
Description
This layer translates a set of arbitrary strings into integer output via a
table-based vocabulary lookup. This layer will perform no splitting or
transformation of input strings. For a layer than can split and tokenize
natural language, see the layer_text_vectorization
layer.
The vocabulary for the layer must be either supplied on construction or
learned via adapt()
. During adapt()
, the layer will analyze a data set,
determine the frequency of individual strings tokens, and create a
vocabulary from them. If the vocabulary is capped in size, the most frequent
tokens will be used to create the vocabulary and all others will be treated
as out-of-vocabulary (OOV).
There are two possible output modes for the layer.
When output_mode
is "int"
,
input strings are converted to their index in the vocabulary (an integer).
When output_mode
is "multi_hot"
, "count"
, or "tf_idf"
, input strings
are encoded into an array where each dimension corresponds to an element in
the vocabulary.
The vocabulary can optionally contain a mask token as well as an OOV token
(which can optionally occupy multiple indices in the vocabulary, as set
by num_oov_indices
).
The position of these tokens in the vocabulary is fixed. When output_mode
is "int"
, the vocabulary will begin with the mask token (if set), followed
by OOV indices, followed by the rest of the vocabulary. When output_mode
is "multi_hot"
, "count"
, or "tf_idf"
the vocabulary will begin with
OOV indices and instances of the mask token will be dropped.
Note: This layer uses TensorFlow internally. It cannot be used as part of the compiled computation graph of a model with any backend other than TensorFlow. It can however be used with any backend when running eagerly. It can also always be used as part of an input preprocessing pipeline with any backend (outside the model itself), which is how we recommend to use this layer.
Note: This layer is safe to use inside a tf.data
pipeline
(independently of which backend you're using).
Usage
layer_string_lookup(
object,
max_tokens = NULL,
num_oov_indices = 1L,
mask_token = NULL,
oov_token = "[UNK]",
vocabulary = NULL,
idf_weights = NULL,
invert = FALSE,
output_mode = "int",
pad_to_max_tokens = FALSE,
sparse = FALSE,
encoding = "utf-8",
name = NULL,
...,
vocabulary_dtype = NULL
)
Arguments
object |
Object to compose the layer with. A tensor, array, or sequential model. |
max_tokens |
Maximum size of the vocabulary for this layer. This should
only be specified when adapting the vocabulary or when setting
|
num_oov_indices |
The number of out-of-vocabulary tokens to use.
If this value is more than 1, OOV inputs are modulated to
determine their OOV value.
If this value is 0, OOV inputs will cause an error when calling
the layer. Defaults to |
mask_token |
A token that represents masked inputs. When |
oov_token |
Only used when |
vocabulary |
Optional. Either an array of integers or a string path to a
text file. If passing an array, can pass a list, list,
1D NumPy array, or 1D tensor containing the integer vocbulary terms.
If passing a file path, the file should contain one line per term
in the vocabulary. If this argument is set,
there is no need to |
idf_weights |
Only valid when |
invert |
Only valid when |
output_mode |
Specification for the output of the layer. Values can be
|
pad_to_max_tokens |
Only applicable when |
sparse |
Boolean. Only applicable to |
encoding |
Optional. The text encoding to use to interpret the input
strings. Defaults to |
name |
String, name for the object |
... |
For forward/backward compatability. |
vocabulary_dtype |
The dtype of the vocabulary terms, for example
|
Value
The return value depends on the value provided for the first argument.
If object
is:
a
keras_model_sequential()
, then the layer is added to the sequential model (which is modified in place). To enable piping, the sequential model is also returned, invisibly.a
keras_input()
, then the output tensor from callinglayer(input)
is returned.-
NULL
or missing, then aLayer
instance is returned.
Examples
Creating a lookup layer with a known vocabulary
This example creates a lookup layer with a pre-existing vocabulary.
vocab <- c("a", "b", "c", "d") data <- rbind(c("a", "c", "d"), c("d", "z", "b")) layer <- layer_string_lookup(vocabulary=vocab) layer(data)
## tf.Tensor( ## [[1 3 4] ## [4 0 2]], shape=(2, 3), dtype=int64)
Creating a lookup layer with an adapted vocabulary
This example creates a lookup layer and generates the vocabulary by analyzing the dataset.
data <- rbind(c("a", "c", "d"), c("d", "z", "b")) layer <- layer_string_lookup() layer %>% adapt(data) get_vocabulary(layer)
## [1] "[UNK]" "d" "z" "c" "b" "a"
Note that the OOV token "[UNK]"
has been added to the vocabulary.
The remaining tokens are sorted by frequency
("d"
, which has 2 occurrences, is first) then by inverse sort order.
data <- rbind(c("a", "c", "d"), c("d", "z", "b")) layer <- layer_string_lookup() layer %>% adapt(data) layer(data)
## tf.Tensor( ## [[5 3 1] ## [1 2 4]], shape=(2, 3), dtype=int64)
Lookups with multiple OOV indices
This example demonstrates how to use a lookup layer with multiple OOV indices. When a layer is created with more than one OOV index, any OOV values are hashed into the number of OOV buckets, distributing OOV values in a deterministic fashion across the set.
vocab <- c("a", "b", "c", "d") data <- rbind(c("a", "c", "d"), c("m", "z", "b")) layer <- layer_string_lookup(vocabulary = vocab, num_oov_indices = 2) layer(data)
## tf.Tensor( ## [[2 4 5] ## [0 1 3]], shape=(2, 3), dtype=int64)
Note that the output for OOV value 'm' is 0, while the output for OOV value
"z"
is 1. The in-vocab terms have their output index increased by 1 from
earlier examples (a maps to 2, etc) in order to make space for the extra OOV
value.
One-hot output
Configure the layer with output_mode='one_hot'
. Note that the first
num_oov_indices
dimensions in the ont_hot encoding represent OOV values.
vocab <- c("a", "b", "c", "d") data <- c("a", "b", "c", "d", "z") layer <- layer_string_lookup(vocabulary = vocab, output_mode = 'one_hot') layer(data)
## tf.Tensor( ## [[0 1 0 0 0] ## [0 0 1 0 0] ## [0 0 0 1 0] ## [0 0 0 0 1] ## [1 0 0 0 0]], shape=(5, 5), dtype=int64)
Multi-hot output
Configure the layer with output_mode='multi_hot'
. Note that the first
num_oov_indices
dimensions in the multi_hot encoding represent OOV values.
vocab <- c("a", "b", "c", "d") data <- rbind(c("a", "c", "d", "d"), c("d", "z", "b", "z")) layer <- layer_string_lookup(vocabulary = vocab, output_mode = 'multi_hot') layer(data)
## tf.Tensor( ## [[0 1 0 1 1] ## [1 0 1 0 1]], shape=(2, 5), dtype=int64)
Token count output
Configure the layer with output_mode='count'
. As with multi_hot output,
the first num_oov_indices
dimensions in the output represent OOV values.
vocab <- c("a", "b", "c", "d") data <- rbind(c("a", "c", "d", "d"), c("d", "z", "b", "z")) layer <- layer_string_lookup(vocabulary = vocab, output_mode = 'count') layer(data)
## tf.Tensor( ## [[0 1 0 1 2] ## [2 0 1 0 1]], shape=(2, 5), dtype=int64)
TF-IDF output
Configure the layer with output_mode="tf_idf"
. As with multi_hot output,
the first num_oov_indices
dimensions in the output represent OOV values.
Each token bin will output token_count * idf_weight
, where the idf weights
are the inverse document frequency weights per token. These should be
provided along with the vocabulary. Note that the idf_weight
for OOV
values will default to the average of all idf weights passed in.
vocab <- c("a", "b", "c", "d") idf_weights <- c(0.25, 0.75, 0.6, 0.4) data <- rbind(c("a", "c", "d", "d"), c("d", "z", "b", "z")) layer <- layer_string_lookup(output_mode = "tf_idf") layer %>% set_vocabulary(vocab, idf_weights=idf_weights) layer(data)
## tf.Tensor( ## [[0. 0.25 0. 0.6 0.8 ] ## [1. 0. 0.75 0. 0.4 ]], shape=(2, 5), dtype=float32)
To specify the idf weights for oov values, you will need to pass the entire vocabulary including the leading oov token.
vocab <- c("[UNK]", "a", "b", "c", "d") idf_weights <- c(0.9, 0.25, 0.75, 0.6, 0.4) data <- rbind(c("a", "c", "d", "d"), c("d", "z", "b", "z")) layer <- layer_string_lookup(output_mode = "tf_idf") layer %>% set_vocabulary(vocab, idf_weights=idf_weights) layer(data)
## tf.Tensor( ## [[0. 0.25 0. 0.6 0.8 ] ## [1.8 0. 0.75 0. 0.4 ]], shape=(2, 5), dtype=float32)
When adapting the layer in "tf_idf"
mode, each input sample will be
considered a document, and IDF weight per token will be calculated as
log(1 + num_documents / (1 + token_document_count))
.
Inverse lookup
This example demonstrates how to map indices to strings using this layer.
(You can also use adapt()
with inverse=TRUE
, but for simplicity we'll
pass the vocab in this example.)
vocab <- c("a", "b", "c", "d") data <- rbind(c(1, 3, 4), c(4, 0, 2)) layer <- layer_string_lookup(vocabulary = vocab, invert = TRUE) layer(data)
## tf.Tensor( ## [[b'a' b'c' b'd'] ## [b'd' b'[UNK]' b'b']], shape=(2, 3), dtype=string)
Note that the first index correspond to the oov token by default.
Forward and inverse lookup pairs
This example demonstrates how to use the vocabulary of a standard lookup layer to create an inverse lookup layer.
vocab <- c("a", "b", "c", "d") data <- rbind(c("a", "c", "d"), c("d", "z", "b")) layer <- layer_string_lookup(vocabulary = vocab) i_layer <- layer_string_lookup(vocabulary = vocab, invert = TRUE) int_data <- layer(data) i_layer(int_data)
## tf.Tensor( ## [[b'a' b'c' b'd'] ## [b'd' b'[UNK]' b'b']], shape=(2, 3), dtype=string)
In this example, the input value "z"
resulted in an output of "[UNK]"
,
since 1000 was not in the vocabulary - it got represented as an OOV, and all
OOV values are returned as "[UNK]"
in the inverse layer. Also, note that
for the inverse to work, you must have already set the forward layer
vocabulary either directly or via adapt()
before calling
get_vocabulary()
.
See Also
Other categorical features preprocessing layers:
layer_category_encoding()
layer_hashed_crossing()
layer_hashing()
layer_integer_lookup()
Other preprocessing layers:
layer_category_encoding()
layer_center_crop()
layer_discretization()
layer_feature_space()
layer_hashed_crossing()
layer_hashing()
layer_integer_lookup()
layer_mel_spectrogram()
layer_normalization()
layer_random_brightness()
layer_random_contrast()
layer_random_crop()
layer_random_flip()
layer_random_rotation()
layer_random_translation()
layer_random_zoom()
layer_rescaling()
layer_resizing()
layer_text_vectorization()
Other layers:
Layer()
layer_activation()
layer_activation_elu()
layer_activation_leaky_relu()
layer_activation_parametric_relu()
layer_activation_relu()
layer_activation_softmax()
layer_activity_regularization()
layer_add()
layer_additive_attention()
layer_alpha_dropout()
layer_attention()
layer_average()
layer_average_pooling_1d()
layer_average_pooling_2d()
layer_average_pooling_3d()
layer_batch_normalization()
layer_bidirectional()
layer_category_encoding()
layer_center_crop()
layer_concatenate()
layer_conv_1d()
layer_conv_1d_transpose()
layer_conv_2d()
layer_conv_2d_transpose()
layer_conv_3d()
layer_conv_3d_transpose()
layer_conv_lstm_1d()
layer_conv_lstm_2d()
layer_conv_lstm_3d()
layer_cropping_1d()
layer_cropping_2d()
layer_cropping_3d()
layer_dense()
layer_depthwise_conv_1d()
layer_depthwise_conv_2d()
layer_discretization()
layer_dot()
layer_dropout()
layer_einsum_dense()
layer_embedding()
layer_feature_space()
layer_flatten()
layer_flax_module_wrapper()
layer_gaussian_dropout()
layer_gaussian_noise()
layer_global_average_pooling_1d()
layer_global_average_pooling_2d()
layer_global_average_pooling_3d()
layer_global_max_pooling_1d()
layer_global_max_pooling_2d()
layer_global_max_pooling_3d()
layer_group_normalization()
layer_group_query_attention()
layer_gru()
layer_hashed_crossing()
layer_hashing()
layer_identity()
layer_integer_lookup()
layer_jax_model_wrapper()
layer_lambda()
layer_layer_normalization()
layer_lstm()
layer_masking()
layer_max_pooling_1d()
layer_max_pooling_2d()
layer_max_pooling_3d()
layer_maximum()
layer_mel_spectrogram()
layer_minimum()
layer_multi_head_attention()
layer_multiply()
layer_normalization()
layer_permute()
layer_random_brightness()
layer_random_contrast()
layer_random_crop()
layer_random_flip()
layer_random_rotation()
layer_random_translation()
layer_random_zoom()
layer_repeat_vector()
layer_rescaling()
layer_reshape()
layer_resizing()
layer_rnn()
layer_separable_conv_1d()
layer_separable_conv_2d()
layer_simple_rnn()
layer_spatial_dropout_1d()
layer_spatial_dropout_2d()
layer_spatial_dropout_3d()
layer_spectral_normalization()
layer_subtract()
layer_text_vectorization()
layer_tfsm()
layer_time_distributed()
layer_torch_module_wrapper()
layer_unit_normalization()
layer_upsampling_1d()
layer_upsampling_2d()
layer_upsampling_3d()
layer_zero_padding_1d()
layer_zero_padding_2d()
layer_zero_padding_3d()
rnn_cell_gru()
rnn_cell_lstm()
rnn_cell_simple()
rnn_cells_stack()