cooccurrence {udpipe}

Create a cooccurence data.frame

Description

A cooccurence data.frame indicates how many times each term co-occurs with another term.

There are 3 types of cooccurrences:

• Looking at which words are located in the same document/sentence/paragraph.

• Looking at which words are followed by another word

• Looking at which words are in the neighbourhood of the word as in follows the word within skipgram number of words

The output of the function gives a cooccurrence data.frame which contains the fields term1, term2 and cooc where cooc indicates how many times term1 and term2 co-occurred. This dataset can be constructed

• based upon a data frame where you look within a group (column of the data.frame) if 2 terms occurred in that group.

• based upon a vector of words in which case we look how many times each word is followed by another word.

• based upon a vector of words in which case we look how many times each word is followed by another word or is followed by another word if we skip a number of words in between.

Note that

• For cooccurrence.data.frame no ordering is assumed which implies that the function does not return self-occurrences if a word occurs several times in the same group of text and term1 is always smaller than term2 in the output

• For cooccurrence.character we assume text is ordered from left to right, the function as well returns self-occurrences

You can also aggregate cooccurrences if you decide to do any of these 3 by a certain group and next want to obtain an overall aggregate.

Usage

cooccurrence(x, order = TRUE, ...)

## S3 method for class 'character'
cooccurrence(
  x,
  order = TRUE,
  ...,
  relevant = rep(TRUE, length(x)),
  skipgram = 0
)

## S3 method for class 'cooccurrence'
cooccurrence(x, order = TRUE, ...)

## S3 method for class 'data.frame'
cooccurrence(x, order = TRUE, ..., group, term)

Arguments

x

either

• a data.frame where the data.frame contains 1 row per document/term, in which case you need to provide group and term where term is the column containing 1 term per row and group indicates something like a document id or document + sentence id. This uses cooccurrence.data.frame.

• a character vector with terms where one element contains 1 term. This uses cooccurrence.character.

• an object of class cooccurrence. This uses cooccurrence.cooccurrence.

order

logical indicating if we need to sort the output from high cooccurrences to low coccurrences. Defaults to TRUE.

...

other arguments passed on to the methods

relevant

a logical vector of the same length as x, indicating if the word in x is relevant or not. This can be used to exclude stopwords from the cooccurrence calculation or selecting only nouns and adjectives to find cooccurrences along with each other (for example based on the Parts of Speech upos output from udpipe_annotate).
Only used if calculating cooccurrences on x which is a character vector of words.

skipgram

integer of length 1, indicating how far in the neighbourhood to look for words.
skipgram is considered the maximum skip distance between words to calculate co-occurrences (where co-occurrences are of type skipgram-bigram, where a skipgram-bigram are 2 words which occur at a distance of at most skipgram + 1 from each other).
Only used if calculating cooccurrences on x which is a character vector of words.

group

character vector of columns in the data frame x indicating to calculate cooccurrences within these columns.
This is typically a field like document id or a sentence identifier. To be used if x is a data.frame.

term

character string of a column in the data frame x, containing 1 term per row. To be used if x is a data.frame.

Value

a data.frame with columns term1, term2 and cooc indicating for the combination of term1 and term2 how many times this combination occurred

Methods (by class)

• character: Create a cooccurence data.frame based on a vector of terms

• cooccurrence: Aggregate co-occurrence statistics by summing the cooc by term/term2

• data.frame: Create a cooccurence data.frame based on a data.frame where you look within a document / sentence / paragraph / group if terms co-occur

Examples

data(brussels_reviews_anno)

## By document, which lemma's co-occur
x <- subset(brussels_reviews_anno, xpos %in% c("NN", "JJ") & language %in% "fr")
x <- cooccurrence(x, group = "doc_id", term = "lemma")
head(x)

## Which words follow each other
x <- c("A", "B", "A", "A", "B", "c")
cooccurrence(x)

data(brussels_reviews_anno)
x <- subset(brussels_reviews_anno, language == "es")
x <- cooccurrence(x$lemma)
head(x)
x <- subset(brussels_reviews_anno, language == "es")
x <- cooccurrence(x$lemma, relevant = x$xpos %in% c("NN", "JJ"), skipgram = 4)
head(x)

## Which nouns follow each other in the same document
library(data.table)
x <- as.data.table(brussels_reviews_anno)
x <- subset(x, language == "nl" & xpos %in% c("NN"))
x <- x[, cooccurrence(lemma, order = FALSE), by = list(doc_id)]
head(x)

x_nodoc <- cooccurrence(x)
x_nodoc <- subset(x_nodoc, term1 != "appartement" & term2 != "appartement")
head(x_nodoc)