R: Read Sparse Matrix from Text File

read.sparse {readsparse}

R Documentation

Read Sparse Matrix from Text File

Description

Read a labelled sparse CSR matrix in text format as used by libraries such as SVMLight, LibSVM, ThunderSVM, LibFM, xLearn, XGBoost, LightGBM, and more.

The format is as follows:

<label(s)> <column>:<value> <column>:<value> ...

with one line per observation/row.

Example line (row):

1 1:1.234 3:20

This line denotes a row with label (target variable) equal to 1, a value for the first column of 1.234, a value of zero for the second column (which is missing), and a value of 20 for the third column.

The labels might be decimal (for regression), and each row might contain more than one label (must be integers in this case), separated by commas without spaces inbetween - e.g.:

1,5,10 1:1.234 3:20

This line indicates a row with labels 1, 5, and 10 (for multi-class classification). If the line has no labels, it should still include a space before the features.

The rows might additionally contain a 'qid' parameter as used in ranking algorithms, which should always lay inbetween the labels and the features and must be an integer - e.g.:

1 qid:2 1:1.234 3:20

The file might optionally contain a header as the first line with metadata (number of rows, number of columns, number of classes). Presence of a header will be automatically detected, and is recommended to include it for speed purposes. Datasets from the extreme classification repository (see references) usually include such a header.

Lines might include comments, which start after a '#' character. Lines consisting of only a '#' will be ignored. When reading from a file, such file might have a BOM (information about encoding uses in Windows sytems), which will be automatically skipped.

Usage

read.sparse(
  file,
  multilabel = FALSE,
  has_qid = FALSE,
  integer_labels = FALSE,
  index1 = TRUE,
  sort_indices = TRUE,
  ignore_zeros = TRUE,
  min_cols = 0L,
  min_classes = 0L,
  limit_nrows = 0L,
  use_altrep = TRUE,
  from_string = FALSE
)

Arguments

`file`	Either a file path from which the data will be read, or a string ('character' variable) containing the text from which the data will be read. In the latter case, must pass 'from_string=TRUE'.
`multilabel`	Whether the input file can have multiple labels per observation. If passing 'multilabel=FALSE' and it turns out to have multiple labels, will only take the first one for each row. If the labels are non-integers or have decimal point, the results will be invalid.
`has_qid`	Whether the input file has 'qid' field (used for ranking). If passing 'FALSE' and the file does turns out to have 'qid', the features will not be read for any observations.
`integer_labels`	Whether to output the observation labels as integers.
`index1`	Whether the input file uses numeration starting at 1 for the column numbers (and for the label numbers when passing 'multilabel=TRUE'). This is usually the case for files downloaded from the repositories in the references. The function will check for whether any of the column indices is zero, and will ignore this option if so (i.e. will assume it is 'FALSE').
`sort_indices`	Whether to sort the indices of the columns after reading the data. These should already be sorted in the files from the repositories in the references.
`ignore_zeros`	Whether to avoid adding features which have a value of zero. If the zeros are caused due to numerical rounding in the software that wrote the input file, they can be post-processed by passing 'ignore_zeros=FALSE' and then something like 'X@x[X@x == 0] = 1e-8'.
`min_cols`	Minimum number of columns that the output 'X' object should have, in case some columns are all missing in the input data.
`min_classes`	Minimum number of columns that the output 'y' object should have, in case some columns are all missing in the input data. Only used when passing 'multilabel=TRUE'.
`limit_nrows`	Maximum number of rows to read from the data. If there are more than this number of rows, it will only read the first 'limit_nrows' rows. If passing zero (the default), there will be no row limit.
`use_altrep`	Whether to use R's ALTREP system to return C++ vector objects without generating extra data copies. If passing 'FALSE', each piece of data will be copied into a an R-allocated vector and returned as such. Passing 'TRUE' is faster and uses less memory (as there are no redundant data copies), but these ALTREP'd objects can potentially result in some functions/methods running slower on them than on R objects (for example, manually sub-setting the vectors in the S4 Matrix classes that are returned can potentially be slower by some microseconds).
`from_string`	Whether to read the data from a string variable instead of a file. If passing 'from_string=TRUE', then 'file' is assumed to be a variable with the data contents on it.

Details

Note that this function:

Will not make any checks for negative column indices.
Has a precision of C type 'int' for column indices and integer labels (the maximum value that this type can hold can be checked in '.Machine$integer.max').
Will fill missing labels with NAs when passing 'multilabel=FALSE'.
Will fill with zeros (empty values) the lines that are empty (that is, they generate a row in the data), but will ignore (that is, will not generate a row in the data) the lines that start with '#'.

Be aware that the data is represented as a CSR matrix with index pointer of class C 'int', thus the number of rows/columns/non-zero-elements cannot exceed '.Machine$integer.max'.

On 64-bit Windows systems, if compiling the library with a compiler other than MinGW or MSVC, it will not be able to read files larger than 2GB. This should not be a concern if installing it from CRAN or from R itself, as the Windows version at the time of writing can only be compiled with MinGW.

If the file contains a header, and this header denotes a larger number of columns or of labels than the largest index in the data, the resulting object will have this dimension set according to the header. The third entry in the header (number of classes/labels) will be ignored when passing 'multilabel=FALSE'.

The function uses different code paths when reading from a file or from a string, and there might be slight differences between the obtained results from them. For example, reading from a file might produce the desired output if the file uses tabs as separators instead of spaces (not supported by most other software and not standard), whereas reading from a string will not. If any such difference is encountered, please submit a bug report in the package's GitHub page.

Value

A list with the following entries:

'X': the features, as a CSR Matrix from package 'Matrix' (class 'dgRMatrix').
'y': the labels. If passing 'multilabel=FALSE' (the default), will be a vector (class 'numeric' when passing 'integer_labels=FALSE', class 'integer' when passing 'integer_labels=TRUE'), otherwise will be a binary CSR Matrix (class 'ngRMatrix').
'qid': the query IDs used for ranking, as an integer vector. This entry will only be present when passing 'has_qid=TRUE'.

These can be easily transformed to other sparse matrix types through e.g. 'X <- as(X, "CsparseMatrix")'.

References

Datasets in this format can be found here:

LibSVM Data: https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/
Extreme Classification Repository: http://manikvarma.org/downloads/XC/XMLRepository.html

The format is also described at the SVMLight webpage: http://svmlight.joachims.org.

Examples

library(Matrix)
library(readsparse)

### Example input file
"1 2:1.21 5:2.05
-1 1:0.45 3:0.001 4:-10" -> coded.matrix

r <- read.sparse(coded.matrix, from_string=TRUE)
print(r)

### Convert it back to text
recoded.matrix <- write.sparse(file=NULL, X=r$X, y=r$y, to_string=TRUE)
cat(recoded.matrix)

### Example with real file I/O
## generate a random sparse matrix and labels
set.seed(1)
X <- rsparsematrix(nrow=5, ncol=10, nnz=8)
y <- rnorm(5)

## save into a text file
temp_file <- file.path(tempdir(), "matrix.txt")
write.sparse(temp_file, X, y, integer_labels=FALSE)

## inspect the text file
cat(paste(readLines(temp_file), collapse="\n"))

## read it back
r <- read.sparse(temp_file)
print(r)

### (Note that columns with all-zeros are discarded,
###  this behavior can be avoided with 'add_header=TRUE')

[Package readsparse version 0.1.5-6 Index]