{BayesSummaryStatLM}R Documentation

Read in Tabulated Data and Compute Summary Statistics


This function reads in tabulated data sets and produces summary statistics needed for Bayesian linear regression models for use in the function bayes.regress() (in this package). Big data sets that are too large to fit into R memory are handled using functions from package ff. The function takes as input data files with the predictor variables X and response values Y, and returns the summary statistics X'X, X'Y and Y'Y that are used as an input to the function bayes.regress() (in this package) for Bayesian linear regression models. The function supports reading data sets that are split across multiple files.

                   , fileEncoding = "", nrows = -1, first.rows = 1e5, next.rows = 1e5
                   , levels = NULL, appendLevels = TRUE,FUN = "read.table",transFUN = NULL
                   , asffdf_args = list(), BATCHBYTES = getOption("ffbatchbytes")
                   , VERBOSE = FALSE, header = FALSE, sep = ",", quote = "\"'", dec = "."
                   , numerals = c("allow.loss", "warn.loss", "no.loss")
                   , na.strings = "NA", colClasses = "numeric", skip = 0
                   , check.names = TRUE, fill = TRUE, strip.white = FALSE
                   , blank.lines.skip = TRUE, comment.char = "#", allowEscapes = FALSE
                   , flush = FALSE, skipNul = FALSE)



the name of a file or a list of file names, from which the data will be read. By default it is assumed that the file(s) contain data in comma separated format; this can be changed using the sep argument. File names must be in the format acceptable by standard functions such as read.table. Default = NULL.


a vector of integers that specifies the columns to treat as predictor variables, to create the design matrix X. By default, all columns after the first are treated as columns with predictor variables. Default = NA.


an integer that specifies the column that contains the response variable values, to create the response vector Y. By default, the first column that is not used as a predictor variable column is selected as the response values column. Default = NA.


The name of the R object containing previously-calculated summary statistics (if applicable), to be updated with new data. This must be a list similar in structure to the returned value, containing entries xtx, xty, yty and of appropriate dimensions and class that match the data contained in filename. Default = NULL.

The remaining arguments are passed directly to the function read.table.ffdf(). Below is a short description of the arguments we recommend to set manually in accordance with memory limitations and data structure:


the number of rows to read in the first chunk of data. Default = 100,000.


the number of rows to read in the remaining chunks of data. Default = 100,000.


the character that separates the columns of data. Default = ",".

For the arguments below the default settings should perform well. However, in some situations adjusting these arguments may improve memory use and running time.


a string that describes the file's character encoding


an integer specifying how many rows should be read from the file


an optional list of items with col.names or factor columns. See read.table.ffdf().


a logical vector of permission to expand levels


specifies which standard R function is used to read the data.


an optional filtering function to be applied to each chunk of data. See read.table.ffdf().


an optional list of parameters to be passed to as.ffdf() function.


an integer limiting the size of the data.frame used to store each chunk of the data


See read.table.ffdf().


a logical value indicating if the first row is the header row


a character string specifying which character will be treated as quoting characters


a character used for decimal dot


see read.table().


strings treated as NA values


a vector that describes the data types in each column. Numeric by default.


how many first lines in the file should be skipped


see read.table().


logical value that turns on automatic padding of the rows in case they have different lengths


affects the processing of the columns with declared character type. See scan().


logical value, indicating whether empty lines should be ignored


character specifying the comment marker


logical. If TRUE, the escaped strings will be parsed. See scan().


see read.table().


logical: should nuls be skipped?


The function reads in data and computes summary statistics to be used in Bayesian linear regression by the function bayes.regress() (in this package). The function assumes the linear regression model will have a non-zero y-intercept; this option can be changed in the bayes.regress() function (see bayes.regress() help for details).


The returned value for the function is a list containing the summary statistics named xtx (for X'X), xty (for X'Y), yty (for Y'Y) and the total number of data values The summary statistic xtx contains a square matrix obtained by computing a dot product of the predictor variables data X with itself; a leading column of 1's is added to X for the y-intercept term. xty contains the vector obtained by computing the dot product of the transposed predictor variables data X with response variable data Y; a leading column of 1's is added to X for the y-intercept term. yty contains the dot product of the response variable data Y with itself. is the number of data values read from the data file(s); this number may be smaller than the number of rows in the data file, since some of the rows with missing data may be skipped according to specified function arguments. The summary statistics X'X, X'Y and Y'Y are summed over data chunks by the following, for m = 1,...,M chunks:

X'X = sum_{m=1}^{M} (X'_{m})( X_{m})

X'Y = sum_{m=1}^{M} (X'_{m})( Y_{m})

Y'Y = sum_{m=1}^{M} (Y'_{m})( Y_{m})

The returned values are used as input to the function bayes.regress() (in this package). Note that the matrix X is given a leading column of 1's by default, for the y-intercept term of the Bayesian linear regression model. This can be removed by specifying a model with zero intercept in the function bayes.regress() (see bayes.regress() help for details).


Carlin, B.P. and Louis, T.A. (2009) Bayesian Methods for Data Analysis, 3rd ed., Boca Raton, FL: Chapman and Hall/CRC Press.

Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A. and Rubin, D.B. (2013) Bayesian Data Analysis, 3rd ed., Boca Raton, FL: Chapman and Hall/CRC Press.

Adler, D., Glaser, C., Nenadic, O., Oehlschlagel, J. and Zucchini, W. (2013) ff: memory-efficient storage of large data on disk and fast access functions. R package:


# The package includes several example data files, illustrated here.

# Example
# The following command finds the location of the data file
# that includes 4 predictor variables and 20,000 simulated data values.

filename <- system.file('data/', package='BayesSummaryStatLM')

# The file is formatted so that the simulated response variable is in the
# first column, and columns 2 to 5 contain simulated predictor variables.
# The simulated coefficients are: beta <- c(0.76, -0.92, 0.64, 0.57, -1.65), 
# where the first value is the y-intercept term in the Bayesian linear 
# regression model.  The sigma-squared term, i.e. the variance of the normally 
# distributed error terms, is simulated as: sigmasq <- 0.25

## Next, read the data and compute the summary statistics using the
# "" function.  By default, the first column is assumed 
# to be the response variable, and the remaining columns are assumed to contain
# predictor variable values.  The function will check if the file exists and 
# can be read.

data.values <-

[Package BayesSummaryStatLM version 2.0 Index]