mmap {mmap}

Map And Unmap Pages of Memory

Description

Wrapper to POSIX ‘mmap’ and Windows MapViewOfFile system calls.

Usage

mmap(file, mode = int32(), 
     extractFUN=NULL, replaceFUN=NULL,
     prot=mmapFlags("PROT_READ","PROT_WRITE"),
     flags=mmapFlags("MAP_SHARED"),
     len, off=0L, endian=.Platform$endian,
     ...)

munmap(x)

as.mmap(x, mode, file, ...)
is.mmap(x)

extractFUN(x)
replaceFUN(x)

extractFUN(x) <- value
replaceFUN(x) <- value

Arguments

file

name of file holding data to be mapped into memory

mode

mode of data on disk. Use one of ‘char()’ (char <-> R raw), ‘int8()’ (char <-> R integer), ‘uint8()’ (unsigned char <-> R integer), ‘int16()’ (short <-> R integer), ‘uint16()’ (unsigned short <-> R integer), ‘int24()’ (3 byte integer <-> R integer), ‘uint24()’ (unsigned 3 byte integer <-> R integer), ‘int32()’ (R integer), ‘real32()’ (float <-> R double), ‘real64()’ (R double), ‘cplx()’ (R complex), ‘cstring()’ (R variable length character array), ‘struct()’ (Collection of Ctypes as defined by mmap). See the related functions for details.

extractFUN

A function to convert the raw/integer/double values returned by subsetting into a complex R class. If no change is needed, set to NULL (default).

replaceFUN

A function to convert the R classes to underlying C types for storage.

prot

access permission to data being mapped. Set via bitwise OR with mmapFlags to one or more of ‘PROT_READ’: Data can be read, ‘PROT_WRITE’: Data can be written, ‘PROT_EXEC’: Data can be executed, ‘PROT_NONE’: Data cannot be accessed. Not all will apply within the context of R objects. The default is PROT_READ | PROT_WRITE.

flags

additional flags to mmap. Set via bitwise OR with mmapFlags to one or more of ‘MAP_SHARED’: Changes are shared (default), ‘MAP_PRIVATE’: Changes are private, ‘MAP_FIXED’: Interpret addr exactly (Not Applicable). Not all will apply within the context of R objects.

len

length in bytes of mapping from offset. (EXPERT USE ONLY)

off

offset in bytes to start mapping. This must be a multiple of the system pagesize. No checking is currently done, nor is there any mmap provision to find pagesize automatically. (EXPERT USE ONLY)

endian

endianess of data. At present this is only applied to int8,int16, int32,float,real32,double, and real64 types for both atomic and struct types. It is applied universally, and not at struct member elements.

...

unused

x

an object of class ‘mmap’

value

a function to apply upon extraction or replacement.

Details

The general semantics of the R function map to the underlying operating system C function call. On unix-alikes this is ‘mmap’, on Windows similar functionality is provided by the system call ‘MapViewOfFile’. The notable exception is the use of the R argument file in place of void *addr and int fildes. Additionally len and off arguments are made available to the R level call, though require special care based on the system's mmap and are advised for expert use only.

as.mmap allows for in-memory objects to be converted to mmapped version on-disk. The files are stored in the location specified by file. Passing an object that has an appropriate as.mmap method will allow R objects to be automatically created as memory-mapped object. This works for most atomic types in R, including numeric, complex, and character vectors. A special note on character vectors: the implementation supports both variable width character vectors (native R) as well as fixed width arrays requiring a constant number of bytes per element. The current default is to use fixed width, with variable width enabled by setting mode=cstring(). See as.mmap.character for more details.

Complex data types, such as 2 dimesioned vectors (matrix) and data.frames can be supported using appropriate extractFUN and replaceFUN functions to convert the raw data. Basic object conversion is made available in included as.mmap methods for boths types as of version 0.6-3.

All mode types are defined for single-column atomic data, with the exception of structs. Multiple column objects are supported by the use of setting dim. All data is column major. Row major orientation, as well as supporting multiple types in one object - imitating a data.frame, is supported via the struct mode.

Using struct as the mode will organize the binary data on-disk (or more correctly read data organized on disk) in a row-major orientation. This is similar to how a row database would be oriented, and will provide faster access to data that is typically viewed by row. See help(struct) for examples of semantics as well as performance comparisons.

Value

The mmap and as.mmap call returns an object of class mmap containing the fields:

data:

pointer to the ‘mmap’ped file.

bytes:

size of file in bytes. This is not in resident memory.

filedesc:

A names integer file descriptor, where the name is path to the file mapped.

storage.mode:

R type of raw data on disk. See types for details.

pagesize:

operating system pagesize.

extractFUN:

conversion function on extraction (optional).

replaceFUN:

conversion function for replacement (optional).

Author(s)

Jeffrey A. Ryan

References

mmap: http://www.opengroup.org/onlinepubs/000095399/functions/mmap.html

See Also

See Also as mmapFlags,

Examples

# create a binary file and map into 'ints' object
# Note that we are creating a file of 1 byte integers,
# and that the conversion is handled transparently
tmp <- tempfile()
ints <- as.mmap(1:100L, mode=int8(), file=tmp)

ints[1]
ints[]
ints[22]
ints[21:23] <- c(0,0,0)
ints[]  # changes are now on disk

# add dimension
dim(ints) <- c(10,10)
ints[]
ints[6,2] # 6th row of 2nd column
ints[ ,2] # entire 2nd column
munmap(ints)

# store Dates as natural-size 'int' on disk
writeBin(as.integer(Sys.Date()+1:10), tmp)
DATE <- mmap(tmp,extractFUN=function(x) structure(x,class="Date"))
DATE[]
munmap(DATE)

# store 2 decimal numeric as 'int' on disk, and convert on extraction
num <- mmap(tmp,extractFUN=function(x) x/100)
num[]
munmap(num)

unlink(tmp)

# convert via as.mmap munmap
int <- as.mmap(1:10L)
num <- as.mmap(rnorm(10))