storr_rds {storr}

rds object cache driver

Description

Object cache driver that saves objects using R's native serialized file format (see saveRDS) on the filesystem.

Usage

storr_rds(path, compress = NULL, mangle_key = NULL,
  mangle_key_pad = NULL, hash_algorithm = NULL,
  default_namespace = "objects")

driver_rds(path, compress = NULL, mangle_key = NULL,
  mangle_key_pad = NULL, hash_algorithm = NULL)

Arguments

path

Path for the store. tempdir() is a good choice for ephemeral storage, The rappdirs package (on CRAN) might be nice for persistent application data.

compress

Compress the generated file? This saves a small amount of space for a reasonable amount of time.

mangle_key

Mangle keys? If TRUE, then the key is encoded using base64 before saving to the filesystem. See Details.

mangle_key_pad

Logical indicating if the filenames created when using mangle_key should also be "padded" with the = character to make up a round number of bytes. Padding is required to satisfy the document that describes base64 encoding (RFC 4648) but can cause problems in some applications (see this issue. The default is to not pad new storr archives. This should be generally safe to leave alone.

hash_algorithm

Name of the hash algorithm to use. Possible values are "md5", "sha1", and others supported by digest. If not given, then we will default to "md5".

default_namespace

Default namespace (see storr).

Details

The mangle_key argument will run each key that is created through a "base 64" encoding. This means that keys that include symbols that are invalid on filesystems (e.g, "/", ":") will be replaced by harmless characters. The RFC 4648 dialect is used where "-" and "_" are used for character 62 and 63 (this differs from most R base64 encoders). This mangling is designed to be transparent to the user – the storr will appear to store things with unmangled keys but the names of the stored files will be different.

Note that the namespace is not mangled (at least not yet) so needs to contain characters that are valid in a filename.

Because the actual file will be stored with mangled names it is not safe to use the same path for a storr with and without mangling. So once an rds storr has been created its "mangledness" is set. Using mangle_key = NULL uses whatever mangledness exists (or no mangledness if creating a new storr).

Corrupt keys

Some file synchronisation utilities like dropbox can create file that confuse an rds storr (e.g., "myobject (Someone's conflicted copy)". If mangle_key is FALSE these cannot be detected but at the same time are not a real problem for storr. However, if mangle_key is TRUE and keys are base64 encoded then these conflicted copies can break parts of storr.

If you see a warning asking you to deal with these files, please delete the offending files; the path will be printed along with the files that are causing the problem.

Alternatively, you can try (assuming a storr object st) running

st$driver$purge_corrupt_keys()

which will delete corrupted keys with no confirmation. The messages that are printed to screen will be printed by default at most once per minute per namespace. You can control this by setting the R option storr.corrupt.notice.period - setting this to NA suppresses the notice and otherwise it is interpreted as the number of seconds.

Examples

# Create an rds storr in R's temporary directory:
st <- storr_rds(tempfile())

# Store some data (10 random numbers against the key "foo")
st$set("foo", runif(10))
st$list()

# And retrieve the data:
st$get("foo")

# Keys that are not valid filenames will cause issues.  This will
# cause an error:
## Not run: 
st$set("foo/bar", letters)

## End(Not run)

# The solution to this is to "mangle" the key names.  Storr can do
# this for you:
st2 <- storr_rds(tempfile(), mangle_key = TRUE)
st2$set("foo/bar", letters)
st2$list()
st2$get("foo/bar")

# Behind the scenes, storr is safely encoding the filenames with base64:
dir(file.path(st2$driver$path, "keys", "objects"))

# Clean up the two storrs:
st$destroy()
st2$destroy()