hipread_long_yield {hipread}

Read a hierarchical fixed width data file, in yields

Description

Enhances hipread_long() or hipread_list() to allow you to read hierarchical data in pieces (called 'yields') and allow your code to have full control between reading pieces, allowing for more freedom than the 'callback' method introduced in the chunk functions (like hipread_long_chunked()).

Usage

hipread_long_yield(
  file,
  var_info,
  rt_info = hip_rt(1, 0),
  compression = NULL,
  skip = 0,
  encoding = "UTF-8"
)

hipread_list_yield(
  file,
  var_info,
  rt_info = hip_rt(1, 0),
  compression = NULL,
  skip = 0,
  encoding = "UTF-8"
)

Arguments

file

A filename

var_info

Variable information, specified by either hip_fwf_positions() or hip_fwf_widths(). For hierarchical data files, there should be a named list, where the name is the value indicated by the record type variable and there is one variable information per record type.

rt_info

A record type information object, created by hip_rt(), which contains information about the location of the record type variable that defines the record type for each observation. The default contains width 0, which indicates that there the data is rectangular and does not have a record type variable.

compression

If NULL, guesses the compression from the file extension (if extension is "gz" uses gzip, otherwise treats as plain text), can specify it with a string ("txt" indicates plain text and "gz" for gzip).

skip

Number of lines to skip at the start of the data (defaults to 0).

encoding

(Defaults to UTF-8) A string indicating what encoding to use when reading the data, but like readr, the data will always be converted to UTF-8 once it is imported. Note that UTF-16 and UTF-32 are not supported for non-character columns.

Details

These functions return a HipYield R6 object which have the following methods:

• yield(n = 10000) A function to read the next 'yield' from the data, returns a tbl_df (or list of tbl_df for hipread_list_yield()) with up to n rows (it will return NULL if no rows are left, or all available ones if less than n are available).

• reset() A function to reset the data so that the next yield will read data from the start.

• is_done() A function that returns whether the file has been completely read yet or not.

• cur_pos A property that contains the next row number that will be read (1-indexed).

Value

A HipYield R6 object (See 'Details' for more information)

Methods

Public methods

• HipYield$new()

• HipYield$yield()

• HipYield$reset()

• HipYield$is_done()

Method new()

Usage

HipYield$new(
  file,
  var_info,
  rt_info = hip_rt(0, 1),
  compression = NULL,
  skip = 0,
  encoding = NULL
)

Method yield()

Usage

HipYield$yield(n = 10000)

Method reset()

Usage

HipYield$reset()

Method is_done()

Usage

HipYield$is_done()

Super class

hipread::HipYield -> HipLongYield

Methods

Public methods

• HipLongYield$new()

• HipLongYield$yield()

Inherited methods

• hipread::HipYield$is_done()
• hipread::HipYield$reset()

Method new()

Usage

HipLongYield$new(
  file,
  var_info,
  rt_info = hip_rt(0, 1),
  compression = NULL,
  skip = 0,
  encoding = NULL
)

Method yield()

Usage

HipLongYield$yield(n = 10000)

Super class

hipread::HipYield -> HipListYield

Methods

Public methods

• HipListYield$new()

• HipListYield$yield()

Inherited methods

• hipread::HipYield$is_done()
• hipread::HipYield$reset()

Method new()

Usage

HipListYield$new(
  file,
  var_info,
  rt_info = hip_rt(0, 1),
  compression = NULL,
  skip = 0,
  encoding = NULL
)

Method yield()

Usage

HipListYield$yield(n = 10000)

Examples

library(hipread)
data <- hipread_long_yield(
  hipread_example("test-basic.dat"),
  list(
    H = hip_fwf_positions(
      c(1, 2, 5, 8),
      c(1, 4, 7, 10),
      c("rt", "hhnum", "hh_char", "hh_dbl"),
      c("c", "i", "c", "d")
    ),
    P = hip_fwf_widths(
      c(1, 3, 1, 3, 1),
      c("rt", "hhnum",  "pernum", "per_dbl", "per_mix"),
      c("c", "i", "i", "d", "c")
    )
  ),
  hip_rt(1, 1)
)
# Read the first 4 rows
data$yield(4)

# Read the next 2 rows
data$yield(2)

# Reset and then read the first 4 rows again
data$reset()
data$yield(4)