struct_type {sparklyr.nested}

Spark Data Types

Description

These function support supplying a spark read schema. This is particularly useful when reading data with nested arrays when you are not interested in several of the nested fields.

Usage

struct_type(sc, struct_fields)

struct_field(sc, name, data_type, nullable = FALSE)

array_type(sc, data_type, nullable = FALSE)

binary_type(sc)

boolean_type(sc)

byte_type(sc)

date_type(sc)

double_type(sc)

float_type(sc)

integer_type(sc)

numeric_type(sc)

long_type(sc)

map_type(sc, key_type, value_type, nullable = FALSE)

string_type(sc)

character_type(sc)

timestamp_type(sc)

Arguments

sc

A spark_connection

struct_fields

A vector or fields obtained from struct_field()

name

A field name to use in the output struct type

data_type

A (java) data type (e.g., string_type() or double_type())

nullable

Logical. Describes whether field can be missing for some rows.

key_type

A (java) data type describing the map keys (usually string_type())

value_type

A (java) data type describing the map values