S3SparkResourceConnector {s3.resourcer}

Apache Spark DBI resource connector for S3

Description

Apache Spark DBI resource connector for S3

Apache Spark DBI resource connector for S3

Format

A R6 object of class SparkResourceConnector

Details

Makes a Apache Spark connection object, that is also a DBI connection object, from a S3 resource description.

Super classes

resourcer::DBIResourceConnector -> resourcer::SparkResourceConnector -> S3SparkResourceConnector

Methods

Public methods

• S3SparkResourceConnector$new()

• S3SparkResourceConnector$isFor()

• S3SparkResourceConnector$createDBIConnection()

• S3SparkResourceConnector$getTableName()

• S3SparkResourceConnector$readDBTable()

• S3SparkResourceConnector$readDBTibble()

• S3SparkResourceConnector$closeDBIConnection()

• S3SparkResourceConnector$clone()

Method new()

Create a SparkResourceConnector instance.

Usage

S3SparkResourceConnector$new()

Returns

A SparkResourceConnector object.

Method isFor()

Check if the provided resource applies to a Apache Spark server. The resource URL scheme must be one of "s3+spark", "s3+spark+http" or "s3+spark+https".

Usage

S3SparkResourceConnector$isFor(resource)

Arguments

resource

The resource object to validate.

Returns

A logical.

Method createDBIConnection()

Creates a DBI connection object from a Apache Spark resource.

Usage

S3SparkResourceConnector$createDBIConnection(resource)

Arguments

resource

A valid resource object.

Returns

A DBI connection object.

Method getTableName()

Get the SQL table name from the resource URL.

Usage

S3SparkResourceConnector$getTableName(resource)

Arguments

resource

A valid resource object.

Returns

The SQL table name.

Method readDBTable()

Read a table as a vanilla tibble using DBI connection object.

Usage

S3SparkResourceConnector$readDBTable(conn, resource)

Arguments

conn

A DBI connection object.

resource

A valid resource object.

Method readDBTibble()

Read a table as a SQL tibble using DBI connection object.

Usage

S3SparkResourceConnector$readDBTibble(conn, resource)

Arguments

conn

A DBI connection object.

resource

A valid resource object.

Method closeDBIConnection()

Close the DBI connection to Apache Spark.

Usage

S3SparkResourceConnector$closeDBIConnection(conn)

Arguments

conn

A DBI connection object.

Method clone()

The objects of this class are cloneable with this method.

Usage

S3SparkResourceConnector$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.