stream_watermark {sparklyr}

Watermark Stream

Description

Ensures a stream has a watermark defined, which is required for some operations over streams.

Usage

stream_watermark(x, column = "timestamp", threshold = "10 minutes")

Arguments

x

An object coercable to a Spark Streaming DataFrame.

column

The name of the column that contains the event time of the row, if the column is missing, a column with the current time will be added.

threshold

The minimum delay to wait to data to arrive late, defaults to ten minutes.