acero {arrow} | R Documentation |
Functions available in Arrow dplyr queries
Description
The arrow
package contains methods for 37 dplyr
table functions, many of
which are "verbs" that do transformations to one or more tables.
The package also has mappings of 212 R functions to the corresponding
functions in the Arrow compute library. These allow you to write code inside
of dplyr
methods that call R functions, including many in packages like
stringr
and lubridate
, and they will get translated to Arrow and run
on the Arrow query engine (Acero). This document lists all of the mapped
functions.
dplyr
verbs
Most verb functions return an arrow_dplyr_query
object, similar in spirit
to a dbplyr::tbl_lazy
. This means that the verbs do not eagerly evaluate
the query on the data. To run the query, call either compute()
,
which returns an arrow
Table, or collect()
, which pulls the resulting
Table into an R tibble
.
-
anti_join()
: thecopy
andna_matches
arguments are ignored -
distinct()
:.keep_all = TRUE
not supported -
full_join()
: thecopy
andna_matches
arguments are ignored -
inner_join()
: thecopy
andna_matches
arguments are ignored -
left_join()
: thecopy
andna_matches
arguments are ignored -
mutate()
: window functions (e.g. things that require aggregation within groups) not currently supported -
pull()
: thename
argument is not supported; returns an R vector by default but this behavior is deprecated and will return an Arrow ChunkedArray in a future release. Provideas_vector = TRUE/FALSE
to control this behavior, or setoptions(arrow.pull_as_vector)
globally. -
right_join()
: thecopy
andna_matches
arguments are ignored -
semi_join()
: thecopy
andna_matches
arguments are ignored -
slice_head()
: slicing within groups not supported; Arrow datasets do not have row order, so head is non-deterministic;prop
only supported on queries wherenrow()
is knowable without evaluating -
slice_max()
: slicing within groups not supported;with_ties = TRUE
(dplyr default) is not supported;prop
only supported on queries wherenrow()
is knowable without evaluating -
slice_min()
: slicing within groups not supported;with_ties = TRUE
(dplyr default) is not supported;prop
only supported on queries wherenrow()
is knowable without evaluating -
slice_sample()
: slicing within groups not supported;replace = TRUE
and theweight_by
argument not supported;n
only supported on queries wherenrow()
is knowable without evaluating -
slice_tail()
: slicing within groups not supported; Arrow datasets do not have row order, so tail is non-deterministic;prop
only supported on queries wherenrow()
is knowable without evaluating -
summarise()
: window functions not currently supported; arguments.drop = FALSE
and '.groups = "rowwise" not supported
Function mappings
In the list below, any differences in behavior or support between Acero and the R function are listed. If no notes follow the function name, then you can assume that the function works in Acero just as it does in R.
Functions can be called either as pkg::fun()
or just fun()
, i.e. both
str_sub()
and stringr::str_sub()
work.
In addition to these functions, you can call any of Arrow's 262 compute
functions directly. Arrow has many functions that don't map to an existing R
function. In other cases where there is an R function mapping, you can still
call the Arrow function directly if you don't want the adaptations that the R
mapping has that make Acero behave like R. These functions are listed in the
C++ documentation, and
in the function registry in R, they are named with an arrow_
prefix, such
as arrow_ascii_is_decimal
.
arrow
base
-
as.Date()
: MultipletryFormats
not supported in Arrow. Consider using the lubridate specialised parsing functionsymd()
,ymd()
, etc. -
as.difftime()
: only supportsunits = "secs"
(the default) -
data.frame()
:row.names
andcheck.rows
arguments not supported;stringsAsFactors
must beFALSE
-
difftime()
: only supportsunits = "secs"
(the default);tz
argument not supported -
nchar()
:allowNA = TRUE
andkeepNA = TRUE
not supported -
paste()
: thecollapse
argument is not yet supported -
paste0()
: thecollapse
argument is not yet supported -
strptime()
: accepts aunit
argument not present in thebase
function. Valid values are "s", "ms" (default), "us", "ns". -
substr()
:start
andstop
must be length 1
bit64
dplyr
-
case_when()
:.ptype
and.size
arguments not supported
lubridate
-
dmy()
:locale
argument not supported -
dmy_h()
:locale
argument not supported -
dmy_hm()
:locale
argument not supported -
dmy_hms()
:locale
argument not supported -
dpicoseconds()
: not supported -
dym()
:locale
argument not supported -
fast_strptime()
: non-default values oflt
andcutoff_2000
not supported -
force_tz()
: Timezone conversion from non-UTC timezone not supported;roll_dst
values of 'error' and 'boundary' are supported for nonexistent times,roll_dst
values of 'error', 'pre', and 'post' are supported for ambiguous times. -
make_datetime()
: only supports UTC (default) timezone -
make_difftime()
: only supportsunits = "secs"
(the default); providing bothnum
and...
is not supported -
mdy()
:locale
argument not supported -
mdy_h()
:locale
argument not supported -
mdy_hm()
:locale
argument not supported -
mdy_hms()
:locale
argument not supported -
my()
:locale
argument not supported -
myd()
:locale
argument not supported -
parse_date_time()
:quiet = FALSE
is not supported Available formats are H, I, j, M, S, U, w, W, y, Y, R, T. On Linux and OS X additionally a, A, b, B, Om, p, r are available. -
ydm()
:locale
argument not supported -
ydm_h()
:locale
argument not supported -
ydm_hm()
:locale
argument not supported -
ydm_hms()
:locale
argument not supported -
ym()
:locale
argument not supported -
ymd()
:locale
argument not supported -
ymd_h()
:locale
argument not supported -
ymd_hm()
:locale
argument not supported -
ymd_hms()
:locale
argument not supported -
yq()
:locale
argument not supported
methods
rlang
stats
-
median()
: approximate median (t-digest) is computed -
quantile()
:probs
must be length 1; approximate quantile (t-digest) is computed
stringi
stringr
Pattern modifiers coll()
and boundary()
are not supported in any functions.
-
str_c()
: thecollapse
argument is not yet supported -
str_count()
:pattern
must be a length 1 character vector -
str_split()
: Case-insensitive string splitting and splitting into 0 parts not supported -
str_sub()
:start
andend
must be length 1