neon_download {neonstore}

Download NEON data products into a local store

Description

Download NEON data products into a local store

Usage

neon_download(
  product,
  table = NA,
  site = NA,
  start_date = NA,
  end_date = NA,
  type = "basic",
  release = NA,
  quiet = FALSE,
  verify = TRUE,
  unique = TRUE,
  dir = neon_dir(),
  get_zip = FALSE,
  unzip = FALSE,
  api = "https://data.neonscience.org/api/v0",
  .token = Sys.getenv("NEON_TOKEN")
)

Arguments

product

A NEON productCode or list of product codes, see examples.

table

Include only files matching this table name (or regex pattern). (optional).

site

4-letter site code(s) to filter on. Leave as NA to search all.

start_date

Download only files as recent as (YYYY-MM-DD). Leave as NA to download up to the most recent available data.

end_date

Download only files up to end_date (YYYY-MM-DD). Leave as NA to download all prior data.

type

Should we prefer the basic or expanded version of this product? Note that not all products have expanded formats.

release

Select only data files associated with a particular release tag, see https://www.neonscience.org/data-samples/data-management/data-revisions-releases, e.g. "RELEASE-2021". Releases are associated with a specific DOI and the promise that files associated with a particular release will not change.

quiet

Should download progress be displayed?

verify

Should downloaded files be compared against the MD5 hash reported by the NEON API to verify integrity? (default TRUE)

unique

Should we skip downloads of files we already have? Note: file comparisons are based on file hash, which will omit files that have identical content but different names.

dir

Location where files should be downloaded. By default will use the appropriate applications directory for your system (see tools::R_user_dir()). This default also be configured by setting the environmental variable NEONSTORE_HOME, see Sys.setenv or Renviron.

get_zip

should we attempt to download .zip archive versions of files? default FALSE, as zip archives are being deprecated from NEON API starting in early 2021.

unzip

should we extract .zip files? (default TRUE). Note: .zip files are preserved in the store to avoid repeated downloads. Use of .zip files in NEON API is now deprecated in favor of requesting individual files.

api

the URL to the NEON API, leave as default.

.token

an authentication token from NEON. A token is not required but will allow access to a higher number of requests before rate limiting applies, see https://data.neonscience.org/data-api/rate-limiting/#api-tokens. Note that once files are downloaded once, neonstore provides persistent access to them without further interaction required with the API.

Details

Each NEON data product consists of a collection of objects (e.g. tables), which are in turn broken into individual files by site and sampling month. Additionally, many NEON products have been expanded, including some additional columns. Consequently, users must specify if they want the "basic" or "expanded" version of this data.

In the products table (see neon_products), the productHasExpanded column indicates if the data product has expanded, and the columns productHasBasicDescription and productHasExpandedDescription provide a detailed explanation of the differences between the "expanded" and "basic" versions of that particular product.

The API allows users to request component files directly. By default, neon-download() will download all available extensions. Users can request only products of a certain format (e.g. .csv or .h5) by altering the file_regex argument (see examples).

Prior to 2021, the API provided access to a .zip file containing all the component objects (e.g. tables) for that product at that site and sampling month.

neon_download() will avoid downloading metadata files which are bitwise identical to other files in the same download request, as indicated by the crc32 hash reported by the API. These typically include metadata that are shared across the product as a whole, but are for some reason included in each sampling month for each site – potentially thousands of duplicates. These duplicates are also packaged within the .zip downloads where it is not possible to exclude them from the download.

Examples

 
 ## Omit dir=tempfile() to use persistent storage
 neon_download("DP1.10003.001", 
               start_date = "2018-01-01", 
               end_date = "2019-01-01",
               site = "YELL",
               dir = tempfile())
               
 ## Advanced use: filter for a particular table in the product
 neon_download(product = "DP1.10003.001",
               start_date = "2018-01-01",
               end_date = "2019-01-01",
               site = "YELL",
               table = "countdata",
               dir = tempfile())