dataset_shard {tfdatasets}

Creates a dataset that includes only 1 / num_shards of this dataset.

Description

This dataset operator is very useful when running distributed training, as it allows each worker to read a unique subset.

Usage

dataset_shard(dataset, num_shards, index)

Arguments

dataset

A dataset

num_shards

A integer representing the number of shards operating in parallel.

index

A integer, representing the worker index.

Value

A dataset