DataLoader
(
data
,
batchsize
=
1;
partial
=
true
,
collate
=
true
,
buffered
=
collate
,
parallel
=
Threads
.
nthreads
(
)
>
1
,
useprimary
=
false
,
)
Create an efficient iterator of batches over
data container
data
.
Arguments
Positional
-
data: A data container supporting theLearnBasedata access pattern -
batchsize = 1: Number of samples to batch together . Disable batching by setting tonothing.
Keyword
-
partial::Bool = true: Whether to include the last batch whennobs(dataset)is not divisible bybatchsize.trueensures all batches have the same size, but some samples might be dropped . -
buffered::Bool = collate: Ifbufferedistrue, loads data inplace usinggetobs!. See Data containers for details on buffered loading . -
parallel::Bool = Threads.nthreads() > 1): Whether to load data in parallel, keeping the primary thread is . Default istrueif more than one thread is available . -
useprimary::Bool = false: Iffalse, keep the main thread free when loading data in parallel . Is ignored ifparallelisfalse.
Examples
Creating a data loader with batch size 16 and iterating over it:
data
=
(
rand
(
128
,
10000
)
,
rand
(
1
,
10000
)
)
dataloader
=
DataLoader
(
data
,
16
)
for
(
xs
,
ys
)
in
dataloader
endCreating a data loader that uses buffers to load batches:
data
=
rand
(
100
,
64
)
dataloader
=
DataLoader
(
data
,
16
,
buffered
=
true
)
size
(
first
(
dataloader
)
)
==
(
100
,
16
)Turning off collating :
dataloader
=
DataLoader
(
data
,
16
,
collate
=
false
)
# Batches are a vector of observations
length
(
first
(
dataloader
)
)
==
16