Title: | Download and Process GIMMS NDVI3g Data |
---|---|
Description: | This is a set of functions to retrieve information about GIMMS NDVI3g files currently available online; download (and re-arrange, in the case of NDVI3g.v0) the half-monthly data sets; import downloaded files from ENVI binary (NDVI3g.v0) or NetCDF format (NDVI3g.v1) directly into R based on the widespread 'raster' package; conduct quality control; and generate monthly composites (e.g., maximum values) from the half-monthly input data. As a special gimmick, a method is included to conveniently apply the Mann-Kendall trend test upon 'Raster*' images, optionally featuring trend-free pre-whitening to account for lag-1 autocorrelation. |
Authors: | Florian Detsch [cre, aut] |
Maintainer: | Florian Detsch <[email protected]> |
License: | MIT + file LICENSE |
Version: | 1.2.2 |
Built: | 2024-11-11 06:00:22 UTC |
Source: | https://github.com/environmentalinformatics-marburg/gimms |
Download and Process GIMMS NDVI3g Data
This is a set of functions to retrieve information about GIMMS NDVI3g files
currently available online; download (and re-arrange, in the case of
NDVI3g.v0) the half-monthly data sets; import downloaded files from ENVI
binary (NDVI3g.v0) or NetCDF format (NDVI3g.v1) directly into R based on the
widespread raster package; conduct quality control; and generate
monthly composites (e.g., maximum values) from the half-monthly input data.
As a special gimmick, a method is included to conveniently apply the
Mann-Kendall trend test upon Raster*
images, optionally featuring
trend-free pre-whitening to account for lag-1 autocorrelation.
Florian Detsch
Pinzon, JE & Tucker, CJ (2014). A Non-Stationary 1981-2012 AVHRR NDVI3g Time Series. Remote Sensing, 6(8), 6929-6960, doi:10.3390/rs6086929.
The National Center for Atmospheric Research (2018). A Big Earth Data Platform for Three Poles. Global GIMMS NDVI3g v1 dataset (1981-2015). Available online at http://poles.tpdc.ac.cn/en/data/9775f2b4-7370-4e5e-a537-3482c9a83d88/ (accessed on 2021-04-15).
Bale Mountains NDVI3g.v1.
raster::RasterStack
This dataset contains NDVI3g.v1 observations for the Bale Mountains National Park, southern Ethiopia (Jul 1981 to Dec 2015).
Download GIMMS NDVI3g data, optionally for a given period of time. NDVI3g.v1
(NetCDF, until end 2015) is available from the "poles"
and
"ecocast"
servers, whereas NDVI3g.v0 (ENVI binary, until end 2013) is
"ecocast"
and "nasanex"
exclusive.
## S4 method for signature 'dateORmissing,dateORmissing' downloadGimms( x, y, version = 1L, dsn = getwd(), overwrite = FALSE, quiet = TRUE, mode = "wb", cores = 1L, server = c("poles", "nasanex", "ecocast"), ... ) ## S4 method for signature 'numericORmissing,numericORmissing' downloadGimms( x, y, version = 1L, dsn = getwd(), overwrite = FALSE, quiet = TRUE, mode = "wb", cores = 1L, server = c("poles", "nasanex", "ecocast"), ... ) ## S4 method for signature 'character,ANY' downloadGimms( x, dsn = getwd(), overwrite = FALSE, quiet = TRUE, mode = "wb", cores = 1L, ... ) ## S4 method for signature 'missing,missing' downloadGimms( version = 1L, dsn = getwd(), overwrite = FALSE, quiet = TRUE, mode = "wb", cores = 1L, server = c("poles", "nasanex", "ecocast"), ... )
## S4 method for signature 'dateORmissing,dateORmissing' downloadGimms( x, y, version = 1L, dsn = getwd(), overwrite = FALSE, quiet = TRUE, mode = "wb", cores = 1L, server = c("poles", "nasanex", "ecocast"), ... ) ## S4 method for signature 'numericORmissing,numericORmissing' downloadGimms( x, y, version = 1L, dsn = getwd(), overwrite = FALSE, quiet = TRUE, mode = "wb", cores = 1L, server = c("poles", "nasanex", "ecocast"), ... ) ## S4 method for signature 'character,ANY' downloadGimms( x, dsn = getwd(), overwrite = FALSE, quiet = TRUE, mode = "wb", cores = 1L, ... ) ## S4 method for signature 'missing,missing' downloadGimms( version = 1L, dsn = getwd(), overwrite = FALSE, quiet = TRUE, mode = "wb", cores = 1L, server = c("poles", "nasanex", "ecocast"), ... )
x |
Start time for data download as either |
y |
End time for data download as either |
version |
|
dsn |
|
overwrite |
|
quiet |
|
mode |
|
cores |
|
server |
See |
... |
Further arguments passed to |
A character
vector of local filepaths.
updateInventory
, download.file
.
## Not run: tmp <- tempdir() ## 'Date' method gimms_files_date <- downloadGimms(x = as.Date("2000-01-01"), y = as.Date("2000-12-31"), dsn = tmp) ## 'numeric' method (i.e., particular years) gimms_files_year <- downloadGimms(x = 2000, y = 2002, dsn = tmp) ## 'character' method (i.e., particular files) ecocast <- system.file("extdata", "inventory_ecv1.rds", package = "gimms") gimms_files_char <- readRDS(ecocast) gimms_files_char <- downloadGimms(x = gimms_files_char[1:6], dsn = tmp) ## 'missing' method (i.e., entire collection) gimms_files_full <- downloadGimms(dsn = tmp) ## End(Not run)
## Not run: tmp <- tempdir() ## 'Date' method gimms_files_date <- downloadGimms(x = as.Date("2000-01-01"), y = as.Date("2000-12-31"), dsn = tmp) ## 'numeric' method (i.e., particular years) gimms_files_year <- downloadGimms(x = 2000, y = 2002, dsn = tmp) ## 'character' method (i.e., particular files) ecocast <- system.file("extdata", "inventory_ecv1.rds", package = "gimms") gimms_files_char <- readRDS(ecocast) gimms_files_char <- downloadGimms(x = gimms_files_char[1:6], dsn = tmp) ## 'missing' method (i.e., entire collection) gimms_files_full <- downloadGimms(dsn = tmp) ## End(Not run)
Kilimanjaro NDVI3g.v0.
raster::RasterStack
This dataset contains NDVI3g.v0 observations for the Kilimanjaro region, northern Tanzania (Jul 1981 to Dec 2013).
Based on a user-defined function, e.g. max
for maximum value
composites (MVC), aggregate half-monthly GIMMS data sets to monthly composites.
## S4 method for signature 'RasterStackBrick' monthlyComposite(x, indices, fun = max, cores = 1L, filename = "", ...) ## S4 method for signature 'character' monthlyComposite( x, version = 1L, pos1 = ifelse(version == 1, 15L, 4L), pos2 = ifelse(version == 1, 23L, 8L), fun = max, cores = 1L, filename = "", ... )
## S4 method for signature 'RasterStackBrick' monthlyComposite(x, indices, fun = max, cores = 1L, filename = "", ...) ## S4 method for signature 'character' monthlyComposite( x, version = 1L, pos1 = ifelse(version == 1, 15L, 4L), pos2 = ifelse(version == 1, 23L, 8L), fun = max, cores = 1L, filename = "", ... )
x |
Multi-layered |
indices |
|
fun |
|
cores |
|
filename |
|
... |
Further arguments passed to |
version |
|
pos1 , pos2
|
|
If length(x) == 2
, a single RasterLayer
object, else a
RasterStack
object with monthly composite layers.
stackApply
, monthlyIndices
,
writeRaster
.
data("bale3g.v1") ## select layers from 1981 only fls <- system.file("extdata/inventory_ecv1.rds", package = "gimms") fls <- readRDS(fls)[1] rst <- bale3g.v1[[1:12]] ## aggregate to monthly mvc layers mvc <- monthlyComposite(rst, indices = monthlyIndices(fls))
data("bale3g.v1") ## select layers from 1981 only fls <- system.file("extdata/inventory_ecv1.rds", package = "gimms") fls <- readRDS(fls)[1] rst <- bale3g.v1[[1:12]] ## aggregate to monthly mvc layers mvc <- monthlyComposite(rst, indices = monthlyIndices(fls))
Create numeric monthly indices from (local or online) GIMMS NDVI3g filenames
as input for the monthlyComposite
function.
monthlyIndices( x, version = 1L, pos1 = ifelse(version == 1, 15L, 4L), pos2 = ifelse(version == 1, 23L, 8L), timestamp = FALSE, ... )
monthlyIndices( x, version = 1L, pos1 = ifelse(version == 1, 15L, 4L), pos2 = ifelse(version == 1, 23L, 8L), timestamp = FALSE, ... )
x |
|
version |
|
pos1 , pos2
|
|
timestamp |
|
... |
Currently not used. |
A numeric
vector with unique monthly indices or, if
timestamp = TRUE
, the actual timestamps as Date
objects.
## NDVI3g.v1 gimms_files_v1 <- readRDS(system.file("extdata", "inventory_ecv1.rds", package = "gimms")) monthlyIndices(gimms_files_v1[1], version = 1) # indices monthlyIndices(gimms_files_v1[1], version = 1, timestamp = TRUE) # dates ## Similarly, NDVI3g.v0 gimms_files_v0 <- readRDS(system.file("extdata", "inventory_ecv0.rds", package = "gimms")) gimms_files_v0 <- rearrangeFiles(gimms_files_v0) # reorder by date monthlyIndices(gimms_files_v0[1:12], version = 0) monthlyIndices(gimms_files_v0[1:12], version = 0, timestamp = TRUE)
## NDVI3g.v1 gimms_files_v1 <- readRDS(system.file("extdata", "inventory_ecv1.rds", package = "gimms")) monthlyIndices(gimms_files_v1[1], version = 1) # indices monthlyIndices(gimms_files_v1[1], version = 1, timestamp = TRUE) # dates ## Similarly, NDVI3g.v0 gimms_files_v0 <- readRDS(system.file("extdata", "inventory_ecv0.rds", package = "gimms")) gimms_files_v0 <- rearrangeFiles(gimms_files_v0) # reorder by date monthlyIndices(gimms_files_v0[1:12], version = 0) monthlyIndices(gimms_files_v0[1:12], version = 0, timestamp = TRUE)
Reformat the names of local NDVI3g.v1 files according to "traditional" NDVI3g.v0 naming convention.
oldNaming(x, suffix = "")
oldNaming(x, suffix = "")
x |
|
suffix |
|
The satellite number, which was an essential constituent of the NDVI3g.v0
file naming convention (see 'References'), has been removed from NDVI3g.v1
file names. However, it can easily be accessed via ncvar_get
given that the file has previously been downloaded. If an NDVI3g.v1 file
specified to 'x' is not available locally, by contrast, the satellite number
is retrieved from the built-in NDVI3g.v0 file inventory (until end 2013; see
'Examples') and newer files are tagged with "XX"
indicating missing
information.
A character
vector of file names following the "traditional" NDVI3g.v0
naming convention.
http://nasanex.s3.amazonaws.com/AVHRR/GIMMS/3G/00READMEgeo.txt (accessed on 2021-04-15).
fls <- system.file("extdata", "inventory_ecv1.rds", package = "gimms") fls <- basename(readRDS(fls)) oldNaming(fls[64:65], suffix = ".tif") # adds .tif extension oldNaming(fls[65:66]) # tags newer files with 'XX'
fls <- system.file("extdata", "inventory_ecv1.rds", package = "gimms") fls <- basename(readRDS(fls)) oldNaming(fls[64:65], suffix = ".tif") # adds .tif extension oldNaming(fls[65:66]) # tags newer files with 'XX'
Perform quality control on GIMMS NDVI3g data based on the companion flag
information. This is merely a wrapper around overlay
and, since
quality control is readily available through rasterizeGimms
, it
is strongly recommended to use the latter function for this purpose.
## S4 method for signature 'RasterStackBrick' qualityControl(x, keep = NULL, filename = "", ...)
## S4 method for signature 'RasterStackBrick' qualityControl(x, keep = NULL, filename = "", ...)
x |
A single 2-layered |
keep |
|
filename |
|
... |
Further arguments passed to |
If 'keep' is missing, the function will automatically skip quality control and return the input object.
A quality-controlled 'RasterLayer' object.
## Not run: tmp <- tempdir() ## Download NDVI3g.v1 sample data ecocast <- system.file("extdata", "inventory_ecv1.rds", package = "gimms") gimms_files <- downloadGimms(readRDS(ecocast)[1], dsn = tmp) ## Import data as 'Raster*' objects ndvi <- raster::raster(gimms_files, varname = "ndvi") ndvi[ndvi[] %in% c(-32768, -3000)] <- NA ndvi <- ndvi / 1e4 flag <- floor(raster::raster(gimms_files, varname = "percentile") / 2e3) ## Perform quality control and visualize to_check <- stack(ndvi[[1]], flag[[1]]) qcl <- qualityControl(to_check, keep = 0) plot(qcl) ## End(Not run) ## see also 'Examples' section in ?rasterizeGimms for automated quality check
## Not run: tmp <- tempdir() ## Download NDVI3g.v1 sample data ecocast <- system.file("extdata", "inventory_ecv1.rds", package = "gimms") gimms_files <- downloadGimms(readRDS(ecocast)[1], dsn = tmp) ## Import data as 'Raster*' objects ndvi <- raster::raster(gimms_files, varname = "ndvi") ndvi[ndvi[] %in% c(-32768, -3000)] <- NA ndvi <- ndvi / 1e4 flag <- floor(raster::raster(gimms_files, varname = "percentile") / 2e3) ## Perform quality control and visualize to_check <- stack(ndvi[[1]], flag[[1]]) qcl <- qualityControl(to_check, keep = 0) plot(qcl) ## End(Not run) ## see also 'Examples' section in ?rasterizeGimms for automated quality check
Import GIMMS NDVI3g (binary or NetCDF) data into R as Raster*
objects.
rasterizeGimms( x, ext = NULL, snap = "out", keep = NULL, split = FALSE, cores = 1L, filename = "", ... )
rasterizeGimms( x, ext = NULL, snap = "out", keep = NULL, split = FALSE, cores = 1L, filename = "", ... )
x |
|
ext |
|
snap |
|
keep |
|
split |
|
cores |
|
filename |
|
... |
Further arguments passed to |
If split = TRUE
, a list of NDVI3g RasterStack
objects
corresponding to the files specified in 'x'; else a single NDVI3g
RasterStack
object.
crop
, qualityControl
,
writeRaster
.
## Not run: tmp <- tempdir() ## Download NDVI3g.v1 sample data gimms_files <- downloadGimms(x = as.Date("2000-01-01"), y = as.Date("2000-12-31"), dsn = tmp) ## Extent for clipping shp <- getData("GADM", country = "DEU", level = 0, path = tmp) ## Rasterize without quality control gimms_raster <- rasterizeGimms(x = gimms_files, ext = shp) # clipping plot(gimms_raster[[1]]) lines(shp) ## Rasterize with quality control gimms_rasterq <- rasterizeGimms(x = gimms_files, ext = shp, # clipping keep = 0) # quality control plot(gimms_rasterq[[1]]) lines(shp) ## End(Not run)
## Not run: tmp <- tempdir() ## Download NDVI3g.v1 sample data gimms_files <- downloadGimms(x = as.Date("2000-01-01"), y = as.Date("2000-12-31"), dsn = tmp) ## Extent for clipping shp <- getData("GADM", country = "DEU", level = 0, path = tmp) ## Rasterize without quality control gimms_raster <- rasterizeGimms(x = gimms_files, ext = shp) # clipping plot(gimms_raster[[1]]) lines(shp) ## Rasterize with quality control gimms_rasterq <- rasterizeGimms(x = gimms_files, ext = shp, # clipping keep = 0) # quality control plot(gimms_rasterq[[1]]) lines(shp) ## End(Not run)
Rearrange local GIMMS NDVI3g.v0 files in ascending order of time. Since the naming convention has significantly changed towards NDVI3g.v1, such a measure should only be relevant for older file formats.
rearrangeFiles( x, dsn = getwd(), pattern = "^geo.*.VI3g$", pos = c(4, 6, 11), ... )
rearrangeFiles( x, dsn = getwd(), pattern = "^geo.*.VI3g$", pos = c(4, 6, 11), ... )
x |
|
dsn |
|
pattern |
|
pos |
|
... |
Further arguments passed to |
A character
vector of filepaths arranged in ascending order of time.
Apply the Mann-Kendall trend test (Mann, 1945) to a series of observations
and return Kendall's (Kendall, 1938) based on a predefined
significance level. In contrast to other readily available implementations,
it is up to the user to decide whether or not to apply pre-whitening as
described in the zyp package vignette (Bronaugh and Werner, 2013).
## S4 method for signature 'numeric' significantTau( x, p = 0.001, prewhitening = TRUE, method = c("yuepilon", "zhang"), df = FALSE ) ## S4 method for signature 'RasterStackBrick' significantTau( x, p = 0.001, prewhitening = TRUE, method = c("yuepilon", "zhang"), filename = "", ... )
## S4 method for signature 'numeric' significantTau( x, p = 0.001, prewhitening = TRUE, method = c("yuepilon", "zhang"), df = FALSE ) ## S4 method for signature 'RasterStackBrick' significantTau( x, p = 0.001, prewhitening = TRUE, method = c("yuepilon", "zhang"), filename = "", ... )
x |
Either a |
p |
|
prewhitening |
|
method |
|
df |
|
filename |
|
... |
Further arguments passed to |
If available, the function will automatically use open multi-core clusters
for parallel processing (see beginCluster
and Examples).
numeric
input: If df = FALSE
(default), a single
numeric
or logical
(i.e., NA
) depending on whether or
not 'p' was exceeded; else a data.frame
with Kendall's and
the corresponding significance level.
RasterStackBrick
input: A RasterLayer
with values of
Kendall's . Values exceeding the specified 'p' are discarded.
Kendall, M.G. (1938). A new measure of rank correlation. Biometrika 30(1/2), 81-93, doi: 10.2307/2332226.
Mann, H.B. (1945). Nonparametric tests against trend. Econometrica 13(3), 245-259, doi: 10.2307/1907187.
Zhang, X., Vincent, L.A., Hogg, W.D. and A. Niitsoo (2000). Temperature and Precipitation Trends in Canada during the 20th Century. Atmosphere-Ocean 38(3), 395-429, doi:10.1080/07055900.2000.9649654.
Yue, S., Pilon, P., Phinney, B. and G. Cavadias (2002). The influence of autocorrelation on the ability to detect trend in hydrological series. Hydrological Processes 16, 1807-1829,doi:10.1002/hyp.1095.
MannKendall
, zyp.trend.vector
.
## Example taken from ?Kendall::MannKendall library(Kendall) data(PrecipGL) plot(PrecipGL) ## Mann-Kendall trend test without pre-whitening x <- as.numeric(PrecipGL) significantTau(x, p = 0.001, prewhitening = FALSE, df = TRUE) ## Mann-Kendall trend test with pre-whitening significantTau(x, p = 0.001, prewhitening = TRUE, df = TRUE) ############################################################################# ### use case: significant mann-kendall trends in ndvi3g.v0 ######### ############################################################################# ## Not run: ## Sample data from 1982 to 2013 data("kili3g.v0") rst <- kili3g.v0[[13:nlayers(kili3g.v0)]] ## Remove seasonal signal library(remote) dsn <- deseason(rst, cycle.window = 24) ## Apply trend-free pre-whitened Mann-Kendall test (note that ## non-significant pixels are set to NA) trd1 <- significantTau(dsn, p = 0.01, prewhitening = TRUE) plot(trd1) ## Or, alternatively, use multi-core functionality cores <- parallel::detectCores() - 1 if (require(snow)) { beginCluster(cores) trd2 <- significantTau(dsn, p = 0.01, prewhitening = TRUE) endCluster() } ## End(Not run)
## Example taken from ?Kendall::MannKendall library(Kendall) data(PrecipGL) plot(PrecipGL) ## Mann-Kendall trend test without pre-whitening x <- as.numeric(PrecipGL) significantTau(x, p = 0.001, prewhitening = FALSE, df = TRUE) ## Mann-Kendall trend test with pre-whitening significantTau(x, p = 0.001, prewhitening = TRUE, df = TRUE) ############################################################################# ### use case: significant mann-kendall trends in ndvi3g.v0 ######### ############################################################################# ## Not run: ## Sample data from 1982 to 2013 data("kili3g.v0") rst <- kili3g.v0[[13:nlayers(kili3g.v0)]] ## Remove seasonal signal library(remote) dsn <- deseason(rst, cycle.window = 24) ## Apply trend-free pre-whitened Mann-Kendall test (note that ## non-significant pixels are set to NA) trd1 <- significantTau(dsn, p = 0.01, prewhitening = TRUE) plot(trd1) ## Or, alternatively, use multi-core functionality cores <- parallel::detectCores() - 1 if (require(snow)) { beginCluster(cores) trd2 <- significantTau(dsn, p = 0.01, prewhitening = TRUE) endCluster() } ## End(Not run)
Download the latest version of the GIMMS NDVI3g file inventory from the National Center for Atmospheric Research, NASA Earth Exchange (NEX) Amazon AWS or Ames Ecological Forecasting Lab (ECOCAST). If the specified endpoint is not reachable (e.g., if there is no active internet connection), the latest local version of the file inventory is used.
updateInventory( server = c("poles", "nasanex", "ecocast"), version = 1L, quiet = FALSE )
updateInventory( server = c("poles", "nasanex", "ecocast"), version = 1L, quiet = FALSE )
server |
|
version |
|
quiet |
|
GIMMS NDVI3g.v1 is currently available from ECOCAST and The National Center
for Atmospheric Research until end 2015 and comes in NetCDF (.nc4
)
format. In contrast, NDVI3g.v0 is available as ENVI binary imagery and
available from ECOCAST (NASANEX) until end 2013 (2012) only.
A character
vector of online filepaths.
The National Center for Atmospheric Research (2018). A Big Earth Data Platform for Three Poles. Global GIMMS NDVI3g v1 dataset (1981-2015). Available online at http://poles.tpdc.ac.cn/en/data/9775f2b4-7370-4e5e-a537-3482c9a83d88/ (accessed on 2021-04-15).
## Not run: updateInventory() updateInventory(server = "nasanex", version = 0) ## End(Not run) ## note that local versions of the online file inventories are also available ofl_ecv1 <- system.file("extdata", "inventory_ecv1.rds", package = "gimms") readRDS(ofl_ecv1) ofl_v0 <- system.file("extdata", "inventory_ecv0.rds", package = "gimms") readRDS(ofl_v0) ofl_plv1 <- system.file("extdata", "inventory_plv1.rds", package = "gimms") readRDS(ofl_plv1)
## Not run: updateInventory() updateInventory(server = "nasanex", version = 0) ## End(Not run) ## note that local versions of the online file inventories are also available ofl_ecv1 <- system.file("extdata", "inventory_ecv1.rds", package = "gimms") readRDS(ofl_ecv1) ofl_v0 <- system.file("extdata", "inventory_ecv0.rds", package = "gimms") readRDS(ofl_v0) ofl_plv1 <- system.file("extdata", "inventory_plv1.rds", package = "gimms") readRDS(ofl_plv1)