Product Quality Assurance Documen= t (PQAD) (2024)

Last modified on J=un 07, 2024 14:12

Contributors: Jacqueline Bannwart (University of Zurich), In=C3==A9s Dussailant (University of Zurich), Frank Paul (Unive=rsity of Zurich), Michael Zemp (University of Zurich)

Issued by: UZH / Frank Paul

Date: 12/10/2023

Ref: C3S2_312a_Lot4.WP1-PDDP-GL-v2_202306_A_PQAD-v5_i1.1

Official reference number service contract: 2021/C3S2_312a_Lot4_EODC/SC1=

Brokered data set: A dataset that is made available in =the Climate Data Store (CDS) but freely available (under given license cond=itions) from external sources. In the case of the glacier distribution serv=ice, the Randolph Glacier Inventory (RGI) is brokered for the CDS from https://glims.org/RGI under a CC-BY 4.0 license.

Debris-cover: Debris on a glacier is usually composed o=f unsorted rock fragments with highly variable grain size (from mm to sever=al m). These might cover the ice in lines of variable width separating ice =with origin in different accumulation regions of a glacier (so called media=l moraines) up to a complete coverage of the ablation region. Automated map=ping of glacier ice is only possible when the debris is not covering the ic=e completely when compared to the pixel size of the satellite image, i.e. s=ome clean ice must be visible too.

Glacier area: The area (or size) of a glacier, usually =given in the unit km². Also used by the Global Climate Observing= System (GCOS) to name the related Essential Climate Variable (ECV) product=.

Glacier outline: A vector dataset with polygon topology= marking the boundary of a glacier.

Glacier inventory: A compilation of glacier outlines wi=th associated attribute information.

Glacier complex: A contiguous ice mass that is the resu=lt of the binary (yes/no) glacier classification after conversion from rast=er to vector format. Usually, the glacier complexes are divided into indivi=dual glaciers by digital intersection with a vector layer containing hydrol=ogic divides derived from watershed analysis of a digital elevation model (=DEM).

Geographic Information System (GIS): A software to visu=alize, process and edit spatial datasets in vector and raster format.

This document is the Product Quality Assurance Document (PQAD) for the C=opernicus Climate Change Service (C3S) glacier distribution service. It des=cribes the validated products (glacier outlines), the datasets and methods =used for validation and uncertainty assessment, and provides an overarching= view on uncertainties derived for the dataset in the Climate Data Store (C=DS), the Randolph Glacier Inventory (RGI).

The glacier distribution datasets (RGI v5.0, v6.0 and v7.0) as delivered= to the CDS, consists of a global compilation of glacier outlines derived f=rom air and space-borne sensors or maps. Each dataset in this compilation h=as been quality checked and corrected by the analyst providing it, usually =by on-screen digitizing to remove systematic errors of the classification. =The classical concept of validation as applied to global satellite products= (e.g. M=C3=BCller 2014 and references therein, Wan et al. 2004), hence, ca=nnot be applied. Instead, the uncertainty of the glacier outlines is determ=ined from a range of methods that can be ranked by the workload required to= apply them.

In this document we first provide an overview on the validated products =(Section 1) and what =E2=80=98validation=E2=80=99 means in this context bef=ore we describe in Section 2 the characteristics of the datasets used for v=alidation. In the main part of Section 3 we introduce the various sources o=f uncertainty and describe the methods used to determine uncertainty. A col=lective summary of the uncertainty assessment for datasets in the RGI is pr=ovided in Section 4, the uncertainty for the datasets produced by C3S are p=rovided in the PQAR [RD2].

1. Validated products

As a general remark, we have to distinguish (A) the product that is prov=ided to the Climate Data Store (CDS) from (B) the products that are created= by the C3S glacier distribution service. The dataset provided to the CDS i=s the latest version of the Randolph Glacier Inventory (RGI). It contains e=xactly one outline and related attribute information (e.g. topographic data=) in an open vector format (shape file) along with information about glacie=r hypsometry (area-elevation distribution) in an additional csv file (see t=he Product User Guide and Specification [RD1] for details) for all =of 215,000 glaciers globally. The dataset is extracted from the Global Land= Ice Measurements from Space (GLIMS) glacier database (https://glims.org<=/a>), which is multi-temporal and might contain several outlines for= the same glacier, usually from different points in time. The GLIMS databas=e contains datasets provided by the glacier mapping community using a range= of methods and datasets (e.g. satellite images, aerial photography, topogr=aphic maps) to derive them. These outlines are thus very diverse in quality= and characteristics. The products created by the C3S glacier distribution =service cover glacierized sub-regions (e.g. glaciers on Baffin Island) and =are also submitted to GLIMS from where they are possibly extracted for the =RGI (https://glims.org/RGI¹). We are thus a par=t of the glacier mapping community feeding the GLIMS database.

We also have to clarify that glacier outlines are in general not validat=ed in a traditional sense. This is due to the fact that appropriate validat=ion data (i.e. higher resolution datasets from about the same date) are sel=dom available or too expensive. Hence, only measures determining uncertaint=ies (random errors) are usually applied and reported (Section 3). On the ot=her hand, all glacier outlines are quality checked and corrected against th=e satellite data from which they are derived. This correction of omission a=nd commission errors removes the systematic errors of automated glacier map=ping (this is not required for fully manual digitizing). For example, lakes= are misclassified as glaciers and have to be removed, whereas debris-cover=ed glacier parts or regions in shadow might have been missed and have to be= added. Hence, all glacier outlines (in the RGI and as provided to GLIMS) a=re validated and corrected against reference data, i.e. the images they are= derived from. This =E2=80=98standard validation=E2=80=99 removes =systematic errors, but does not have a quantitative measure.

Validation against external datasets (e.g. higher resolution imagery) is= applied at three levels:

(1) visual inspection to improve interpre=tation (for outline correction);
(2) if orthorectified, direct digit=izing of outlines (for outline correction);
(3) independent digitizing t=o create a reference dataset for 'real' validation, i.e. accuracy assessmen=t.

Points (1) and (2) are part of the standard validation and only= (3) can be seen as a =E2=80=98real=E2=80=99 validation, at least technical=ly (if the datasets are really appropriate for validation is another issue)=. All glacier outlines are subject to a =E2=80=99standard validation=E2=80=99, but the effect is in general not quantified. On the other hand=, a =E2=80=98real=E2=80=99 validation is only rarely performed and results =might be very specific for the region. What remains to be reported for all =datasets are uncertainties, e.g. resulting from a variable digitiz=ing when interpreting glacier features. A range of methods have been develo=ped for this purpose (see Section 3) and results are usually reported in re=lated publications. For the RGI as a whole we present generalized results o=f the uncertainty assessment in Section 4 and, for the datasets created by =C3S, uncertainties are reported in the PQAR [RD2].

2. Descrip=tion of validating datasets

As described in Section 1, the datasets used for the standard =validation are:

(a) the satellite images used for creatin=g the original glacier outlines (usually Landsat, Sentinel-2 or Advanced Sp=aceborne Thermal Emission and Reflection Radiometer (ASTER), sometimes also= Satellites Pour l'Observation de la Terre (SPOT) and other multispectral o=ptical sensors);
(b) higher resol=ution datasets available from Google Earth, Bing, and national mapping agen=cies (for improved visual interpretation);
(c) orthorectified and geocoded images provided by web-map= services for direct digitizing.

The orthorectified images from (a) and (c) can be directly imported into= a Geographic Information System (GIS) and used as a background for correct=ing the glacier outlines. The satellite images used to create glacier outli=nes and perform the standard validation differ in spatial resolutions (in g=eneral 10 to 30 m), spectral bands and radiometric resolution. This means t=hat visibility of details and possibilities for contrast enhancement varies= for each sensor. Moreover, mapping conditions (e.g. clouds, shadow, season=al snow) change from image to image and the analysts performing the correct=ions have differing experience. In effect, the corrected / validated datase=ts also have differing quality. The quality can be slightly improved using =visual inspection of higher resolution images (b), but only corrections usi=ng type (c) images can improve outline quality substantially, at least when= the available images are of good quality (e.g. regarding snow and cloud co=nditions).

The major benefit of type (c) images is their much higher spatial resolu=tion (up to 30 cm), which allows a much better interpretation of glaciologi=cal and geomorphological features, in particular of debris-covered glaciers=. However, there are also disadvantages, e.g. the image is fixed and can ha=ve adverse snow conditions, other band combinations or contrast enhancement= cannot be applied, and sometimes the geolocation is shifted or has artefac=ts. This might limit their applicability for individual glaciers or smaller= regions. If, however, high quality datasets are available that match to th=e acquisition date of the satellite images used to create the outlines, the=y can be used for both direct digitizing of glacier extents and/or creating= an independent reference dataset that can then be used for a =E2=80=98real==E2=80=99 validation (e.g. Paul et al. 2013, Andreassen et al. 2022). For t=he datasets created by C3S, we will report details of the validation datase=ts used along with the results of the uncertainty assessment in the PQAR [<=a href=3D"#GlacierAreadataversion7.0:ProductQualityAssuranceDocument(PQAD)-=rd">RD2].

3. Description of product validation methodology

As mentioned in Section 1, glacier outlines are in general not validated= (against a reference dataset), but instead an uncertainty assessment is pe=rformed. Uncertainty has three main sources:

(a) the geo-location uncertainty,
(b) the digitizing uncertainty and
=span>(c) the interpretation uncertai=nty.

The cause and consequences of these (and fur=ther) uncertainty sources are summarized in Table 1 and described =in detail in this section. Figure 2 illustrates how the measures t=o determine uncertainty are connected to the above sources of uncertainty a=nd some other details.

Table 1:Overview of= the uncertainty sources when digitizing glacier extents and their impact o=n glacier size.

As the geo-location uncertainty (a) has no direct impact on the derived gl=acier area (but large impacts when used together with other geocoded datase=ts), we focus here on the digitizing (b) and interpretation (c) uncertainti=es. Both are calculated using a range of methods listed below under points =(1) to (4). They are used by us in C3S as well as by the analysts that have= provided glacier outlines to the RGI. We have ranked the available methods= in terms of the effort required to perform the assessment from (1) low to =4 (high). The low-effort methods are:

(1) using an uncertainty value derived by= earlier, more detailed studies (e.g. Paul et al. 2013);
(2) the so-called 'buffer method'.

Method (1) typically assumes an overall uncertainty of the derived glacier= areas of 3 to 5% and method (2) calculates a potential minimum and maximum= area with a buffer of =C2=B1=C2=BD or =C2=B11 image pixel from the existin=g outlines. This buffer value has been derived from more detailed investiga=tions that placed outlines derived by various analysts on top of each other= to reveal the variability in interpretation (Figure 2).

Whereas the buffer method works well on individual glaciers, it overesti=mates uncertainties for glacier complexes when these are already split into= individual glaciers, as shared boundaries do not contribute to the uncerta=inty. Hence, internal ice divides should be removed before the buffer metho=d is applied. Apart from this, the method shows a strongly increasing uncer=tainty towards smaller glaciers, as the fraction of perimeter pixels increa=ses. As smaller glaciers produce larger uncertainties with this method, thi=s aspect needs to be considered when assessing small numbers of glaciers to=gether, as the higher uncertainty from the smaller glaciers can skew the me=an uncertainty of the whole sample. The same applies when the sample is dom=inated by larger glaciers, but then in the other direction as larger glacie=rs produce smaller uncertainties. Hence, the buffer method will give a real=istic mean value for a large sample, but for individual glaciers uncertaint=ies can be smaller or larger.

Two further methods are applied to determine uncertainties:

(3) independent multiple digitising of a =small glacier sample, and
(4) com=parison with glacier extents obtained from higher resolution datasets.

Whereas method (4) provides a measure of accuracy, method (3) provides the= digitizing uncertainty (b) when performed by the same person and the inter=pretation uncertainty (c) when applied by different persons to the same sam=ple of glaciers. Method (3) can be applied independently of validation data= and is likely the most accurate method to estimate uncertainty as it consi=ders the performance of the analyst(s) responsible for correcting the outli=nes and thus introducing the uncertainties. In Figure 2a we show a=n example of the digitizing uncertainty and in Figure 2b of the i=nterpretation uncertainty where five analysts corrected the same sample of =glaciers. As can be seen, the latter has a larger variability (and thus unc=ertainty) than the former. The digitizing was performed on the original 10 =m resolution Sentinel-2 images used for an updated alpine-wide glacier inve=ntory (Paul et al. 2020).

Figure 1:Methods for uncertainty assessment and how they are connected to =uncertainty sources.

a)
=bFigure 2:=Multiple digitizing experiment for a sample of glaciers in southern S=witzerland by a) the same analyst (each coloured line represents one round =of digitizing) and b) by five different analysts (all lines refer to their =respective third digitizing). Image width is 12.2 km, north is up (Copernic=us Sentinel data 2015).

Even when reference datasets are available f=or method (4), uncertainties can arise in the manual digitising, and the ar=ea value that is finally used to determine the accuracy is a mean value of =at least three (better five) independent digitisations. Figure 3 i=s showing an example of such a validation with a reference dataset. As this= requires considerable extra work (in particular for a larger sample of gla=ciers), method (4) has only rarely been used for accuracy assessment (e.g. =Andreassen et al. 2022, Fischer et al. 2014). On the other hand, method (3)= is increasingly used by the community (Fischer et al. 2014, Guo et al. 201=5, Paul et al. 2020). The specific method among the four methods presented =here that has been used for the datasets in the RGI might be determined fro=m the respective publications (but not all include uncertainty information)=. This effort has been made collectively for the paper describing the RGI (=Pfeffer et al. 2014) and we report in Section 4 on the results of this over=arching assessment.

=Figure 3: Accuracy assessment for a small glacier in the Swiss Alp=s. The white outline is derived from 30 m Landsat TM data (this is the leng=th of the shortest line segment), whereas the coloured outlines represent d=ifferent digitisations of the glacier extent by different analysts using th=e high-resolution (50 cm) aerial image shown in the background (North is at= top). Variability in the coloured outlines is, in general, not more than 1= Landsat pixel in width (image taken from Paul et al. 2013).

For individual glaciers, or even for entire =regions, much larger uncertainties are present in the RGI than those introd=uced by the'standard validation' when correcting regions in =shadow or ice under debris cover (see Table 1). Likely the largest= one, on a regional scale, refers to image conditions and results =from using scenes with adverse snow conditions for glacier mapping so that =snow rather than glacier ice is mapped, and glacier extent is largely overe=stimated (commission error). This is still a major issue for glacier outlin=es in the Andes (see Figure 4) and to a lesser extent in most othe=r regions. On a somewhat smaller scale, local clouds might cover the real g=lacier extent in all available images leading to a local underestimation (o=mission error) of glacier area, if not corrected. Ice and snow in shadow ca=n both be missed or wrongly added when image contrast is not sufficient. On= a local scale (individual glaciers), missed debris cover (omission error) =and inclusion of pro-glacial lakes (commission error) creates the largest u=ncertainties in glacier area. Andreassen et al. (2022) presents several exa=mples of the impacts of clouds, snow, shadow and lakes on glacier classific=ation. Apart from incorrectly mapped snow cover, the errors introduced by t=he other uncertainties might average out at a somewhat larger scale so that= the total glacier-covered area for a larger region might still be correct.=

Figure 4:=Wrong glacier areas in Bolivia due to adverse snow conditions in RGI6= (black lines). The red outlines mark the real (very small) glaciers. The b=ackground image is a subset from scene 3-71 acquired by Landsat 5 on 26.05.=1998 in false colours. Image width is 18.5 km, north is at top. Image: earthexplorer.usgs.gov URL resource last viewed 22^nd J=une 2023.

A final class of uncertainties is related to=methodological differences. These are difficult to quantify =as they are more differences of opinion rather than real errors or uncertai=nties, i.e. large differences in glacier area can occur without one of them= being wrong. Typical examples are related to the inclusion of rock glacier=s (that are often difficult to discriminate from debris-covered glaciers) o=r the interpretation of perennial snowfields (with possible ice underneath)= at high elevations in the Himalaya (near mountain crests) or at low elevat=ions in polar regions (in topographically protected niches). Considering th=ese features or not can regionally have an impact on the mapped total glaci=er area. In this regard, it is also important to consider that glacier inve=ntories are often created for a specific purpose. Whereas rock glaciers and= perennial snow fields might be considered for an overall hydrologic assess=ment (water resources), it would be better to exclude them when the purpose= is detection of climate change impacts or a strict glaciological assessmen=t (glaciers must flow by definition whereas rock glaciers creep). In view o=f the datasets provided to the CDS (the RGI 5.0 and 6.0), one has to be awa=re that the glacier outlines are a mix of all of the above. They include po=orly mapped regions (missing debris-covered glaciers, including lakes), out=lines including seasonal and perennial snow, as well as rock glaciers, miss=ed regions in shadow or under clouds, and glaciers missing their accumulati=on regions. The work in C3S will improve on these issues.

4. Summary of v=alidation results

In this section we provide results of an overarching accuracy assessment= for the baseline dataset (the RGI) provided to the CDS. This assessment su=mmarises the results from the individual studies that contributed datasets =to the global product (Pfeffer et al. 2014). The results obtained for the d=atasets created by C3S2 will be presented in the PQAR [RD2].

For the mapped glacier areas in the RGI, the increase in uncertainty tow=ards smaller glaciers is clearly visible in Figure 5.= The graph has been created from published estimates of uncertainty for sin=gle glaciers and glacier complexes. As described above, most studies have u=sed the buffer method for uncertainty assessment that tends to give too hig=h values for small glaciers. However, for glaciers larger than 1 km^{2=sup> uncertainties are in general <5%.}

Figure 5. Published estimates of the uncertainty of= area measurements of single glaciers (diamonds) and collections of glacier=s (dots). Solid line: best-fitting relationship between measured area and i=ts standard error. Dashed line: relationship adopted for estimation of RGI =errors (from Pfeffer et al. 2014).

Andreassen, L.M., Nagy, T., Kj=C3=B8llmoen, B., Leigh, J.R. (2022). An i=nventory of Norway's glaciers and ice-marginal lakes from 2018=E2=80=9319 S=entinel-2 data. Journal of Glaciology 1=E2=80=9322. https://doi.org/10.1017/jog.2022.20(last acc=essed 22nd June 2023)

Fischer, M., Huss, M., Barboux, C. and Hoelzle, M. (2014). The new Swiss= Glacier Inventory SGI2010: Relevance of using high- resolution source data= in areas dominated by very small glaciers,Arctic, Antarctic= and Alpine Research, 46, 935-947, DOI: 10.1657/1938-4246- 46.4.93=3

Guo, W.Q., S.Y. Liu, J.L. Xu, L.Z. Wu, D.H. Shangguan, X.J. Yao, J.F. We=i, W.J. Bao, P.C. Yu, Q. Liu and Z.L. Jiang (2015). The second Chinese glac=ier inventory: data, methods and results.Journal of Glaciology, 61(226), 357-372. DOI: 10.3189/2015JoG14J209

M=C3=BCller, R. (2014). Calibration and Verification of Remote Sensing I=nstruments and Observations. Remote Sensing, 6(6), 5692=E2=80=935695. DOI: =10.3390/rs6065692

Paul, F., N. Barrand, E. Berthier, T. Bolch, K. Casey, H., rey, S.P. Jos=hi, V. Konovalov, R. Le Bris, N. Mo=CC=88lg, G. Nosenko, C. Nuth, A. Pope, =A. Racoviteanu, P. Rastner, B. Raup, K. Scharrer, S. Steffen and S. Winsvol=d (2013): On the accuracy of glacier outlines derived from remote sensing d=ata.Annals of Glaciology, 54 (63), 171-182. DOI: 10.3189/201=3AoG63A296

Paul, F., Rastner, P., Azzoni, R.S., Diolaiuti, G., Fugazza, D., Le Bris=, R., Nemec, J., Rabatel, A., Ramusovic, M., Schwaizer, G., and Smiraglia, =C. (2020): Glacier shrinkage in the Alps continues unabated as revealed by =a new glacier inventory from Sentinel-2.Earth Systems Science Dat=a, 12(3), 1805-1821. DOI: 10.5194/essd-12-1805-2020

Pfeffer, W. T., Arendt, A. A., Bliss, A., Bolch, T., Cogley, J. G., Gard=ner, A.S., =E2=80=A6 Sharp, M. J. (2014). The Randolph Glacier Inventory: a= globally complete inventory of glaciers.Journal of Glaciology, 60(221), 537-552. DOI: 10.3189/2014JoG13J176

RGI Consortium (2017): RGI consortium: Randolph Glacier Inventory =E2=80==93 A Dataset of Global Glacier Outlines: Version 6.0, GLIMS Technical Repo=rt, 71 pp., available at:http://glims.org/R=GI/00_rgi60_TechnicalNote.pdf (last accessed 22^nd June 2023)=

Wan, Z., Zhang, Y., Zhang, Q. and Li, Z.-L. (2004). Quality assessment a=nd validation of the MODIS global land surface temperature. International J=ournal of Remote Sensing, 25(1), 261-274. DOI: 10.1080/0143116031000116417&=nbsp;

• Page:

  Glacier Area document= version 2023: Target Requirements and Gap Analysis Document (TRGAD))

• Page:

  Glacier Area data ver=sion 7.0: System Quality Assurance Document (SQAD)

• Page:

  Glacier Area data ver=sion 7.0: Product Quality Assurance Document (PQAD)

• Page:

  Glacier Area data ver=sion 7.0: Algorithm Theoretical Basis Document (ATBD)

• Page:

  Glacier Area data ver=sion 7.0: Product User Guide and Specification (PUGS)