Skip navigation.

Product Validation and Selection Report (PVASR) accepted

Printer-friendly versionSend by email

One of the crucial documents, which summarizes the validation of round robin datasets produced with the precursor algorithms participating in the Aerosol_cci project, was accepted.

The deliverable with corresponding attachments can be found under / documents / public / Accepted deliverable versions.

This document summarizes the validation of round robin (RR) datasets produced with the precursor algorithms participating in the Aerosol_cci project. Each of the precursor algorithms has been used in a series of experiments to test their sensitivities to the variation of key parameters (aerosol model and cloud screening; surface reflectance will be tested later) which have lead to significant improvements of most algorithms. Each retrieval group selected their best algorithm for participation in the RR analysis. Six of the seven nadir column aerosol optical depth (AOD) retrieval algorithms delivered four months of global data (March, June, September and December 2008) covering all seasons. The participating algorithms exploit data from the sensors AATSR (3 algorithms: ADV, ORAC, SU), MERIS (ESA standard algorithm), PARASOL (over ocean only) and synergetically AATSR+SCIAMACHY (SYNAER). One external dataset (MERIS ALAMO over ocean only) provided by one project partner but not included in the original project plan participated as well. Data from one algorithm contained in the original plan (MERIS BAER) were not delivered in time for consideration in the round robin data analysis but the coding problems seem to have been resolved and data will be delivered within a few months.
The round robin data sets were validated by comparison with AOD data provided by the AERONET ground-based sun photometer network using existing and further developed tools from AEROCOM / MetNo and LOA / ICARE. Level2 (orbit projection) and level3 (daily gridded) datasets were used in the quantification of validation statistics and scoring of spatial and temporal correlations. Additionally, global average maps of each dataset for the four months were assessed to understand coverage and capabilities to capture key features of the global aerosol distribution. Furthermore, the results were compared with datasets from other satellite instruments (MODIS, MISR) and with model results (AEROCOM median). Pixel uncertainties as specified in the first version of the uncertainty characterisation report have been provided for three of the algorithms and their evaluation has just started. The round robin analysis was not made for stratospheric extinction profiles or for the total column absorbing aerosol index (AAI). However, the stratospheric extinction profiles have been validated versus ground-based datasets.. For the AAI validation reference data do not exist – therefore, an algorithm is under development to transfer atmospheric model concentration fields of various aerosol types into comparable AAI fields.
The analysis of the various round robin datasets shows that algorithms for three instruments are mature enough for ECV production with a limited effort. These are a combination of precursor algorithms for AATSR over ocean and land including bright surfaces, and the algorithms over ocean for PARASOL and MERIS (ALAMO). Work is needed to optimize cloud masking / post-processing and to add pixel uncertainties to some of the algorithms and validate these uncertainties. The other algorithms (SYNAER, MERIS over land) are not yet ready for ECV production and more substantial work (in addition focusing on surface reflectance treatment over land) will be conducted to.
This first version of the product validation and algorithm selection report (PVASR1) describes the results achieved thus far. It has been iterated by the Aerosol_cci team to represent the current consensus on the status of the project’s achievements. The results and initial conclusions were discussed during a teleconference   of an expert team for MERIS / PARSOL and another one between experts for AATSR / SCIAMACHY). Subsequently the available results and preliminary conclusions were discussed by the two co-science leaders who drafted the first conclusions. This draft was then distributed to the entire team one week before the sixth progress meeting, where the validation results and draft conclusions were thoroughly discussed. The PVASR1 text has been finalized through a last iteration with the consortium after this progress meeting.
In conclusion it can be stated that most Aerosol_cci precursor algorithms have benefitted from the joint efforts and the results were significantly improved. Differences between the various algorithms have decreased. The accuracy of the best algorithms comes nearer to MODIS and MISR and is clearly better than the performance of earlier European algorithms for the same sensors (e.g. GLOBAEROSOL). All algorithms use the common CCI aerosol components which makes comparison and merging easier. Most algorithms need partly an aerosol type climatology as a priori or as prescription of one or two of the 3 fractions (fine/coarse, fine strongly absorbing / weakly absorbing, coarse dust / salt).
The role of OMI AAI with averaging kernel products needs definition and evaluation by model users and MERIS algorithm teams (absorption ancillary information). Use of stratospheric extinction profiles needs demonstration (integrated stratospheric column AOD maps or global average stratospheric AOD for “clean months” for correction of total column products).
Lessons learned from the round robin exercise are:
The team effort to understand sensitivities of algorithms, to improve the critical modules and to collaborate to harmonize the algorithms where feasible was highly successful as can be seen in the advanced validation statistics and monthly mean maps getting closer to those of MODIS and MISR. Such a team effort needs a spirit of open collaboration to foster a free exchange on problems and possible solutions, thus achieving a clear learning curve for all algorithms.
A round robin exercise for aerosol ECVs cannot be conducted using a fully automatic scoring since trade-offs between coverage and accuracy or between added value and accuracy need to be made. This requires scientific expertise and a team dialogue to come up with conclusions which meet the standards of peer review by    the scientific community. A strong user involvement in the whole validation and selection process is crucial to understand and take into account the user needs.
With each problem solved and sensitivity understood new open questions arise (e.g. assessing the benefit by use of aerosol type climatologies, trade-off between cloud masking and post-processing, need for use of common thresholds for numbers of cloud-free retrievals within a super pixel, need for common pixel filtering to strengthen statistical analysis, limitations of temporal scoring due to satellite product coverage but also AERONET coverage, open ocean background validation,…). Accordingly, the improvement of algorithms needs to continue and repeated round robin exercises are needed to document the learning curve and algorithm improvements. Continued refinement of the evaluation methodology is also required to better pinpoint and understand deficiencies of the retrieval products.
Harmonization of optical aerosol components between all algorithms was achieved. A common cloud masking needs further testing and trade-off of a common cloud mask versus post-processing as effective cloud contamination filtering needs to be made. Surface treatment harmonization seems feasible over ocean, but is sensor / algorithm specific over land.
Initial analysis shows that the quality flags in some of the products are meaningful to identify pixels with lower accuracy. Further evaluation is needed for the pixel uncertainty estimates and quality flags as well as for additional aerosol variables such as fine / coarse mode, absorption information, etc.