Sensitivity Analysis to Aid Model Certification: An Example on the Wetland Valuation Assessment Model Applied in Louisiana

Regardless of purpose or type, all models are limited by scientific understanding of the ecosystem being modeled, validity of input parameters, and ability of the model structure to capture relevant processes. Sensitivity analysis is a technique for systematically changing parameters in a model to determine the effects of such changes on model outcomes. It is an essential process for model building and quality assurance.

The USACE requires that planning models be coordinated for certification through the Ecosystem Restoration Planning Center of Expertise (ECO-PCX). The purpose of the review is to evaluate the scientific validity and technical quality of tools used for planning, and to ensure the tools conform to policy requirements and are usable. Sensitivity analyses can help address questions raised in the model certification process.

Sensitivity Analysis Applied for Ecosystem Restoration
USACE New Orleans District (CEMVN) is using a suite of community-based habitat suitability models developed by other agencies collectively titled Wetlands Value Assessment (WVA) for various projects in Louisiana. As part of the model certification process, reviewers recommended changes for two components of the WVA model structure. In response, sensitivity analyses were undertaken to examine the influence of the suggested changes on model results and subsequent decision making. Specific concerns related to:

1. the influence of suitability curve boundary conditions
2. the influence of aggregation techniques for combining habitat suitability curves into a Composite Habitat Suitability Index (HSI)

To address the comments, the WVA model's sensitivity to adjustments in the above factors was examined to determine if the proposed adjustments would affect the ranking of alternatives for the Barataria Basin Barrier Shoreline restoration project. Because the construct of the models did not lend themselves readily to systematic sensitivity analysis, ERDC scientists reformulated the models in order to provide greater flexibility and utility for analyses.

Boundary Conditions
ERDC scientists examined the sensitivity of three WVA models to changes in suitability curve intercepts.

In the WVA models, some suitability curves have non-zero y-intercepts, so the value of the habitat for fish and wildlife can never be zero. The reviewers suggested that HSI values should approach zero to indicate that quality is insufficient for the community as a whole for these circumstances. The three WVA models were examined for sensitivity to changes in these intercepts. Two scenarios were specifically evaluated, as shown in the accompanying figure and described as follows:

• One in which all intercepts are as reported in the WVA model documentation
• One in which the suitability index curves are forced through a near-zero intercept.

Figure 1. Suitability index curves as specified bu WVA (solid lines) and adjusted by ERDC (dashed lines) to address review comments. (a-d) saline marsh (SIV₁, SIV₂, SIV₄, SIV₆), (e-h) barrier headland (SIV₁, SIV₂, SIV₃, SIV₄), and (i-m) barrier island (SIV₁, SIV₂, SIV₃, SIV₄, SIV₅).

It is important to note that not all WVA parameters were examined as some were pictorial or categorical in nature and others provided maximum suitability at zero values (i.e., SIV = 1 at parameter = 0). The two intercept sensitivity scenarios that were evaluated as part of the effort reported here were intended to assess model responsiveness to the suggested changes, and in particular determine if those changes affected the outcome of the modeling effort in terms of alternative prioritization.

Aggregation Methods
Suitability indices can be combined in countless ways to generate a composite HSI (see USFWS 1981 for guidelines on HSI development). WVA uses an aggregation algorithm based upon a weighted arithmetic mean for two models (Barrier Headland and Barrier Island), and a combination of arithmetic and geometric means for the Saline Emergent and Open Water models. The sensitivity of the models to the aggregation method were evaluated by computing changes in model outcomes under four alternative aggregation scenarios:

• WVA-specified formula which contains weighting factors
• Geometric mean without weighting factors
• Arithmetic mean without weighting factors
• Harmonic mean without weighting factors

The arithmetic, geometric, and harmonic averaging scenarios do not capture the relative importance of model parameters as defined in the WVA algorithm. However, these scenarios provide an indication of the influence of different aggregation algorithms and the sensitivity of the model to these options. Due to complications arising from zero values in these aggregation schemes, an intercept of 10-10 was used to test the influence of a "zero intercept" and also maintain numerical continuity.

The motivation behind pursuing different aggregation methods is that geometric and harmonic means will more accurately reflect limiting factors for the HSI curve; thus, this test identified how small a value had to be to become a "limiting factor" which we assumed to be < 0.05. These near-zero intercepts are referred to as the zero-intercept condition.

Test Matrix
In order to test sensitivity to both changes in boundary conditions (i.e., intercepts) as well as aggregation techniques, we examined combinations of the two analyses as shown in Table 1.

Aggregation Technique	Non-zero intercept suitability curves	Zero intercept suitability curves
WVA-specified	WVA-i	WVA-0
Geometric mean	GEO-i	GEO-0
Arithmetic mean	ARI-i	ARI-0
Harmonic mean	HAR-i	HAR-0

Results
This sensitivity analysis provides insight into the response of the models relative to variation in the Y-intercept value for the HSIs and the method for aggregating model results. In terms of the overall magnitude of average annual habitat units (AAHUs), the WVA models were more sensitive to changes in aggregation method (average change in model results of 15.8%) than Y-intercepts of the suitability curves (average change in model results of 8.7%). Additionally, the individual models vary in sensitivity.

While the absolute value of these changes may be significant, in relative terms they're virtually inconsequential. Of 144 possible rankings of alternatives, only 20 were changed as a result of the eight intercept/aggregation combinations. In no case was the preferred alternative replaced by another alternative as a consequence of these analyses. Again, most of the changes in rankings were associated with the application of the harmonic mean, and the replacement of the WVA-intercept with a zero-intercept.

Sensitivity Analysis on Your Project
Regardless of purpose or function, all models are limited by scientific understanding of the process being modeled, validity of input parameters, and ability of the model structure to capture understood processes. As shown here, there is value in examining the sensitivity of a model to changes in one or all of these and how that sensitivity alters conclusions. While we recommend moving beyond sensitivity analysis and suggest accounting for uncertainty explicitly with models developed for that purpose, simple sensitivity analyses are helpful in almost any model appliction.

EBA Resources

• Modeling & Forecasting
• Decision Making
• Risk & Uncertainty Management