Sensitivity analysis

Sensitivity analysis (SA) is the study of how the variation (uncertainty) in the output of a mathematical model can be apportioned, qualitatively or quantitatively, to different sources of variation in the input of a model Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D. Saisana, M., and Tarantola, S., 2008, Global Sensitivity Analysis. The Primer, John Wiley & Sons.] .

In more general terms uncertainty and sensitivity analyses investigate the robustness of a study when the study includes some form of mathematical modelling. While uncertainty analysis studies the overall uncertainty in the conclusions of the study, sensitivity analysis tries to identify what source of uncertainty weights more on the study's conclusions. For example, several guidelines for modelling [http://www.epa.gov/crem/library/CREM%20Guidance%20Draft%2012_03.pdf (see e.g. one from the US EPA)] or for impact assessment [http://ec.europa.eu/governance/impact/docs/SEC2005_791_IA%20guidelines_annexes.pdf (see one from the European Commission)] prescribe sensitivity analysis as a tool to ensure the quality of the modelling/assessment.

The problem setting in sensitivity analysis has strong similarities with Design of experiments. In design of experiments one studies the effect of some process or intervention (the 'treatment') on some objects (the 'experimental units'). In sensitivity analysis one looks at the effect of varying the inputs of a mathematical model on the output of the model itself. In both disciplines one strives to obtain information from the system with a minimum of physical or numerical experiments.

In uncertainty and sensitivity analysis there is a crucial trade off between how scrupolous an analysist is in exploring the input assumptions and how wide the resulting inference may be. The point is well illustrated by the econometrician Edward E. Leamer (1990) [Leamer, E., (1990) Let's take the con out of econometrics, and Sensitivity analysis would help. In C. Granger (ed.), Modelling Economic Series. Oxford: Clarendon Press 1990.] :

I have proposed a form of organized sensitivity analysis that I call ‘global sensitivity analysis’ in which a neighborhood of alternative assumptions is selected and the corresponding interval of inferences is identified. Conclusions are judged to be sturdy only if the neighborhood of assumptions is wide enough to be credible and the corresponding interval of inferences is narrow enough to be useful.

Note Leamer’s emphasis is on the need for 'credibility' in the selection of assumptions. The easiest way to invalidate a model is to demonstrate it fragile with respect to the uncertainty in the assumptions or to show that its assumptions have not been taken 'wide enough'. The same concept is expressed by Jerome R. Ravetz, for whom bad modelling is when "uncertainties in inputs must be suppressed least outputs become indeterminate." [Ravetz, J.R., 2007, No-Nonsense Guide to Science, New Internationalist Publications Ltd.]

In modern econometrics the use of sensitivity analysis to anticipate criticism is the subject of one of the ten commandments of applied econometrics (from Kennedy, 2007 [Kennedy, P. (2007). A guide to econometrics, Fifth edition. Blackwell Publishing.] ):

Thou shall confess in the presence of sensitivity. Corollary: Thou shall anticipate criticism [···] When reporting a sensitivity analysis, researchers should explain fully their specification search so that the readers can judge for themselves how the results may have been affected. This is basically an ‘honesty is the best policy’ approach, advocated by Leamer, (1978 [Leamer, E. (1978). Specification Searches: Ad Hoc Inferences with Nonexperimental Data. John Wiley & Sons, Ltd, p. vi. ] ).

The use of mathematical modelling can be the subject of controversies, see Nassim Nicholas Taleb [Taleb, N. N., (2007) The Black Swan: The Impact of the Highly Improbable, Random House.] in Economics, and Orrin H. Pilkey and Linda Pilkey Jarvis [Pilkey, O. H. and L. Pilkey-Jarvis (2007), Useless Arithmetic. Why Environmental Scientists Can't Predict the Future. New York: Columbia University Press.] in Environmental Sciences. As noted by the latter Authors, this increases the relevance of sensitivity analysis in today's modelling practice .

Overview

Most mathematical problems met in social, economic or natural sciences entail the use of mathematical models, which are generally too complex for an easy appreciation of the relationship between input factors (what goes into the model) and output (the model’s dependent variables). Such an appreciation, i.e. the understanding of how the model behaves in response to changes in its inputs, is of fundamental importance to ensure a correct use of the models.

A mathematical model is defined by a series of equations, input factors, parameters, and variables aimed to characterize the process being investigated.

Input is subject to many sources of uncertainty including errors of measurement, absence of information and poor or partial understanding of the driving forces and mechanisms. This uncertainty imposes a limit on our confidence in the response or output of the model. Further, models may have to cope with the natural intrinsic variability of the system, such as the occurrence of stochastic events.

Good modeling practice requires that the modeler provides an evaluation of the confidence in the model, possibly assessing the uncertainties associated with the modeling process and with the outcome of the model itself. Uncertainty and Sensitivity Analysis offer valid tools for characterizing the uncertainty associated with a model. Uncertainty analysis (UA) quantifies the uncertainty in the outcome of a model. Sensitivity Analysis has the complementary role of ordering by importance the strength and relevance of the inputs in determining the variation in the output.

In models involving many input variables sensitivity analysis is an essential ingredient of model building and quality assurance. National and international agencies involved in impact assessment studies have included sections devoted to sensitivity analysis in their guidelines. Examples are the European Commission, the White House Office of Management and Budget, the Intergovernmental Panel on Climate Change and the US Environmental Protection Agency.

Methodology

There are several possible procedures to perform uncertainty (UA) and sensitivity analysis (SA). Important classes of methods are:

* Local methods, such as the simple derivative of the output $Y$ with respect to an input factor $X\_i$: $left|\; frac\{partial\; Y\}\{partial\; X\_i\}\; ight\; |\_\{\; extbf\; \{x\}^0\; \}$, where the subscript $extbf\; \{x\}^0$ indicates that the derivative is taken at some fixed point in the space of the input (hence the 'local' in the name of the class). Adjoint modelling [Cacuci, Dan G., Sensitivity and Uncertainty Analysis: Theory, Volume I, Chapman & Hall.] [Cacuci, Dan G., Mihaela Ionescu-Bujor, Michael Navon, 2005, Sensitivity And Uncertainty Analysis: Applications to Large-Scale Systems (Volume II), Chapman & Hall.] and Automated Differentiation [Grievank, A. (2000). Evaluating derivatives, Principles and techniques of algorithmic differentiation. SIAM publisher.] are methods in this class.
* sampling-based method [J.C. Helton, J.D. Johnson, C.J. Salaberry, and C.B. Storlie, 2006, Survey of sampling based methods for uncertainty and sensitivity analysis. Reliability Engineering and System Safety, 91:1175-1209.] A sampling-based sensitivity is one in which the model is executed repeatedly for combinations of values sampled from the distribution (assumed known) of the input factors. Once the sample is generated, several strategies (including simple input-output scatterplots) can be used to derive sensitivity measures for the factors.

* Methods based on emulators (e.g. Bayesian [Oakley, J. and A. O'Hagan (2004). Probabilistic sensitivity analysis of complex models: a Bayesian approach. J. Royal Stat. Soc. B 66, 751-769.] ). With these methods the value of the output $Y$, or directly the value of the sensitvity measure of a factor $X\_i$, is treated as a stochastic process and estimated from the available computer-generated data points. This is useful when the computer program which describes the model is expensive to run.
* Screening methods. This is a particular instance of sampling based methods. The objective here is to estimate a few active factors in models with many factors [Morris, M. D. (1991). Factorial sampling plans for preliminary computational experiments. Technometrics, 33, 161–174.] [Campolongo, F., J. Cariboni, and A. Saltelli (2007). An effective screening design for sensitivity analysis of large models. Environmental Modelling and Software, 22,1509-1518.] .
* Variance based methods [Sobol’, I. (1990). Sensitivity estimates for nonlinear mathematical models. Matematicheskoe Modelirovanie 2, 112–118. in Russian, translated in English in Sobol’ , I. (1993). Sensitivity analysis for non-linear mathematical models. Mathematical Modeling & Computational Experiment (Engl. Transl.), 1993, 1, 407–414. ] [Homma, T. and A. Saltelli (1996). Importance measures in global sensitivity analysis of nonlinear models. Reliability Engineering and System Safety, 52, 1–17.] [Saltelli, A., K. Chan, and M. Scott (Eds.) (2000). Sensitivity Analysis. Wiley Series in Probability and Statistics. New York: John Wiley and Sons.] . Here the unconditional variance $V(Y)$ of $Y$ is decomposed into terms due to individual factors plus terms due to interaction among factors. Full variance decompositions are only meaningfull when the input factors are independent from one another [Saltelli, A. and S. Tarantola (2002). On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal. Journal of American Statistical Association, 97, 702–709.] .
* High Dimensional Model Representations (HDMR) [Li, G., J. Hu, S.-W. Wang, P. Georgopoulos, J. Schoendorf, and H. Rabitz (2006). Random Sampling-High Dimensional Model Representation (RS-HDMR) and orthogonality of its different order component functions. Journal of Physical Chemistry A 110, 2474-2485.] [ Li, G., W. S. W., and R. H. (2002). Practical approaches to construct RS-HDMR component functions. Journal of Physical Chemistry 106, 8721{8733.] . The term is due to H. Rabitz [Rabitz, H. (1989). System analysis at molecular scale. Science, 246, 221–226.] and include as a particular case the variance based methods. In HDMR the output $Y$ is expresed as a linear combination of terms of increasing dimensionality.
* Methods based on Monte Carlo filtering [Hornberger, G. and R. Spear (1981). An approach to the preliminary analysis of environmental systems. Journal of Environmental Management 7, 7-18.] [Saltelli, A., S. Tarantola, F. Campolongo, and M. Ratto (2004). Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. John Wiley and Sons.] . These are also sampling-based and the objective here is to identify regions in the space of the input factors corresponding particular values (e.g. high or low) of the output.

Often (e.g. in sampling-based methods) UA and SA are performed jointly by executing the model repeatedly for combination of factor values sampled with some probability distribution. The following steps can be listed:

*Specify the target function of interest.
**It is easier to communicate the results of a sensitivity analysis when the target of interest has a direct relation to the problem tackled by the model.
*Assign a probability density function to the selected factors.
**When this involves eliciting experts' opinion this is the most expensive and time consuming part of the analysis.
*Generate a matrix of inputs with that distribution(s) through an appropriate design.
** As in experimental design, a good design for numerical experiments [Sacks, J., W. J. Welch, T. J. Mitchell, and H. P. Wynn (1989). Design and analysis of computer experiments. Statistical Science 4, 409-435.] should give a maximum of effects with a minimum of computed points.
*Evaluate the model and compute the distribution of the target function.
**This is the computer-time intensive step.
*Select a method for assessing the influence or relative importance of each input factor on the target function.
**This depends upon he purpose of the analysis, e.g. model simplification, factor prioritization, uncertainty reduction, etc.

Applications

Sensitivity Analysis can be used

* To simplify models

* To investigate the robustness of the model predictions

* To play what-if analysis exploring the impact of varying input assumptions and scenarios

* As an element of quality assurance (unexpected factors sensitivities may be associated to coding errors or misspecifications).

It provides as well information on:

* Factors that mostly contribute to the output variability

* The region in the space of input factors for which the model output is either maximum or minimum or within pre-defined bounds (see Monte Carlo filtering above)

* Optimal - or instability - regions within the space of factors for use in a subsequent calibration study

* Interaction between factors

Sensitivity Analysis is common in physics and chemistry [Saltelli, A., M. Ratto, S. Tarantola and F. Campolongo (2005) Sensitivity Analysis for Chemical Models, Chemical Reviews, 105(7) pp 2811 – 2828.] , in financial applications, risk analysis, signal processing, neural networks and any area where models are developed. Sensitivity analysis can also be used in model-based [http://www.modeling.uga.edu/tauc/background_material/Washington-Main.pdf policy assessment studies ] . Sensitivity analysis can be used to assess the robustness of [http://composite-indicators.jrc.ec.europa.eu/ composite indicators ] [Saisana M., Saltelli A., Tarantola S., 2005, Uncertainty and Sensitivity analysis techniques as tools for the quality assessment of composite indicators, Journal Royal Statistical Society A, 168 (2), 307-323.] , also know as indices, such as the [http://epi.yale.edu/Home Environmental Pressure Index] .

Errors

In sensitivity analysis Type I error is assessing as important a non important factor, and Type II error assessing as non important an important factor. Type III error corresponds to analysing the wrong problem, e.g. via an incorrect specification of the input uncertainties. Possible pitfalls in sensitivity analysis are:

* Unclear purpose of the analysis. Different statistical tests and measures are applied to the problem and different factors rankings are obtained. The test should instead be tailored to the purpose of the analysis, e.g. one uses Monte Carlo filtering if one is interested in which factors are most responsible for generating high/low values of the output.

* Too many model outputs are considered. This may be acceptable for quality assurance of sub-models but should be avoided when presenting the results of the overall analysis.

* Piecewise sensitivity. This is when one performs sensitivity analysis on one sub-model at a time. This approach is non conservative as it might overlook interactions among factors in different sub-models (Type II error).

Environmental

Computer environmental models are increasingly used in a wide variety of studies and applications.For example global climate model are used for both short term weather forecasts and long term climate change.

Moreover, computer models are increasingly used for environmental decision making at a local scale, for example for assessing the impact of a waste water treatment plant on a river flow, or for assessing the behavior and life length of bio-filters for contaminated waste water.

In both cases sensitivity analysis may help understanding the contribution of the various sources of uncertainty to the model output uncertainty and system performance in general.In these cases, depending on model complexity, different sampling strategies may be advisable and traditional sensitivity indexes have to be generalized to cover [http://www.iemss.org/iemss2006/papers/s7/268_Fasso_0.pdf multivariate sensitivity analysis] , heteroskedastic effects and correlated inputs.

Business

In a decision problem, the analyst may want to identify cost drivers as well as other quantities for which we need to acquire better knowledge in order to make an informed decision. On the other hand, some quantities have no influence on the predictions, so that we can save resources at no loss in accuracy by relaxing some of the conditions. See .Sensitivity analysis can help in a variety of other circumstances which can be handled by the settings illustrated below:
* to identify critical assumptions or compare alternative model structures
* guide future data collections
* detect important criteria
* optimize the tolerance of manufactured parts in terms of the uncertainty in the parameters
* optimize resources allocation
* model simplification or model lumping, etc.However there are also some problems associated with sensitivity analysis in the business context:
* Variables are often interdependent, which makes examining them each individually unrealistic, e.g.: changing one factor such as sales volume, will most likely affect other factors such as the selling price.
* Often the assumptions upon which the analysis is based are made by using past experience/data which may not hold in the future.
* Assigning a maximum and minimum (or optimistic and pessimistic) value is open to subjective interpretation. For instance one persons 'optimistic' forecast may be more conservative than that of another person performing a different part of the analysis. This sort of subjectivity can adversely affect the accuracy and overall objectivity of the analysis.

ee also

* Perturbation analysis
* Robustification

References

Bibliography

*Fassò A. (2007) Statistical sensitivity analysis and water quality. In Wymer L. Ed, Statistical Framework for Water Quality Criteria and Monitoring. Wiley, New York.
*Fassò A., Esposito E., Porcu E., Reverberi A.P., Vegliò F. (2003) Statistical Sensitivity Analysis of Packed Column Reactors for Contaminated Wastewater. Environmetrics. Vol. 14, n.8, 743 - 759.
*Fassò A., Perri P.F. (2002) Sensitivity Analysis. In Abdel H. El-Shaarawi and Walter W. Piegorsch (eds) Encyclopedia of Environmetrics, Volume 4, pp 1968–1982, Wiley.
*Saltelli, A., S. Tarantola, and K. Chan (1999). Quantitative model-independent method for global sensitivity analysis of model output. Technometrics 41(1), 39–56.
*Santner, T. J.; Williams, B. J.; Notz, W.I. Design and Analysis of Computer Experiments; Springer-Verlag, 2003.

pecial issues

* A special issue devoted to sensitivity analysis has been published in September 2008 on the [http://www3.interscience.wiley.com/journal/121401979/issue International Journal of Chemical Kinetics] .

* Another will appear in 2008 on Reliability Engineering and System Safety (RESS).

Both are selection of papers presented at the 2007 Conference of Sensitivity Analysis of Model Output (SAMO) held in Budapest in June. See [http://samo2007.chem.elte.hu/ SAMO 2007] for the slides of the presentations.

* An earlier special issue on sensitivity analysis has been published in 2006 on the journal [http://www.elsevier.com/wps/find/journaldescription.cws_home/405908/description#description Reliability Engineering and System Safety (Volume 91, 2006)] .

External links

* [http://sensitivity-analysis.jrc.ec.europa.eu/ A forum on sensitivity analysis] Main source - includes a tutorial and a bibliography.
* [http://www.nusap.net/ The NUSAP web site] Discusses the use of pedigree for scientific information (including use of models).
* [http://simlab.jrc.ec.europa.eu/ The SIMLAB software for sensitivity analysis: download it for free]
* [http://www.princeton.edu/~hrabitz/ Home page of Hershel A. Rabitz at Princeton.] Works on High Dimensional Model Representation (HDMR), used in sensitivity analysis.
* [http://home.ubalt.edu/ntsbarsh/senanaly/SenAnaly.htm Sensitivity Analysis Index]
* [http://sensitivity.r-forge.r-project.org/ The R software package "sensitivity"]
* [http://library.lanl.gov/ccw/samo2004/ SAMO 2004] , Proceedings of the 4th International Conference on Sensitivity Analysis of Model Output, Santa Fe, New Mexico, March 8-11, 2004. Edited by Kenneth M. Hanson and François M. Hemez