Subjective Uncertainty

As a cost analyst, one of the biggest hurdles to overcome is the lack of data available and minimal amount of documentation provided. How can a cost analyst estimate the cost of a project or system when requirements, historical costs, and other data availability are limited? Gathering uncertainty data is no exception when it comes to this challenge in the data collection process. Uncertainty exists within every cost estimate; however, the application of sound modeling techniques to quantify the uncertainty will aid decision makers in making a more informed decision. When developing cost estimates, it is advisable to use primary data sources, but what do you do when such resources are not available or are completely exhausted?

Solving Subjective Situations

While subject matter experts (SMEs) are not a primary source for information, they are full of working knowledge on their respective programs or systems. Therefore, in the absence of primary sources, they provide valuable insight for cost estimating purposes and become sufficient resources to determine cost boundaries surrounding specific data points. For example, if a cost analyst is provided a high, low, and most likely value, the analyst can apply the triangular distribution which assumes a probability of zero on values outside of the high and low range. The tip of the triangle correlates to the most likely value. When only the high and low values are known, but there is not a determined most likely value, the uniform distribution is appropriate to apply and assumes all values within this range have an equal likelihood of occurring.           While utilizing SME input can be valuable to determine distributions such as the ones stated above, this resource comes with a trade-off- subjective bias. When data is collected through imperfect and flawed human beings, the data will follow suit and therefore be subject to bias. While this is a large obstacle, all is not lost! When eliciting cost bounds, there are techniques one can use to mitigate the impacts of the bias to increase the validity of the cost estimate once uncertainty is appropriately modeled. This is a prime example of how cost estimating is as much of an art as it is a science and where logical math encounters experienced intuition. The easiest and most common cost bounds to obtain from SMEs are the high and low values of a cost element, so let’s start there with techniques to minimize the bias blow derived from subjective data.

  1. Allow the SME to provide the high and low values before providing the most likely value. This tactic lessens the bias known as anchoring, where a SME feels bound to their initial estimate and does not give appropriate attention on how high costs might actually be.
  2. Assume a SME provided range covers 70% of the true bounds. According to the Joint Cost Schedule Risk & Uncertainty Handbook, when there is lack of other supporting data, assuming the SME provided only 70% of the true range is the appropriate approach. This can lessen the bias known as aversion, when someone provides a smaller range in order to appear as more of an expert in their respective field.
  3. Be aware of recent, highly publicized news. People have a tendency to concentrate on items that most easily come to mind and areas of high publicity. This bias of availability, speaking of what comes most easily to the mind, prompts SMEs to perhaps neglect areas that are relevant to their specific programs. Keeping the dialogue flowing on all aspects of the program ensures all facets are being fairly evaluated and analyzed, not simply the ones that most easily come to the mind of the SME.
  4. Utilize multiple experts. This allows the analyst to obtain different points of view and expertise and therefore avoids weighting one person’s opinion/risk range too heavily and decreasing subjective contribution error.
  5. Present the results of the elicitation back to the SMEs. Display the effects of the uncertainty range back to the SME, preferably through visual aids (S-curve, Tornado Chart, etc.), to ensure cost reasonableness and accuracy. This presents the SME with context of their inputs and links their contribution on a specific cost element with a holistic end result.

          Modeling cost uncertainty within a cost estimate involves the application of art and science. A detailed analysis on the program’s risks results in a more valuable estimate that provides a range of possible cost outcomes with their associated probabilities. The 5 techniques listed above provide a solid reference point for modeling cost uncertainty ranges when the cost analyst must rely on subjective data provided by a SME. To learn more about Cask’s Strategy & Decision Support Services, please view the overview page.