Dr. George A. Johnson, CFPIM
Professor of Operations Management, RIT, Retired
A company presently evaluates the performance of
its forecasters against a plus/minus 15% standard. That is, if actual results are not more than
15% above or below a forecast, the forecast is rated as acceptable. There are concerns about this approach, since
the results of some errors clearly are more serious than others, even if the
errors are of the same magnitude. Is
there a better way to establish accuracy standards?
There is a better way to view the situation and
it involves conceiving the issue as one of process control and
improvement. The explanation is
something like this.
Forecasting is a process. Its outputs typically are numbers. Some of the numbers represent expected future
demand (or maybe future shipments); other numbers represent errors, the inability
of the forecasting “model” to make perfect estimates. In this situation we are interested primarily
in the second kind of output, forecast error.
Dr. Deming noted there are two important kinds
of variation in process results and that it is crucial to understand which kind
is being dealt with before taking any corrective action. One type of variation is that which has
“common” causes. It is the largely
unexplainable “background noise” which is present when a process performs the
best it can with its current resources, under normal operating conditions.
Under normal conditions my full-size station
wagon will get from 16-18 MPG on the highway, with the variation resulting from
such typical, essentially random factors as changes in road pitch, elevation,
outside temperature, wind conditions and traffic. To perform significantly
better (less variably) than the typical 16-18 MPG, the vehicle’s (i.e.,
process) characteristics would have to be changed in some major way which
requires an owner’s investment. For
example, the axle ratio might have to be changed or cruise control installed.
The second kind of variation results from
“special” or “assignable” causes. It is
variation for which there is a known or knowable reason. If the highway performance of my wagon
suddenly drops from the 16-18 MPG range to 10 MPG, I should investigate the
unexpected change for its cause. Perhaps
I find that this degradation is associated with a new and unusual operating
circumstance, the strong smell of gasoline.
Inspection reveals that a liquid is escaping from the top of the fuel
pump while the engine runs. The fuel
pump has failed, partially, resulting in a leak. This additional operating factor explains the
extra fuel use which shows up as degraded MPG performance. Replacing the pump, i.e., removing the
“special” cause of variation will restore the performance to a stable 16-18
MPG, where only “common” sources of variation are present.
Returning now to the original question, forecast
accuracy standards, the plus/minus 15% standard has been set arbitrarily. It
seems like a reasonable allowance for error, but is it a rational criterion for
accuracy? Let’s explore the issue using
the example above.
After extensive experience with my station wagon
I have gathered quite a bit of data about its performance. Under normal conditions it averages 17
miles per gallon on the road and has never been observed to go above 18 or
below 16 MPG. One could say the expected limits of highway performance are 17
MPG plus/minus 1 MPG. These are the
typical results when the car is properly maintained and driven. If I am asked to forecast the average highway
MPG for my next trip, I will estimate 17.
I will not be surprised, however, if it actually turns out to be
anywhere in the interval 16-18 MPG. Now,
how does this knowledge play out against the seemingly reasonable but arbitrary
plus/minus 15% standard?
The 15% standard translates to .15 x 17 or 2.55
MPG. Thus, “management” will be happy if
the actual highway MPG is in the range 17 plus/minus 2.55 MPG or 14.45 - 19.55
MPG. When the fuel pump failed, the
highway MPG dropped to 10 MPG. Clearly
this was outside both the expected range (16-18 MPG) and the 15% standard range
(14.45-19.55 MPG) of results. The
forecaster was wrong! He should be
admonished for poor forecast accuracy.
But wait, the forecaster had no control over the failure of the fuel
pump. It was a “special” cause of variation; a one-time event. Fix the special cause and performance should
return to normal. It was good to be able
to detect the abnormality, but why criticize the forecaster? It wasn’t his fault.
Consider another scenario. Prior to a second trip I forecast 17 MPG,
just as above. At trip end, I compute
that the actual MPG is 15. How was my
forecast accuracy? By the 15% standard,
it was OK. My forecast was
accurate. I get a pat on the back. But in the recesses of my brain I know that
something isn’t quite right. The trip
MPG wasn’t in the 16-18 range. A little
checking reveals that I unknowingly bought some inferior gasoline at the last
stop and this explains the result. This
is another example of a “special” cause of variation affecting the process
outcome. (I’ll keep this one to myself,
though, since management thinks everything is OK.)
In still another scenario, let’s assume that I
have a really tired, old car, not as heavy as the station wagon. It also has given me an average of 17 MPG on
the road and has never gone above 20 nor below 14 MPG under normal operating
conditions. Thus, the normal range of
highway MPG is 14-20 with an average of 17.
I forecast 17 MPG for my next trip with this car. At trip end I compute the MPG actually is
14. I’m not surprised, because it has
done this before under typical operating conditions (i.e., “common”
variation).
I’m OK with this result but “management” is
not. The actual trip MPG was below the
limit of 14.45. I’m going to get
criticized. But wait a minute. I
operated the car correctly -- did what I was supposed to do -- and the outcome
was the result of “common” causes of variation.
It’s not fair that I get criticized.
The car and I were doing the best we could do under the circumstances.
After being unreasonably criticized a few times,
I start to get motivated to coast down hills, sneak a little extra gas into the
tank, and play the system in other ways so I don’t get penalized for things I
can’t control. This is not the best way
to use my time (or company resources), but it will keep me out of trouble.
I think the message is clear. A well-intended but arbitrary standard
usually does not correlate well with how a process actually functions under
typical conditions. As a result, the
standards do not help highlight the right things and often give very misleading
impressions. These false impressions can
then trigger “management” actions which actually make things worse. A better approach is based in the theory and
tools of quality control and improvement.
Instead of emphasizing standards, the main focus
should be on when and how to improve the forecasting process as measured by
forecast error. Improvement should be immediately pursued whenever an
unexpected event (i.e., “special” cause), pushes a forecast result out of its
typical range of variation. The root
cause should be uncovered and corrective action taken to eliminate or correct
it.
Example (1):
The special event is a one-time, non-recurring order from outside the
set of established customers. It will
never occur again. In this case, it
probably is best to ignore the demand from this order for purposes of future
forecasts.
Example (2):
The event is the bankruptcy and permanent cessation of business by a
major customer. While this will not
occur again, the effect is lasting. In
this case, the permanent reduction in expected demand should be taken into
account in future forecasts. This may
require that the forecast model be permanently reset to a new, lower level
prior to the next forecasting cycle.
Since a well-run forecasting process should not
have been surprised by the bankruptcy event, there is room for
improvement. The possibility of the
event should have been picked up by marketing intelligence, by the accounts
receivable system sounding an alert, or even by finance keeping an eye on this
major customer’s “health” via public reports and stock market figures. This is an example of why forecasting by team
usually is more effective than that by an individual and these sources of
information should be systematically integrated into the forecasting process
for the future.
Many companies have large numbers of items to
forecast, so many that it is effectively impossible to monitor all of them,
manually. Computers and statistics to
the rescue! Exceptions can be detected
statistically and brought to the attention of forecasters for
investigation. In a statistical sense,
an exception is detected by comparing a given cycle’s forecast error for an
item against its typical forecast error from the past (only “common” cause
variation present). This is how
statistical quality control charts work.
If the current cycle’s error is significantly larger or smaller than
that which is expected, a “flag” is raised.
The item’s forecast is then subject to investigation for the root cause
of the exceptional error.
Once the output of a forecasting process is
stable, i.e., free of “special” cause variation, there may be other reasons to
improve the process. In one case, it may
be that the amount of “common”-cause-only variation which remains is quite
large. This error forces the company to
maintain a big investment in safety stock and/or to sacrifice some degree of
customer service and/or to reduce operating efficiency to meet its
objectives. In another case it may be
that a noticeable pattern, perhaps seasonal, has developed in the
“common”-cause-only error. This would be
an opportunity to explain more of the variation — to remove it from the error
variation and make it part of the demand forecast -- by using a more complete
model.
Quality improvement tools such as check sheets,
Pareto analysis, cause and effect diagrams, flow charts, scatter diagrams, run
charts, regression analysis, etc., are useful for these kinds of process
improvement projects, even though the application is outside the traditional
quality field. It is easy and logical to
portray forecast results as products of a systematic process just as physical
products are the results of production processes. Hence, quality concepts and tools apply.
Now let’s close the loop to the original
standards issue. Holding forecasters to
an arbitrary standard is illogical and a lose-lose situation. Doing so can put more variation into the
situation than if the standard did not exist in the first place (what Deming
refers to as “tampering”). Ratees may
change their behavior arbitrarily trying to live with an invalid standards
system and/or management may make uninformed or misinformed and, therefore,
inappropriate investments in the process, e.g., new software, new forecasters,
new organization structure, etc.
It is far better to measure how the forecasting
system actually performs statistically, to understand its inherent variation
and be able to detect exceptions.
Efforts to improve forecast accuracy, i.e., reduce error, can then be
directed to the right places for the right reasons using the most appropriate
tools and methods. Over time, forecast
accuracy should improve without reference to any standards and this will pay
off in improved customer service, lower operating expense and lower inventory
investment, the real objectives.
Beck, J., “SPC and Selectable Calendars Work
Magic for GE Aircraft Engines’ Service Parts Operation,” 1995 Annual
International Conference Proceedings, APICS, pp. 477-481.
(Describes the use of
several SPC tools for monitoring forecasts.)
Gitlow, H.; Oppenheim, A.; Oppenheim, R., Quality
Management: Tools and Methods for
Improvement, Second Edition, Irwin, 1995.
(See, especially, the
discussion of Deming’s funnel experiment in Chapter 14, which illustrates the
concept of “tampering” with processes.)
Ishikawa, Kaoru, Guide to Quality Control. Asian Productivity Organization, 1986.
Available from GOAL/QPC, Methuen, MA.
Lin, W.;
Nolan, T.; Provost, L., “Understanding
Variation,” Quality Progress, Vol. 23, No. 5, May 1990, pp. 70-78.
(Excellent discussion of
common vs. special causes of variation and the implications for management of
processes, systems and people. If you
can read but one article, read this one.)