Wednesday, April 23, 2008

A ranking of WSJ forecasters

For a number of years, The Wall Street Journal has been conducting a survey of economic forecasts. The newspaper is kind enough to publish the prediction of each forecaster, so we can entertain ourselves observing how they fare.

I restrict my attention to quarterly forecasts of GDP growth. Between 2004 and 2007, 27% of the predictions were within 0.5 percentage points of the actual outcome (see Table 1), whereas 56% (27% + 29%) were within one percentage point. Or, if you’re a member of the empty-glass club, 44% missed the target by more than one point.

Table 1. Distribution of accuracy of forecasts (2004-2007)

Forecasts within...
Percentage
0.5 percentage points (p.p.) of actual value
27.2
0.5 - 1 p.p.
28.9
1 - 1.5 p.p.
22.6
1.5 - 2 p.p.
13.7
2 - 2.5 p.p.
5.2
over 2.5 p.p.
2.4
All
100
Source: WSJ's survey of forecasters and author's calculations

To pit forecasters against each other I use the Root Mean Squared Error (RMSE), a one-number summary of the deviations of several forecasts. The RMSE punishes both positive and negative deviations equally, but penalizes big errors proportionally more than small ones*. I also can use it to form confidence intervals.

According to the RMSE measure, the most accurate forecaster is Gary Thayer, of the firm A.G. Edwards, although he no longer participates in the survey. The second most-accurate forecaster, and still in the panel, is Gene Huang of FedEx. (See Table 2.) The best forecaster is able to predict GDP growth within 1.67 percentage points, at a 90% level of confidence. (That means that if he posted 100 forecasts, 90 of them would deviate from the actual GDP growth rate by plus or minus 1.67 percentage points.)

Table 2. Top-20 WSJ forecasters, by Root Mean Squared Error (RMSE)

Rank
Forecaster

Firm
RMSE
Forecasts'
90% confidence
margin (p.p.)
1
Gary Thayer*
A.G. Edwards
0.95
1.67
2
Gene Huang
FedEx Corp.
0.98
1.72
3
David Resler
Nomura Securities International
1.02
1.79
4
Stuart Hoffman*
PNC Financial Services Group
1.03
1.82
5
Allen Sinai
Decision Economics Inc.
1.05
1.83
--
Median forecast
--
1.05
1.83
6
Mike Cosgrove
Econoclast
1.05
1.84
7
Nicholas S. Perna
Perna Associates
1.05
1.85
8
Dana Johnson
Comerica Bank
1.06
1.88
9
J. Prakken and C. Varvares
Macroeconomic Advisers
1.06
1.86
10
R. Berner and D. Greenlaw*
Morgan Stanley
1.07
1.87
11
Nairmen Behravesh
Global Insight
1.09
1.90
12
Robert DiClemente*
Citibank SSB
1.09
1.92
13
John Lonski
Moody's Investors Service
1.09
1.91
14
Scott Anderson
Wells Fargo & Co.
1.11
1.97
15
Douglas Duncan
Mortgage Bankers Association
1.12
1.97
16
David Rosenberg
Merrill Lynch
1.13
1.97
17
Diane Swonk
Mesirow Financial
1.13
1.99
18
David Lereah*
National Association of Realtors
1.13
1.99
19
Neal Soss
CSFB
1.15
2.01
20
Paul Kasriel
The Northern Trust
1.15
2.02
Source: WSJ's survey of forecasters and author's calculations.
*Not in WSJ group of forecasts anymore, as of November 2007.


It is well known that, over time, a group’s forecast is closer to the mark than almost any particular individual’s. Among the WSJ panel it’s no different: the median forecast is sixth in the ranking, out of 47. The same conclusion applies to the average forecast (average and median are very close to the each other in every release of the WSJ survey).

The top participants in the group hold but a tiny advantage over the rest. Even the 20th most accurate person has a margin of error of just over 2 percentage points, versus 1.67 points for the top forecaster. It’s not surprising then that rankings tend to change frequently. For example, at the end of 2006 the top five forecasters were (latest ranking in parentheses): Thayer (1), Rosenberg (17), Perna (8), Sinai (5) and Lonski (14).

Catching a “hot streak” seems to be exceedingly difficult too. Suppose that we define “winning” as being among the 50% most-accurate accurate forecasts for a given quarter. (A rather modest victory, may I say.) By that measure, only 37% of successes were followed by a second win, 31% of two-in-a-row’s were followed by a third success, and just 17% of those were followed by a fourth one.

Can a simple predictor outperform the pros? Michael Bryan of the Federal Reserve of Cleveland, whose commentary I follow in this post, asks that question. He compares the predictions in the Survey of Professional Forecasters (SPF) with the naïve forecast that next period’s outcome will be the same as the latest observed outcome. In terms of my data, that is the prediction that GDP growth in, say, 2008:Q1 will be the same as in 2007:Q4.

Bryan finds that 53% of economists made worse predictions than the naïve forecast. The WSJ panel shows much better marksmanship. All of them performed better than the naïve forecast, except one. (The exception is James F. Smith of Western Carolina University, and by a long shot. His RMSE is 2.83, whereas that of the naïve forecast is 1.89. Compare with the values in Table 2.)

By Clay Bennett
In five days we will have an advance estimate of how much the economy grew during the first quarter. The naïve forecast indicates 0.6 percentage points. The median WSJ forecast in April is exactly zero —neither cold nor hot, as a friend of mine likes to say. Gene Huang, the top forecaster of the hour, says it will be 0.8%. Which one will be closer to the mark?









* For the ranking of forecasters by RMSE, I include all the participants in the survey who submitted at least ten GDP forecasts between May of 2004 and December of 2007. I only include the predictions submitted at the beginning of the months of February, May, August and November for quarters Q1, Q2, Q3 and Q4, respectively. “Actual” GDP growth is taken to be the advance estimate, released one month after the end of the corresponding quarter. Given the timing of the forecasts and of the advance releases of GDP growth, each forecast appeared about three months before the actual outcome was known.

Subscribe to EconWeekly


(Clicking that button gives $0.10 to Francisco. Join tipjoy! How does tipping work?)

Technorati tags
, , , ,


3 comments :

Stan Jonas said...

Much more interesting I think...rather than GDP is the much more important... what is the FED going to do about it...

Here there is remarkable consistency... There is virtual total agreement that the FED will stop EASING by June ... given their move in JUNE will be on hold for the rest of the YEAR..

This is probably a better indicator than their real GDP forecasts ...

Ivan Kitov said...

There is a big difference between current forecasts from the WSJ forecasters and the naive one. The latter was made 3 months ago, and should be compared to contemporary forecasts, not with the current ones. This is the procedure.
It was checked many times (actually Atkeson and Ohanian published a paper in 2001
Atkeson, A., and L.E. Ohanian. 2001. "Are Phillips Curves Useful for Forecasting Inflation?" FRB Minneapolis Quarterly Review (Winter) pp. 2-11. http://www.mpls.frb.org/research/qr/qr2511.html

) and always showed that the naive one forecast at one quarter horizon is very hard to beat.

Francisco said...

Ivan Kitov:

As I explain in the post:

"I only include the predictions submitted at the beginning of the months of February, May, August and November for quarters Q1, Q2, Q3 and Q4, respectively."

So the prediction for, say, Q1 is the one submitted in the first week of February. The advance estimate for Q4, which is the naive forecast for Q1, is released on January 30, and is therefore roughly contemporary with the WSJ forecasters' predictions.