Aggregating Forecasts

POLS 3220: How to Predict the Future

Motivating Problem

  • Suppose the class is trying to predict whether the Federal Reserve will cut interest rates at its next meeting.

  • Three students submit briefings, and all three say 75%.

  • What should you predict after observing those three forecasts?

  • Previously, we considered taking the median forecast (i.e. majority rule).

    • But that strategy required some strong assumptions.
    • Are those assumptions met here?

Today’s Agenda

  • How best to aggregate individual forecasts into a group forecast?

  • I’ll try to convince you that, if three individual forecasts are at 75%, the best aggregation is probably >75%.

    • This is called extremizing—increasing the confidence of the group forecast when multiple individuals all make a prediction in the same direction.
  • We’ll motivate this idea from the perspective of Bayesian updating.

Warmup

  • Suppose you’re writing a briefing about the interest rates question.

  • At first, you have no idea what will happen (prior odds = 1:1).

  • Then you find a statement from a Fed official expressing concern about unemployment.

  • Historically, he makes statements like this three times more often before an interest rate cut than he does otherwise.

  • Problem: What should your new forecast be, after observing this evidence?

Warmup

Remember Bayes Rule:

\[ \text{Posterior Odds} = \text{Prior Odds} \times \text{Strength of Evidence} \]

Warmup

Remember Bayes Rule:

\[ \underbrace{\text{Posterior Odds}}_{\text{New Forecast}} = \underbrace{\text{Prior Odds}}_{\text{Old Forecast}}\times \underbrace{\text{Strength of Evidence}}_{\text{Likelihood Ratio}} \]

  • Prior Odds: 1:1
  • Likelihood Ratio: 3:1
  • Posterior Odds: 3:1
  • New Forecast: 75%

Bayesian Updating

  • Now suppose you read that JP Morgan’s analysts are predicting an interest rate cut.

  • Historically, this appears to have the same strength of evidence as the Fed official’s statements.

  • Bayes Rule handles this problem exactly like the first one!

    • Prior Odds: 3:1
    • Likelihood Ratio: 3:1
    • Posterior Odds: \(3:1 \times 3:1 = 9:1\)
    • New Prediction: 90%

Bayesian Updating

  • This process of Bayesian updating can proceed indefinitely.

  • For each piece of evidence:

    • Assess its likelihood ratio—conditional on everything you already know, how much more likely are you to observe this evidence if the event happens?
    • Adjust your prediction up or down using Bayes Rule.

Back to the Motivating Problem

  • Consider the three students writing briefings. All three start with prior odds 1:1.

  • Student A only finds the statement by the Fed official (3:1 likelihood ratio).

  • Student B finds the JP Morgan report (3:1 likelihood ratio).

  • Student C finds a third piece of evidence (also 3:1).

  • If nobody shares information with each other, what would each student predict?

  • If everyone shared their information, what would they predict (using Bayesian updating)?

Extremizing

  • This is the intuition behind extremizing predictions.

  • If a group of forecasters have independent pieces of information, they should become more confident if they shared what they know.

  • We can simulate this process by making everyone’s forecast a bit more confident before taking the average.

  • How much you extremize depends on how much independent information you think the crowd has.

Extremizing

If every individual has fully independent evidence informing their prediction:

\[ o_{group} = o_1 \times o_2 \times o_3 \times \ldots \times o_n \]

This is full Bayesian updating; multiply everyone’s odds together to get the posterior.

Extremizing

  • But that independence assumption is wildly optimistic!

  • Forecasters are usually relying on the same sources of information (Google searches, chatbots, newspapers, etc.)

  • More reasonable assumption: the crowd has some independent pieces of information.

Extremizing

If each individual has some independent information:

\[ o_{group} = (o_1 \times o_2 \times o_3 \times \ldots \times o_n)^\frac{\alpha}{n} \]

  • \(\alpha\) represents the amount of independent information.

  • \(\alpha = n\): Everyone has completely independent information. Group forecast is full Bayesian updating.

  • \(\alpha = 1\): No one has independent information. Group forecast is just the average individual forecast.

  • Truth is probably somewhere in the middle. For superforecasters, Satopää et al. (2014) use \(\alpha = 2.5\).

Example

  • Five forecasters predict 40%, 50%, 60%, 75%, and 75%.

  • The median forecast is 60%.

  • The extremized forecast \((\alpha = 2.5)\) is:

\[ (2:3 \times 1:1 \times 3:2 \times 3:1 \times 3:1)^\frac{2.5}{5} = (9:1)^\frac{1}{2} = 3:1 = 75\% \]

  • This isn’t the sort of math I’d ask you to perform on an exam.

  • I just want you to have it in your toolbox.

Takeaways

  • Confident predictions are justified if multiple, independent pieces of evidence point towards the same conclusion (Bayesian updating, dragonfly-eyed foxes).

  • When reading briefings, consider whether each author is bringing a different perspective, or if everyone is relying on the same evidence.

    • If the former, then the group forecast should be more confident than the individual forecasts (extremizing).
  • Organize teams so that they share information rather than just sharing conclusions.

References

Satopää, Ville A., Jonathan Baron, Dean P. Foster, Barbara A. Mellers, Philip E. Tetlock, and Lyle H. Ungar. 2014. “Combining Multiple Probability Predictions Using a Simple Logit Model.” International Journal of Forecasting 30 (2): 344–56. https://doi.org/10.1016/j.ijforecast.2013.09.009.