No surprise to me but a new paper looks at the forecast accuracy of over one million prediction questions from Polymarket, Kalshi, Manifold and the forecasting platform Metaculus. A virtue of the new analysis is that it covers many different areas from business to elections to geo-politics. An event predicted to happen p% of the time happens about p% of the time. Can’t beat that.

Using approximately one million resolved contracts and questions across four collective forecasting platforms, twelve domains, and time horizons ranging from hours to years, we evaluate the calibration and accuracy of collective forecasts in real-world settings. We found that forecast probabilities were overall highly calibrated: events assigned a probability of p% occurred approximately p% of the time. This broad pattern was evident across platforms and many domains, although fine-grained analyses revealed systematic tendencies toward over- or underestimation in highly consequential areas. While this does not mean that forecasting platforms provide certainty about individual events, our findings suggest that large-scale aggregation of judgments contains meaningful information about real-world uncertainty; when examined in aggregate, the predictions generated by the platforms provided informative estimates of event likelihood.