ForecastGeo
Menu

Accountability

The forecast ledger

This site asks a small language model to forecast what happens next, then keeps score. Every forecast gets a deadline and is checked against articles that arrived later. Forecasts that nothing later confirms are recorded as unresolved — that is a real outcome here, not a quiet pass.

17903

forecast runs

2538

runs with a decided outcome

14863

unresolved

nothing later confirmed any path

502

still awaiting a deadline

0.193

Brier score

lower is better; 0.25 is a coin flip

Calibration

A well-calibrated forecaster that says 30% is right about 30% of the time. Each bar is the share of scenarios in that confidence band that actually happened; the tick is what was predicted.

Predicted probability vs observed hit rate observed predicted 0–20%: predicted 12%, observed 21% over 1751 scenarios 0–20% n=1751 20–40%: predicted 28%, observed 27% over 2656 scenarios 20–40% n=2656 40–60%: predicted 44%, observed 26% over 599 scenarios 40–60% n=599 60–80%: predicted 63%, observed 27% over 75 scenarios 60–80% n=75 80–100%: predicted 83%, observed 100% over 2 scenarios 80–100% n=2
Show these numbers as a table
BucketPredictedObservedScenarios
0–20%12%21%1751
20–40%28%27%2656
40–60%44%26%599
60–80%63%27%75
80–100%83%100%2

By model

ModelRunsStill pendingScoredUnresolvedBrier
llama 8894 0 770 8124 0.193
gemma-4-E4B-it-qat-UD-Q4_K_XL 4700 0 341 4359 0.205
Qwen3.6-35B-A3B-UD-Q4_K_S 1297 13 69 1215 0.213
Qwen3.6-35B-A3B-UD-Q5_K_M 1169 0 86 1083 0.211
gemma-4-E4B-it-qat-UD-Q4_K_XL 465 386 3 76 0.212
qwen3.8-flash-next-q2_0 102 102 0 0 awaiting a deadline
gemma-4-E2B-it-qat-UD-Q4_K_XL 7 1 0 6 awaiting a deadline

Runs whose timeframe could not be parsed are excluded from the Brier average, because their deadline was assigned by default rather than read from the forecast.

Resolved runs