ForecastGeo Log in

Accountability

The forecast ledger

This site asks a small language model to forecast what happens next, then keeps score. Every forecast gets a deadline and is checked against articles that arrived later. Forecasts that nothing later confirms are recorded as unresolved — that is a real outcome here, not a quiet pass.

7833

forecast runs

948

runs with a decided outcome

5874

unresolved

nothing later confirmed any path

1011

still awaiting a deadline

0.205

Brier score

lower is better; 0.25 is a coin flip

Calibration

A well-calibrated forecaster that says 30% is right about 30% of the time. Each bar is the share of scenarios in that confidence band that actually happened; the tick is what was predicted.

Predicted probability vs observed hit rate observed predicted 0–20%: predicted 12%, observed 22% over 749 scenarios 0–20% n=749 20–40%: predicted 28%, observed 27% over 791 scenarios 20–40% n=791 40–60%: predicted 44%, observed 26% over 317 scenarios 40–60% n=317 60–80%: predicted 63%, observed 27% over 37 scenarios 60–80% n=37 80–100%: predicted 85%, observed 100% over 1 scenario 80–100% n=1
Show these numbers as a table
BucketPredictedObservedScenarios
0–20%12%22%749
20–40%28%27%791
40–60%44%26%317
60–80%63%27%37
80–100%85%100%1

By model

ModelRunsStill pendingScoredUnresolvedBrier
gemma-4-E4B-it-qat-UD-Q4_K_XL 4700 0 341 4359 0.205
llama 1483 1010 47 426 0.193
Qwen3.6-35B-A3B-UD-Q5_K_M 1169 0 86 1083 0.211
gemma-4-E2B-it-qat-UD-Q4_K_XL 7 1 0 6 awaiting a deadline

Runs whose timeframe could not be parsed are excluded from the Brier average, because their deadline was assigned by default rather than read from the forecast.

Resolved runs