Pairwise accuracy
Take a random sample of pairs, ask AI to predict which received more funding. Score is percentage correct.
We could also weight by log(funding diff) to put more weight on correctly getting big differences than small differences.
Overall this doesn’t seem like it particularly gets at what we want, but it’s simple.
Decompose into two parts: P(project gets any funding) and distribution(log(funding)) conditional on getting any funding.
For P(project gets any funding), use Brier score.
For the distribution, ask it to give 10th, 25th, 50th, 75th, and 90th percentiles of funding in log dollars. Then score it with pinball loss.
Also create a dumb predictor for each one. For P(project gets any funding), use the base rate of projects in the sample size that get funded. For the distribution, use the funding ask * sample distribution of funding fill rate.
Then you can combine them with final_score = 0.5 * (1 - Brier/Brier_dumb) + 0.5 * (1 - pinball/pinball_dumb). (Or whatever weights, doesn’t have to be 0.5.)
The second part of this is similar to the current FundingBench scoring, but eliciting a distribution rather than a point estimate gets you more info. And the first part, asking whether a project gets any funding, gives you an element of discrimination/distinguishing bad projects from good small projects that I think currently isn’t that rewarded.
You could also give it all the proposals and the total amount of funding they received, and ask it to allocate funding among the proposals.
Then most straightforwardly, score = (overlap dollars) / (total dollars).
I think this is sort of what we care about, but it weights big projects heavily, which decreases the effective sample size. So maybe we should do log(dollars) instead.