If you're working in AI governance, safety, or product development, HELM is a must-know.

← All articles
Artificial intelligence · EN

If you’re working in AI governance, safety, or product development, HELM is a must-know.

Led by Stanford’s Center for Research on Foundation Models (CRFM), HELM (Holistic Evaluation of Language Models) benchmarks 30+ of the most prominent LMs (OpenAI, Anthropic, Meta, Google, Cohere, etc.) across:
16 core scenarios, 7 societal-relevant metrics: accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency, 4900 evaluations and over 12B tokens of input/output.

Key Findings:

1. OpenAI and Anthropic’s models, fine-tuned with human feedback, perform significantly better in accuracy, robustness, and fairness. In some cases, smaller tuned models outperform models 10x their size.
2. Open vs. Closed: While open models like BLOOM and OPT are improving, they still lag behind closed ones like GPT-3.5 and Anthropic Claude. This raises questions about power concentration and accessibility in the LM ecosystem.
3. Most models that are accurate are also more fair and typo-resistant. But some fail dramatically on dialect shifts or factual tweaks. A multi-metric approach is essential to catch these blind spots.
4. Standardized prompting is key to comparing models meaningfully and building interoperable language interfaces.
5. We need better evaluation methods, datasets and frequent updates to keep up with risks and capabilities.

Source:

This post has an image on LinkedIn. View on LinkedIn →

Discuss on LinkedIn