Trending: On-device modelsSearch
iHeartGeek
iTECH

OpenAI proposes perplexity as a shared AI progress metric

OpenAI chief scientist Jakub Pachocki wants perplexity to anchor more transparent comparisons between frontier AI models and future safety limits.

Abstract illustration of a glowing neural-network brain for an OpenAI article about AI intelligence

OpenAI chief scientist Jakub Pachocki has argued that perplexity — the measure of how well a model predicts the next token — should become a shared yardstick for AI progress and a reference point for industry safety standards. The proposal arrives with a rare public look at OpenAI’s internal pretraining measurements.1

Perplexity is the statistical score attached to a model’s predictions: lower is better. OpenAI says its log-perplexity fell from roughly 0.99 for GPT-3.5 to about 0.91 for GPT-4, with newer systems now below 0.8 on the company’s main evaluation dataset. OpenAI links those improvements to increasingly capable behaviour, while stressing that the exact moments when new abilities appear cannot be predicted in advance.1

Why OpenAI wants a common measure

Pachocki’s argument is less about turning one number into a complete intelligence score and more about creating a metric that different labs can discuss openly. He proposes shared transparency standards, common development limits and concrete safety milestones before companies move beyond those limits.1

The proposal has obvious complications. Labs use different datasets and data mixtures, so a score from one developer may not compare cleanly with a score from another. Perplexity also misses progress that happens outside pretraining, including reasoning techniques, and it does not directly measure a practical task such as writing code or discovering a drug.1

OpenAI nevertheless believes the metric remains useful because it has tracked broad capability gains over several generations. The company expects further changes as models approach a log-perplexity range of roughly 0.68 to 0.7, although Pachocki says nobody can know in advance what those capabilities will be.

Our opinion

The proposal is valuable chiefly because it asks AI companies to publish a number that can be challenged, compared and debated rather than relying on vague claims about being more capable. Perplexity should not become a magic score for intelligence, but it could form one part of a wider safety dashboard if labs disclose their datasets, methods and limitations alongside it.

The harder test is whether companies accept meaningful limits when the metric improves faster than their ability to monitor the systems. A shared benchmark is useful; a shared willingness to pause is what would make it matter.