Why Bad Incentives Fuel AI Hallucinations

An in-depth analysis of how bad incentives in AI model evaluations encourage hallucinations and how they can be corrected.

Why Bad Incentives Fuel AI Hallucinations
Diagram illustrating the causes and solutions of hallucinations in AI models

The problem of hallucinations in artificial intelligence

Hallucinations generated by large language models are a fundamental problem in the field of artificial intelligence, appearing as plausible but false statements, often highly convincing. These errors, although they improve as technologies evolve, remain one of the main obstacles to overcome.

Origins of hallucinations

The causes of such errors are many, but one of the main ones lies in the pretraining method used. Models are trained to predict the next word in a sequence of text, with no true or false labels present. This method relies on learning linguistic structures and the most frequent patterns, eliminating repeated errors in areas such as spelling or punctuation. However, for infrequent or specific facts, such as a date of birth, the model may generate incorrect answers because they lack a real, verifiable context.

Wrong evaluation and incentives

The heart of the problem, according to a recent study, lies in the evaluation system used to measure the performance of AI models. Current methods reward accuracy, thereby encouraging the error of giving highly confident answers even if false, in order to raise the overall score. This behaviour can lead models to 'guess' more often, at the expense of honest participation, especially when answers are difficult or little known.

Proposed solutions

To address this challenge, the authors of the new study suggest changing the evaluation criteria for language models. Introducing scoring systems that penalise overly confident answers and instead reward appropriate uncertainty could reduce hallucinations. Much like school tests, which discourage pure-luck answers, the new criteria should encourage models to state clearly when they are not certain of an answer.

Conclusions

  • AI hallucinations are partly due to training mechanisms and evaluation methods.
  • Improving evaluation metrics is crucial to reducing errors and guessing behaviours.
  • A more balanced approach to evaluations can help create more reliable models.