Cognitive Bias Benchmark for LLMs as EvaluatoRs” (CoBBLEr)

Via AI Brakfast:

“A recent study reveals that large language models exhibit cognitive biases and do not align with human preferences in text evaluation. This issue is critical, as LLMs are increasingly used in applications like content recommendation and job application screening. If an LLM, tasked with assessing the quality of a cover letter, is biased towards longer texts or specific keywords, it could unjustly give preference to certain applicants. This disparity in evaluation could lead to unfair advantages, regardless of the actual qualifications of the candidates. The research, involving an analysis of 15 different LLMs using the Cognitive Bias Benchmark for LLMs as EvaluatoRs” (CoBBLEr), revealed biases like egocentricity and order preference, casting doubt on the suitability of LLMs for unbiased, human-like judgment in real-world scenarios.”