Google Cloud AI Blog published material about methods to audit LLM-based evaluators. In the synopsis it is claimed that such systems are not fully objective and can reproduce bias familiar from human evaluation.
The publication names two examples. The first is evaluators' tendency to prefer longer and more confidently worded answers, even if the facts in them are incorrect. The second is dependence of the decision on the order of options: when they are permuted, the verdict changes in more than 75% cases according to the synopsis.
The practical takeaway is to check not only the final assessment but also the evaluator's stability to the form of the answer and to the arrangement of options. Details of the proposed methods and verification of the reported metrics are not provided in the available source package.
editorial commentary
Why it matters
If the described effects can be reproduced, developers of evaluation systems will need to test the robustness of verdicts to answer length and to permutation of options. The closest observable signal is the publication of a full methods description and independent results of repeated tests. A significant uncertainty is that only a synopsis of one source is available.