Модель 2B набрала 10%. 93% её ответов были правильными

Оригинал

The 2B model scored 10%. 93% of its answers were right

I expected small local models to be worse than a hosted decision API at Japanese classification. I did not expect most of the gap to be something other than judgement. On the hardest of three tasks one model scored 10.8% overall while getting 27 of the 29 answers it produced right. The other 221 rows it never answered. What I ran, and what the accuracy number counted Three Japanese tasks, 250 identical rows each. livedoor is 9-class news-topic classification (mean 1,308 characters),