As part of my work with the organisation Lit in my position at RenPhil, I recently had the opportunity to try out how different LLMs with vision capabilities do at transcribing student assessments into useful schema.
Process
For the models, I gathered all the available models with vision capabilities from the big providers and as many less well known and/or open-weight models as I could find supported on openrouter.ai. I started this over the summer, so a lot of additional models have come out since. I have added as many of them as I could.
For the output format, I decided on a simple json schema, collecting some metadata (trying to capture lesson context) and otherwise categorising each text found into either instructional text, student work, or teacher markings and comments. I tested two prompts, one just containing a few sentences of task explanation and the schema, the other containing more context, some nuances to look out for, and explanations for each of the schema entries.
For test data, I found a few worksheets on the internet, mostly posted for their funny student responses, and manually transcribed them myself. The score I mention going forward is the ratio of information successfully extracted by the LLM, averaged over all samples. This sometimes includes not falsely claiming information exists when it doesn’t. Given there’s only a few samples and they are publicly accessible, this can only be seen as an initial benchmark of models worth investigating in more depth and with data more targeted to any given product.
For orchestration, I also used this project to try out promptfoo which was useful in orchestrating making calls to different model providers. Their platform focusses on other use cases than testing a large number of models against a small number of samples, so for the analysis I mostly used my own visualisations.
The whole project probably cost about $30 in inference. In the following sections, I’ll share some plots on my findings over this small sample of assignments.
Cost Comparison
The number of available models these days quickly makes comparisons like the following graph hard to parse:
Restricting ourselves just to the most promising models, we have:
Still not the most readable, but enough to pick out some interesting insights:
- Claude Fable 5.1 and Opus 5.5 unsurprisingly perform well, with a cost of 10 to 10s of cents per sample and GPT Astra 6 and Sol 5.6 not far behind.
- The region of single cent costs per sample is led by Gemini Flash 3.7 and MuseSpark 1.1 from earlier this summer, with their newer sibling models scoring slightly worse. Still, Gemini Argon 4 and Claude Sonnet 5.5 are only a few cents more expensive and do just as well.
- The sub-cent region models perform slightly worse than the more expensive models do. The standout models there are the Qwen Flash models, 3.7 and 3.8, as well as MiMo 2.6 Pro.
- Some models and model-families respond strongly to the difference in the prompt detail (for example much of the Claude models) while for some both points lie much closer together (like for GPT Luna 6 and Sol 6.1 and Gemini Argon). Very few models perform better under the less-detailed prompt, most noticeably Muse Wonder and Gemini 3.1 Pro.
Latency
In addition to scoring and price, it’s also interesting to consider the latency of the responses:
- As expected, most models have a similar latency for both prompts with the smaller prompt leading to slightly faster outputs. Curiously, some models are slower with a smaller prompt – plausibly spending more thinking tokens to recover some of the guidance missing from their prompt.
- The slowest (well-performing) model by far is Kimi K3 with hundreds of seconds. The Qwen models are not too far behind.
- Most of the commercial models take around half a minute per sample, with Sonnet and Luna on the shorter end and Fable and Sol on the longer.
- The fastest models I tested were Gemini 3.7 Flash and Muse Spark 1.1, both of which have newer sibling models that are slower (as well as performing slightly worse).
Model Size
For openweight models, it is also good fun to check the model size against the scores (in the full range):
While there may be some correlation between the maximum score of these models and their size, it is much less noticeable than expected, and the model family (and prompt, for a lot of them) seems to matter much more.
Conclusions
My main takeaway from this is that, in general, vision models today are capable of transcribing children’s class assignments and only rarely make mistakes in attributing writing to teachers, students, and printed instruction text. Excitingly, model performance does not appear to be strongly correlated to its size. Instead, many smaller and open-weight models perform similarly well to the big players like the Claude and GPT models, especially the Qwen models (both Flash and Pro) and the new MiMo model. These models can often be used at a fraction of the price of frontier models by the big providers and we may be able to run them on our own infrastructure if the providers choose to not support them going forward, which often happens with frontier providers, leading to model churn that necessitates costly re-evaluations and re-optimising of prompts.
Leave a Reply