✗Reasons about codeprobed, and failed
Asked to predict exactly what a short program prints. This is the half of coding that matters for agent work and that writing valid syntax does not demonstrate: following state through control flow.✗Writes codeprobed, and failed
Asked for a specific function and checked with the Go compiler’s own parser. Nothing is executed — running model-written code to score a model would be a security hole opened for a metric — but syntactic validity is a fact rather than an opinion.✗Embeddingsprobed, and failed
The engine was asked for an embedding of a short text and returned a real, non-zero vector.✗Follows instructionsprobed, and failed
Given a plain constraint on its output format and checked for exact obedience. This separates an instruct-tuned model from a base one — a base model answers fluently and ignores every instruction, which then looks like a dozen unrelated weaknesses instead of one.✗Sustained outputprobed, and failed
Asked to produce a long structured answer and checked for length. A model that stops after a couple of hundred tokens cannot write a report, however capable it is otherwise.✗Extended thinkingprobed, and failed
The model emitted reasoning on the engine’s dedicated reasoning channel before answering. Read from that channel rather than by scanning the answer for tags, which is why models that reason are no longer reported as models that do not.✗Synthesises sourcesprobed, and failed
Given three documents with one fact split across them, and required to combine rather than quote. No single document contains the answer, so a model that retrieves without reasoning cannot pass.✗Tool callingprobed, and failed
The node asked the model to call a function and checked that it called the RIGHT one with a well-formed argument — not merely that it emitted a tool name.✗Tool loop finishedprobed, and failed
Whether the conversation ENDED after the tool call, or the model kept calling until the turn budget ran out. Passing “tool calling” and failing this means the model will call a tool in production and never come back. A model chosen on the first flag alone is the reason an agent silently stops delivering work.✗Structured outputprobed, and failed
Asked for JSON matching a schema, and got it. This is the difference between a model you can build on and one you must parse defensively.✗Visionprobed, and failed
Shown an image and asked what was in it. The node checked the answer.✗Audioprobed, and failed
Given audio and asked to transcribe it.✗Faithful to tool resultsprobed, and failed
The model was handed a list through a tool and had to report every item back. Calling a tool and understanding its answer are different abilities: a model can produce a perfect call, receive nine items, and confidently report three.