Models Can't Animate a Design Yet: Figma and Stanford Benchmarked 32 LLMs and Only Four Cleared 50%

November 12, 2026 – On November 10, Figma and Stanford University published MotionGraphicsBench, a benchmark in which 32 language models were asked to animate real design work. Almost none of them could do it reliably, and the interesting part of the result is not the leaderboard. It is how the work is graded. (Hugging Face)

What the benchmark actually measures

MotionGraphicsBench is an expert-curated set of 100 motion editing tasks. Each task gives the model a complete input static SVG plus a motion edit prompt and asks it to animate the supplied elements. That split is deliberate: because the artwork is handed over, a failure cannot be blamed on bad vector generation. The tasks span 17 animation requirement types across three families – spatial structure, appearance and visibility, and temporal sequencing – and the inputs were derived from real-world Lottie animations in the LottieAnimation-660K dataset, with the motion edit prompts human-reviewed. (arXiv 2611.10592)

Thirty-two proprietary and open-weight models from 12 providers were evaluated, spanning the GPT, Claude, Gemini, Muse, Grok, Qwen, DeepSeek, GLM, Kimi, Nemotron and Gemma families, at multiple reasoning-effort settings.

Only four models cleared 50 percent

At their highest available reasoning-effort settings, only four of the 32 models exceeded 50 percent pass@1. The top score was GPT-6 Astra at maximum reasoning, with 67.7 percent. The strongest open-weight settings were GLM 5.3 (max) at 39.8 percent, Qwen 3.8 (xhigh) at 39.4 percent and Qwen 3.8 Flash-Next (xhigh) at 34.0 percent. (Hugging Face)

Both halves of that sentence need to be read together. Only four of 32 models cleared the halfway mark at all, so 67.7 percent is the best result in a weak field, not a strong absolute score. Against that field, 39.8 percent for the best open-weight configuration is not a trivial number either – it means a fully open stack is within striking distance of the proprietary leaders on a task where most models are near the floor.

The grader is the real contribution

Most generation benchmarks are graded by a model judge, which means the thing being measured and the thing doing the measuring are the same kind of artefact. MotionGraphicsBench does something else. Each task ships with a task-specific deterministic verification program: a series of human-readable checks, each tied to a specific part of the motion edit prompt. The programs were built with a multi-agent pipeline.

The pipeline was then tested against humans. On 180 held-out animations, the verification programs agreed with human judgments 93.3 percent of the time, raw, against 83.3 percent for a VLM judge baseline. (Hugging Face)

Two attributions matter here. First, the 83.3 percent figure is the benchmark’s own internal control, not a competing scoring system from another lab. Second, and more importantly, the verification programs were themselves constructed by a multi-agent pipeline, so 93.3 percent measures agreement with that construction’s notion of a correct animation – it is not agreement with an independent ground truth. The honest reading is that a deterministic, auditable checker can be built to stand in for human judgment on this kind of work, and that a model judge on the same tasks was meaningfully worse.

That is the transferable pattern for anyone shipping design tooling: decompose the request into typed requirements, write one check per requirement in a form a human can read and argue with, and let a multi-agent pipeline assemble the checks while humans review them.

One try versus 32 tries

pass@k was measured for 15 of the models with 32 independent attempts per task. pass@32 ranged from 38.0 to 95.0 percent. (Hugging Face)

This is the number that should change a roadmap. The best model still fails roughly a third of tasks in a single shot, while up to 95 percent of tasks are solvable by some attempt within 32 tries. The headline 67.7 percent therefore understates what repeated sampling can reach and overstates what single-shot reliability looks like. A design assistant in production today is sampling repeatedly, not calling once.

The failures are semantic, not visual

The documented failure modes are failing to match requested poses, failing to follow motion trajectories, and failing to coordinate related events correctly. In other words: sequencing and coordination semantics, not rendering. The model reliably produces the parts and unreliably produces the motion.

That distinction is the actionable one for a team shipping a design-assistant feature. Improving visual output quality will not fix a model that cannot keep two animations in sync.

What this benchmark does not tell you

Three caveats worth carrying. The benchmark isolates motion editing of a supplied SVG, so the harder and more commercially interesting case – text-to-animation from scratch – is deliberately out of scope. Figma co-authored the benchmark and sells design tooling, so the framing of what counts as a correct motion edit is not a neutral party’s. And the model names and reasoning-effort tiers quoted above are taken from the primary post; no independent replication of these scores was available when this was written.

The dataset, the benchmark code and the full paper are public: the dataset and code at huggingface.co/datasets/figma-ai/motion-graphics-bench, the project site at motion-graphics-bench.github.io, and the paper at arXiv 2611.10592. The post names Jiaju Ma, Ashvini Kumar Jindal, Chuhua Wang and Sumithra Bhakthavatsalam, of Stanford University and Figma.