All 11 fashions additionally struggled with agentic benchmarks designed to evaluate actual world problem-solving talents round common information, security, and coding. Claude 3.5 Sonnet and o1 ranked the best on this space, notably when it got here to extra structured duties with specific targets. Nonetheless, all fashions had a tough time with software program engineering and different duties requiring open-ended reasoning and planning.
Multimodality is turning into more and more vital for AI methods, because it permits fashions to course of completely different inputs. To measure this, Vector developed the Multimodal Huge Multitask Understanding (MMMU) benchmark, which evaluates a mannequin’s skill to cause about photos and textual content throughout each multiple-choice and open-ended codecs. Questions cowl math, finance, music and historical past and are designated as “simple,” “medium,” and “exhausting.”
In its analysis, Vector discovered that o1 exhibited “superior” multimodal understanding throughout completely different codecs and problem ranges. Claude 3.5 Sonnet additionally did nicely, however not at o1’s degree. Once more, right here, researchers discovered that the majority fashions dropped in efficiency when given more difficult, open-ended duties.
