Interesting observations:
* Llama 3.2 multimodal actually still ranks below Molmo from ai2 released this morning.
* AI2D: 92.3 (3.2 90B) vs 96.3 (of Molmo 72B)
* Llama 3.2 1B and 3B is pruned from 3.1 8B so no leapfrogging unlike 3 -> 3.1.
* Notably no code benchmarks. Deliberate exclusion of code data in distillation to maximize mobile on-device use cases?
Was hoping there would be some interesting models I can add to https://double.bot but doesn't seem like any improvements to frontier performance on coding.
On the second point, you're comparing MMMU-Pro (multimodal) to MMLU-Pro (text only). I don't think they published scores on MMLU-Pro for 3.2.
(Edit: parent comment was corrected, thanks!)
Yep you're right, thanks for catching (sorry for the ninja edit!)
Where do you see the MMLU-Pro evaluation for Llama 3.2 90B? On the link I only see Llama 3.2 90B evaluated against multimodal benchmarks.
Ah you're right I totally misread that!
Announcement post instead of general top-level domain: https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-...
When will it come to ollama? That's my preferred quantization platform.
do you have an idea how long will take to have it available in ollama ?
just arrived :)
No multimodal yet :(
Soon! We're working on it, and it's almost there.
It seems like the weights for Llama3.2-11B-Vision-Instruct are about 20GB. Will ollama run that on an M1 Mac with 32GB RAM? Will the ollama model library have quantized models?
I think it should run fine. Yes, there will be quantized versions.
What is the best provider for API use of llama frontier models considering pricing / reputation?
I’m more excited about Llama stack, I can’t wait for local models to be able to use tools in a standard way.
Are users in the EU not allowed to use it, like they threatened to do so recently?
They are not allowed indeed: https://github.com/meta-llama/llama-models/blob/main/models/...
> With respect to any multimodal models included in Llama 3.2, the rights granted under Section 1(a) of the Llama 3.2 Community License Agreement are not being granted to you if you are an individual domiciled in, or a company with a principal place of business in, the European Union. This restriction does not apply to end users of a product or service that incorporates any such multimodal models.
Interesting though, since (some) EU law applies outside the EU anyway, so I'm not sure how much lawyer there is in the text.