I like that they're being up front about this. Benchmarks are a metric that can be gamed. They give you an approximate estimate of where things land, but you really need to test for your use case.
I used the model and it was good enough for email classification.
I was wondering if I'm supposed to be doing multiple runs for topic classification with grouping related topics i.e. if its sport,economy or tech related news and then another run for soccer,basketball or something else
or batching them like the News article topic classification example in a single run.
Cool to see another model with support for bounding boxes on images. Really wish to see segmentation in a future release; Gemini 3.5 Flash-Lite is mandatory in my use case as it's the best model for semantic image segmentation in my experience.
Interesting way to release this, I'm not sure how to say this, but this has a real "enterprise"/old school taste to it verses how all the other models/AI companies are releasing things.
Well, they do need to try to compete with SubQ’s longer context and image processing that has been coming out ahead in benchmarks, plus the thousand self-hosted options that now exist.
it's actually more usefull than it looks. it is not much useful on its own but it can play a big role in a complex workflow or custom harness.
i used luna 5.6 as the ranker/filter to remove useless items from upstream rag(mixing embedding, regex etc) and it did a good job but took 5-10 seconds each time. now less than 1 second and no observable precision loss. hope it helps
I use it as a drop in replacement for LLMs I had previously doing filtering and classification in a pipeline. It's loads cheaper and faster with roughly equivalent performance on my own internal tests. The confidence scores (which are also reasonably well calibrated on my data) are a super nice bonus.
So they held back a bunch of features because they knew they had basically no moat, and now they're trickling out a bit more functionality to try to convince people they are better than others and move fast? And using Apple advert slogans to show how cool they are?
> Warning: AI benchmarks. Please don't trust them
I like that they're being up front about this. Benchmarks are a metric that can be gamed. They give you an approximate estimate of where things land, but you really need to test for your use case.
In case you need something a bit more lightweight, or want privacy. You can get amazing (better in a lot of cases) accuracy and performance from local models that run and train on CPU: https://jeffyclassify.com/blog/when-llms-fail-at-classificat...
I used the model and it was good enough for email classification. I was wondering if I'm supposed to be doing multiple runs for topic classification with grouping related topics i.e. if its sport,economy or tech related news and then another run for soccer,basketball or something else or batching them like the News article topic classification example in a single run.
Cool to see another model with support for bounding boxes on images. Really wish to see segmentation in a future release; Gemini 3.5 Flash-Lite is mandatory in my use case as it's the best model for semantic image segmentation in my experience.
Curious why you use Gemini 3.5 Flash Lite instead of SAM/Moondream? Any specific workflows where it outperforms?
Interesting way to release this, I'm not sure how to say this, but this has a real "enterprise"/old school taste to it verses how all the other models/AI companies are releasing things.
Very refreshing.
Sounds like an incremental change with context size change, new "box" type, and composable images.
As it's minor change did not expect much, but having a higher context helps for sure. I will own the errors for passing bigger context.
Well, they do need to try to compete with SubQ’s longer context and image processing that has been coming out ahead in benchmarks, plus the thousand self-hosted options that now exist.
I'm curious what people are building with these models
Judging from the tilte, I guess dreams.
Automatic tagging of images is what I'm doing with it. I'm using a modified Gemma model, which ingests images (and audio, for that matter).
it's actually more usefull than it looks. it is not much useful on its own but it can play a big role in a complex workflow or custom harness.
i used luna 5.6 as the ranker/filter to remove useless items from upstream rag(mixing embedding, regex etc) and it did a good job but took 5-10 seconds each time. now less than 1 second and no observable precision loss. hope it helps
I use it as a drop in replacement for LLMs I had previously doing filtering and classification in a pipeline. It's loads cheaper and faster with roughly equivalent performance on my own internal tests. The confidence scores (which are also reasonably well calibrated on my data) are a super nice bonus.
How does JEV prevent distillation?
I don't think they can. But they probably monitor usage and test for patterns to then block/ban accounts
So they held back a bunch of features because they knew they had basically no moat, and now they're trickling out a bit more functionality to try to convince people they are better than others and move fast? And using Apple advert slogans to show how cool they are?
With the speed they're adding features shows you're correct.
Which others?
There are several public benchmarks for decision models where they don't top every single one. https://benchmarkheaven.com/jev-models/all
I guess their cheeky "Warning: AI Benchmarks, don't trust them" is their way out of that one.