This is such a basic thing nowadays, and ElasticSearch is massive overkill for it. Something like SQLite or LanceDB or basically any vector database is much more appropriate.
This seems to be coming from the “we must make ElasticSearch AI-compatible” department more than anything.
ElasticSearch is fine. If your dataset isn't too big you aren't going to hit shard and memory limits and if you do chances are you are already in a large enough organisation that you'll have the manpower to do the required maintenance. It's not rocket science.
> This seems to be coming from the “we must make ElasticSearch AI-compatible” department more than anything.
I don't see the problem in that. It'd be great to have agentic capabilities embedded into Kibana and ES as long as it's not user hostile.
If you already have Elasticsearch, it makes sense to continue utilizing it.
Saying, “just use SQLite” completely dismisses the idea that this is a _shared_ memory across teams. The ability to easily connect to the remote service and have everything “just work” pays dividends when you have dozens or hundreds of users.
I’m literally laughing at the root comment’s idea of proposing we replace ES with SQLite and imagining how that architecture review would go. Not everyone is doing MB/GB scale workloads.
Nah, "Any other vector DB" starts to fall apart once you need stuff like scripted scoring like OP uses. Then it starts to be a question of, "do you need ANN for performance?" since SQLite only does brute-force vector scoring. And granted, brute-force is performant for far more vectors than most people give it credit for, but it definitely hits a wall well below 1 million if you want it to have webpage-type latency.
Maintaining Elasticsearch isn't free, but picking an underpowered db and having to port to the right one is also quite time consuming.
The design they talk about includes 3 different types of memory. They store those kinds of memory separately, so that if there's 10 users, all 10 access memories that are more general ("what bulbs work with this kind of light fixture"), and user-specific memories are segregated ("sarah has three lightbulbs"). The different memory types are ranked together leading to a different result. So this is a novel design and use of ElasticSearch-specific features
Summary of the article (https://pastebin.com/aawJfrF6) since the original one is like reading an academic paper filtered through an LLM that hates human readers.
It seems like a cool approach. Don't know if it's novel but it's much smarter than "shove markdown files into directories".
It's sort of the whole tension when you query vectorized data/embeddings -- you need to balance accuracy and recall against the performance you need.
It took me a while to wrap my head around the two terms since they seem similar -- but Accuracy is basically "did i get mostly good results" and Recall is "did I get most of the good results" and they're subtly different. :)
Those two terms, though, will unlock as deep a rabbit-hole as you'd like on the subject.
Especially for indie users/devs and smaller teams. I built a part of this(the retriever) in < 4 hours https://github.com/itissid/wiki for replacing deepwiki.
I think the challenge is to teach how ranking works to people more effectively so that they can build it for themselves and host them on their own.
Like the other day someone who has worked in search explained to me why you would care about using learning-to-rank(LTR) technique to train your own feature vector weights on your data. My understanding is that weighted features work better(retreival wise) on textual data than plain BM-25 and vector embedding db indexing of text chunks of your data with minimal preprocessing. So if you have lots of conversations you can create a ton of features(like attributes of a conversation) from it and ones that matter more will rank higher. And you can use a regularization(like L1) to kill unimportant ones.
[EDIT]: IIUC, I think LTR is important because you likely want different features to matter more for different parts of your documents, e.g. what matters for codebase documentation is different from your personal journal.
Yeah, after I tokenize them and embed them into vector form. Then it’s a simple cosine distance.
The point about memory is sometimes you remember great detail, sometimes you only remember that the memory exists, so having a good tool loop to attempt to recall and try permutations is good.
so the 11% miss rate - do users actually notice when the agent drops a memory? like if someone already said they tried X and the agent suggests it again.
This is such a basic thing nowadays, and ElasticSearch is massive overkill for it. Something like SQLite or LanceDB or basically any vector database is much more appropriate.
This seems to be coming from the “we must make ElasticSearch AI-compatible” department more than anything.
ElasticSearch is fine. If your dataset isn't too big you aren't going to hit shard and memory limits and if you do chances are you are already in a large enough organisation that you'll have the manpower to do the required maintenance. It's not rocket science.
> This seems to be coming from the “we must make ElasticSearch AI-compatible” department more than anything.
I don't see the problem in that. It'd be great to have agentic capabilities embedded into Kibana and ES as long as it's not user hostile.
If you already have Elasticsearch, it makes sense to continue utilizing it.
Saying, “just use SQLite” completely dismisses the idea that this is a _shared_ memory across teams. The ability to easily connect to the remote service and have everything “just work” pays dividends when you have dozens or hundreds of users.
I’m literally laughing at the root comment’s idea of proposing we replace ES with SQLite and imagining how that architecture review would go. Not everyone is doing MB/GB scale workloads.
Nah, "Any other vector DB" starts to fall apart once you need stuff like scripted scoring like OP uses. Then it starts to be a question of, "do you need ANN for performance?" since SQLite only does brute-force vector scoring. And granted, brute-force is performant for far more vectors than most people give it credit for, but it definitely hits a wall well below 1 million if you want it to have webpage-type latency.
Maintaining Elasticsearch isn't free, but picking an underpowered db and having to port to the right one is also quite time consuming.
The design they talk about includes 3 different types of memory. They store those kinds of memory separately, so that if there's 10 users, all 10 access memories that are more general ("what bulbs work with this kind of light fixture"), and user-specific memories are segregated ("sarah has three lightbulbs"). The different memory types are ranked together leading to a different result. So this is a novel design and use of ElasticSearch-specific features
Would be interesting if one can replace ElasticSearch with something like Typesense here
I have a request: can this text be even more AI generated?
Summary of the article (https://pastebin.com/aawJfrF6) since the original one is like reading an academic paper filtered through an LLM that hates human readers.
It seems like a cool approach. Don't know if it's novel but it's much smarter than "shove markdown files into directories".
For someone who isn't super familiar, what is "R@10", and is 0.89 good? It's impossible to google for
89% chance the thing you want is among the 10 items returned by the system
So over 10% of the time, it fails? That's not a great search engine
It's sort of the whole tension when you query vectorized data/embeddings -- you need to balance accuracy and recall against the performance you need.
It took me a while to wrap my head around the two terms since they seem similar -- but Accuracy is basically "did i get mostly good results" and Recall is "did I get most of the good results" and they're subtly different. :)
Those two terms, though, will unlock as deep a rabbit-hole as you'd like on the subject.
I built one into my agent using sqlite…
Especially for indie users/devs and smaller teams. I built a part of this(the retriever) in < 4 hours https://github.com/itissid/wiki for replacing deepwiki.
I think the challenge is to teach how ranking works to people more effectively so that they can build it for themselves and host them on their own.
Like the other day someone who has worked in search explained to me why you would care about using learning-to-rank(LTR) technique to train your own feature vector weights on your data. My understanding is that weighted features work better(retreival wise) on textual data than plain BM-25 and vector embedding db indexing of text chunks of your data with minimal preprocessing. So if you have lots of conversations you can create a ton of features(like attributes of a conversation) from it and ones that matter more will rank higher. And you can use a regularization(like L1) to kill unimportant ones.
[EDIT]: IIUC, I think LTR is important because you likely want different features to matter more for different parts of your documents, e.g. what matters for codebase documentation is different from your personal journal.
I don't treat memory like RAG. That's the key. I only track decisions, actions, and outcomes.
Ah so you extract decisions, actions and outcomes and you index and search over them?
Yeah, after I tokenize them and embed them into vector form. Then it’s a simple cosine distance.
The point about memory is sometimes you remember great detail, sometimes you only remember that the memory exists, so having a good tool loop to attempt to recall and try permutations is good.
so the 11% miss rate - do users actually notice when the agent drops a memory? like if someone already said they tried X and the agent suggests it again.