Using IWE with embeddings

[kiil]

I am wondering if embeddings could somehow be used alongside or as a supplement to iwe?

@gimalay - do you have any thoughts on if that could make sense?

[gimalay]

Hey @kiil,

Thanks for starting the discussion.

What problem would you try to solve with embeddings? Want is the outcome you are looking for?

[kiil]

For semantic search.

Or is there another way top get that or something similar?

[gimalay]

What would you expect to index via embeddings? Just the titles, titles with parent note titles, full content?

What use case do you see, semantic search similar to fuzzy search? Something else?

[kiil]

I guess it is mostly for the user-facing ux:

1. Embed all notes with e.g. OpenAI text-embedding-3-small
2. Store vectors in a local vector database e.g. sqlite-vec
3. User asks a question → find the 3-5 semantically closest notes
4. Use iwe retrieve -k {key} -d 1 -c 1 on each of them
5. Send the extended context output to LLM

So, they must be stored with the key.

[gimalay]

I was thinking about this and cannot find a good way to use embeddings in iwe.

For this step: User asks a question → find the 3-5 semantically closest notes

AI agent can read through the contents of your library (headings/key) and tell you what's matching and then retrieve. The benefit of doing this using embeddings is the speed. But the downside is that you need to re-index and cache embeddings and it requires an embeddings model which is pretty heavy dependency.

I'm not sure if we need speed for this. LLM can do contents scan relatively fast (for a small library) and update iwe existing indexes (links, etc) for future fast retrieval.

There may be a use case I just do see it yet.

[kiil]

I was thinking about this and cannot find a good way to use embeddings in iwe.

For this step: User asks a question → find the 3-5 semantically closest notes

AI agent can read through the contents of your library (headings/key) and tell you what's matching and then retrieve. The benefit of doing this using embeddings is the speed. But the downside is that you need to re-index and cache embeddings and it requires an embeddings model which is pretty heavy dependency.

I'm not sure if we need speed for this. LLM can do contents scan relatively fast (for a small library) and update iwe existing indexes (links, etc) for future fast retrieval.

There may be a use case I just do see it yet.

I agree, for small to medium sized bases, IWE already has a graph and thereby a better knowledge structure than gained from embeddings.

[kiil]

I would say, though, that for very large knowledge bases, embeddings could help when doing the more hierarchical structuring that iwe excels at.

[gimalay]

I'm currently working on something else. I want to add database like (MongoDB) style query language which will allow to structure and query large knowledge bases in a many new ways. This new set of features will be available soon and I would appreciate your feedback.

[gimalay]

@kiil it's not embeddings but may interesting to you https://www.reddit.com/r/iwe/comments/1t1m93k/markdown_as_a_database_memory_for_humans_and_ai/

[kiil]

btw, @gimalay - have you seen this:

reekta92/graf: A TUI graph view for markdown files with wikilinks. Customizable.

[gimalay]

Haven't seen this, thanks for sharing!

[kiil]

@gimalay I read you piece on performance of IWE. Very impressive!

To get around the cold start on cli invocations, maybe there could at some point be a repl mode.

Also something like this is interesting: nushell/nushell#18142

[gimalay]

Yes, something like this could work! Some kind of cache can dramatically speedup cold start.

So far I personally didn't hit the performance celling. My biggest storage is under 5000 files and 100ms is totally acceptable for my use case. How many files you are working with?

[kiil]

How many files you are working with?

Currently 3286

So far no problems. But I like to think ahead :)

[gimalay]

There are multiple optimization strategies available. The simplest one is to include file path prefix with the query to reduce the corpus before any parsing. If the knowledge base is structured into segments it can allow selecting a sub set of it as a first step before any parsing happens.

Any serious optimization will require more robust persistence layer (with the pre build stored indexes) and gets into database territory.

When you are actively working with hundreds of thousands of documents it's probably a good time to think about a real database.