I would be thrilled to answer any questions or thoughts you might have. An article combined with thoughts, ideas, and considerations holds much more educational power!
You have shown three patterns of multi-modal RAG, but both input and output are in text format. Is this practical in real-world? I mean, what if the input contain both text and image?
Hey, sorry to get back to you so late, I don't know why this slipped through the cracks.
A lot of these examples use text as a central modality, but that is by no means required, and other approaches might be more advantageous in different applications.
If you had a piece of text with an image interlaced that you wanted to search, you could do retrieval on the text and image independently, or you could perhaps average their vectors somehow and do comparison against that. If you wanted to use text and images as the query, you could do a similar process as well.
I would be thrilled to answer any questions or thoughts you might have. An article combined with thoughts, ideas, and considerations holds much more educational power!
You have shown three patterns of multi-modal RAG, but both input and output are in text format. Is this practical in real-world? I mean, what if the input contain both text and image?
Hey, sorry to get back to you so late, I don't know why this slipped through the cracks.
A lot of these examples use text as a central modality, but that is by no means required, and other approaches might be more advantageous in different applications.
If you had a piece of text with an image interlaced that you wanted to search, you could do retrieval on the text and image independently, or you could perhaps average their vectors somehow and do comparison against that. If you wanted to use text and images as the query, you could do a similar process as well.