Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurable image descriptor fn get_image_description and support for list type content in messages #2286

Open
suneeta-mall opened this issue Mar 3, 2025 · 2 comments · May be fixed by #2297
Assignees

Comments

@suneeta-mall
Copy link

suneeta-mall commented Mar 3, 2025

🚀 The feature

Hey, Thanks for your work on mem0. I was wondering what is the vision towards increasing the support for multi-model input/message. At the moment I am running into a few issues, namely:

  1. List-based content can not be added/indexed by the mem0 (with redis backing). An example of list based content is shown here:
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe the this image",
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://someimage_somewhere_jumbo.jpeg"
                    },
                },
            ],
        }
    ]

These fail with error TypeError: list indices must be integers or slices, not str but the entirely correct format for multi-modal OpenAI message format.

I can see that json/dict content is processed fine, however. i.e. the following is okay:

```json
    [
        {
            "role": "user",
            "content": {
                    "type": "text",
                    "text": "Describe the this image",
                },
 }, {
            "role": "user",
            "content": {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://someimage_somewhere_jumbo.jpeg"
                    },
                },
        }
    ]
  1. The image descriptor method get_image_description assumes a call to OpenAI. Can we make this method configurable for base_url, api_key and model i.e., in general, BaseLlmConfig to point it to any LLM?
    def get_image_description(image_url):

Motivation, pitch

To make mem0 more usable in a multi-modal setting where custom image descriptors can be more valuable for cost and domain fit purposes.

@suneeta-mall suneeta-mall changed the title Configurable image descriptor Configurable image descriptor fn get_image_description and support for list type content in messages Mar 3, 2025
@deshraj
Copy link
Collaborator

deshraj commented Mar 4, 2025

Thanks for opening the issue @suneeta-mall. We are happy to add support for it.

@Dev-Khant Dev-Khant linked a pull request Mar 4, 2025 that will close this issue
2 tasks
@Dev-Khant
Copy link
Member

Hey @suneeta-mall I'm working on this issue, so can you please elaborate on why there is a need to pass the text field in the content key? Want to understand the use-case here. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants