ml-ferret

Overview

Ferret is an end-to-end multimodal large language model (MLLM) developed by Apple's team, capable of accepting arbitrary forms of spatial referring (e.g., points, boxes, sketches) and accurately locating objects at any granularity in images while generating rich semantic descriptions. The model innovatively proposes hybrid region representation and spatial-aware visual sampler, achieving open-vocabulary referring and grounding for the first time.

Ferret is built on the LLaVA framework, integrating the Vicuna language model and CLIP visual encoder, and trained with a large-scale hierarchical instruction tuning dataset GRIT (~1.1M samples). It supports 7B and 13B parameter versions, can be trained on 8 A100 GPUs, and provides public weight delta files for the research community to reproduce and use.

The project also releases Ferret-Bench, a multimodal evaluation benchmark that tests joint tasks such as referring, grounding, semantic understanding, and knowledge reasoning. Ferret's research was accepted as a Spotlight paper at ICLR 2024, and subsequent versions Ferret-v2 and Ferret-UI further expand applications in vertical scenarios like UI understanding.

Core capabilities

Hybrid region representation

Combines points, boxes, free-form shapes and other spatial reference forms

Spatially-aware visual sampler

Efficiently extracts region-level visual features and links them to language

Open-vocabulary referring localization

Supports unseen object categories and descriptive references

End-to-end multimodal training

Jointly optimizes visual encoder and language model

GRIT instruction dataset

1.1M hierarchical annotations covering fine-grained localization and description

Ferret-Bench benchmark

Specifically evaluates the combined capability of referring, localization, and reasoning

Flexible deployment

Provides 7B and 13B model weights, compatible with the LLaVA ecosystem

Interactive demo

Gradio-based web UI supporting real-time referring and Q&A

Use cases

Point-and-click localization and attribute description of any object in an image

Selecting specific regions in an image based on text descriptions

Precise referring and diagnosis of lesion regions in medical images

Fine-grained recognition of road targets in autonomous driving scenes

Detailed queries about product components in e-commerce images

Locating interactive UI elements and understanding their functions

Complex reasoning in visual question answering that requires referring support