Llava

orchestra-research/ai-research-skills/18-multimodal/llava

by orchestra-research773a52944ba4MIT13K starsListed Oct 8, 2026Updated Oct 8, 2026Repository updated 3 months ago

Large Language and Vision Assistant. Enables visual instruction tuning and image-based conversations. Combines CLIP vision encoder with Vicuna/LLaMA language models. Supports multi-turn image chat, visual question answering, and instruction following. Use for vision-language chatbots or image understanding tasks. Best for conversational image analysis.

Instructions onlyAI & Agents Software Development

AI-generated overview

Guides use of the LLaVA vision-language model for image chat, visual question answering and custom training.

What it does: Documents how to install and run the open-source LLaVA vision-language model, which pairs a CLIP vision encoder with Vicuna or LLaMA language models. It covers loading pretrained checkpoints, single-image and multi-turn image conversations, image captioning, visual question answering, document understanding, CLI and Gradio interfaces, 4-bit and 8-bit quantization, and two-stage training of custom models. It also lists model sizes, VRAM and speed figures, benchmark scores, limitations and framework integrations.
When to use it: Use it when building vision-language chatbots or image understanding features, running visual question answering or image captioning, or fine-tuning a LLaVA model on custom image-instruction data. It is also useful for choosing a model size or quantization level under GPU memory limits.
Requirements: Requires Python with transformers, torch and pillow, a GPU for practical inference, and network access to download model weights from Hugging Face. The skill is instructions only and ships no scripts; the training commands reference scripts in the upstream LLaVA repository.

LLaVA - Large Language and Vision Assistant

Open-source vision-language model for conversational image understanding.

When to use LLaVA

Use when:

Building vision-language chatbots
Visual question answering (VQA)
Image description and captioning
Multi-turn image conversations
Visual instruction following
Document understanding with images

Metrics:

23,000+ GitHub stars
GPT-4V level capabilities (targeted)
Apache 2.0 License
Multiple model sizes (7B-34B params)

Use alternatives instead:

GPT-4V: Highest quality, API-based
CLIP: Simple zero-shot classification
BLIP-2: Better for captioning only
Flamingo: Research, not open-source

Quick start

Installation

bash

# Clone repositorygit clone https://github.com/haotian-liu/LLaVAcd LLaVA
# Installpip install -e .

# Clone repositorygit clone https://github.com/haotian-liu/LLaVAcd LLaVA
# Installpip install -e .

Basic usage

python

from llava.model.builder import load_pretrained_modelfrom llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_tokenfrom llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKENfrom llava.conversation import conv_templatesfrom PIL import Imageimport torch
# Load modelmodel_path = "liuhaotian/llava-v1.5-7b"tokenizer, model, image_processor, context_len = load_pretrained_model(    model_path=model_path,    model_base=None,    model_name=get_model_name_from_path(model_path))
# Load imageimage = Image.open("image.jpg")image_tensor = process_images([image], image_processor, model.config)image_tensor = image_tensor.to(model.device, dtype=torch.float16)
# Create conversationconv = conv_templates["llava_v1"].copy()conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")conv.append_message(conv.roles[1], None)prompt = conv.get_prompt()
# Generate responseinput_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).to(model.device)
with torch.inference_mode():    output_ids = model.generate(        input_ids,        images=image_tensor,        do_sample=True,        temperature=0.2,        max_new_tokens=512    )
response = tokenizer.decode(output_ids[0], skip_special_tokens=True).strip()print(response)

from llava.model.builder import load_pretrained_modelfrom llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_tokenfrom llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKENfrom llava.conversation import conv_templatesfrom PIL import Imageimport torch
# Load modelmodel_path = "liuhaotian/llava-v1.5-7b"tokenizer, model, image_processor, context_len = load_pretrained_model(    model_path=model_path,    model_base=None,    model_name=get_model_name_from_path(model_path))
# Load imageimage = Image.open("image.jpg")image_tensor = process_images([image], image_processor, model.config)image_tensor = image_tensor.to(model.device, dtype=torch.float16)
# Create conversationconv = conv_templates["llava_v1"].copy()conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")conv.append_message(conv.roles[1], None)prompt = conv.get_prompt()
# Generate responseinput_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).to(model.device)
with torch.inference_mode():    output_ids = model.generate(        input_ids,        images=image_tensor,        do_sample=True,        temperature=0.2,        max_new_tokens=512    )
response = tokenizer.decode(output_ids[0], skip_special_tokens=True).strip()print(response)

Available models

python

# Load different modelsmodel_7b = "liuhaotian/llava-v1.5-7b"model_13b = "liuhaotian/llava-v1.5-13b"model_34b = "liuhaotian/llava-v1.6-34b"
# 4-bit quantization for lower VRAMload_4bit = True  # Reduces VRAM by ~4×

# Load different modelsmodel_7b = "liuhaotian/llava-v1.5-7b"model_13b = "liuhaotian/llava-v1.5-13b"model_34b = "liuhaotian/llava-v1.6-34b"
# 4-bit quantization for lower VRAMload_4bit = True  # Reduces VRAM by ~4×

CLI usage

bash

# Single image querypython -m llava.serve.cli \    --model-path liuhaotian/llava-v1.5-7b \    --image-file image.jpg \    --query "What is in this image?"
# Multi-turn conversationpython -m llava.serve.cli \    --model-path liuhaotian/llava-v1.5-7b \    --image-file image.jpg# Then type questions interactively

# Single image querypython -m llava.serve.cli \    --model-path liuhaotian/llava-v1.5-7b \    --image-file image.jpg \    --query "What is in this image?"
# Multi-turn conversationpython -m llava.serve.cli \    --model-path liuhaotian/llava-v1.5-7b \    --image-file image.jpg# Then type questions interactively

Web UI (Gradio)

bash

# Launch Gradio interfacepython -m llava.serve.gradio_web_server \    --model-path liuhaotian/llava-v1.5-7b \    --load-4bit  # Optional: reduce VRAM
# Access at http://localhost:7860

# Launch Gradio interfacepython -m llava.serve.gradio_web_server \    --model-path liuhaotian/llava-v1.5-7b \    --load-4bit  # Optional: reduce VRAM
# Access at http://localhost:7860

Multi-turn conversations

python

# Initialize conversationconv = conv_templates["llava_v1"].copy()
# Turn 1conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")conv.append_message(conv.roles[1], None)response1 = generate(conv, model, image)  # "A dog playing in a park"
# Turn 2conv.messages[-1][1] = response1  # Add previous responseconv.append_message(conv.roles[0], "What breed is the dog?")conv.append_message(conv.roles[1], None)response2 = generate(conv, model, image)  # "Golden Retriever"
# Turn 3conv.messages[-1][1] = response2conv.append_message(conv.roles[0], "What time of day is it?")conv.append_message(conv.roles[1], None)response3 = generate(conv, model, image)

# Initialize conversationconv = conv_templates["llava_v1"].copy()
# Turn 1conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")conv.append_message(conv.roles[1], None)response1 = generate(conv, model, image)  # "A dog playing in a park"
# Turn 2conv.messages[-1][1] = response1  # Add previous responseconv.append_message(conv.roles[0], "What breed is the dog?")conv.append_message(conv.roles[1], None)response2 = generate(conv, model, image)  # "Golden Retriever"
# Turn 3conv.messages[-1][1] = response2conv.append_message(conv.roles[0], "What time of day is it?")conv.append_message(conv.roles[1], None)response3 = generate(conv, model, image)

Common tasks

Image captioning

Visual question answering

Object detection (textual)

python

question = "List all the objects you can see in this image."response = ask(model, image, question)

question = "List all the objects you can see in this image."response = ask(model, image, question)

Scene understanding

Document understanding

python

question = "What is the main topic of this document?"response = ask(model, document_image, question)

question = "What is the main topic of this document?"response = ask(model, document_image, question)

Training custom model

bash

# Stage 1: Feature alignment (558K image-caption pairs)bash scripts/v1_5/pretrain.sh
# Stage 2: Visual instruction tuning (150K instruction data)bash scripts/v1_5/finetune.sh

# Stage 1: Feature alignment (558K image-caption pairs)bash scripts/v1_5/pretrain.sh
# Stage 2: Visual instruction tuning (150K instruction data)bash scripts/v1_5/finetune.sh

Quantization (reduce VRAM)

python

# 4-bit quantizationtokenizer, model, image_processor, context_len = load_pretrained_model(    model_path="liuhaotian/llava-v1.5-13b",    model_base=None,    model_name=get_model_name_from_path("liuhaotian/llava-v1.5-13b"),    load_4bit=True  # Reduces VRAM ~4×)
# 8-bit quantizationload_8bit=True  # Reduces VRAM ~2×

# 4-bit quantizationtokenizer, model, image_processor, context_len = load_pretrained_model(    model_path="liuhaotian/llava-v1.5-13b",    model_base=None,    model_name=get_model_name_from_path("liuhaotian/llava-v1.5-13b"),    load_4bit=True  # Reduces VRAM ~4×)
# 8-bit quantizationload_8bit=True  # Reduces VRAM ~2×

Best practices

Start with 7B model - Good quality, manageable VRAM
Use 4-bit quantization - Reduces VRAM significantly
GPU required - CPU inference extremely slow
Clear prompts - Specific questions get better answers
Multi-turn conversations - Maintain conversation context
Temperature 0.2-0.7 - Balance creativity/consistency
max_new_tokens 512-1024 - For detailed responses
Batch processing - Process multiple images sequentially

Performance

On A100 GPU

Benchmarks

LLaVA achieves competitive scores on:

VQAv2: 78.5%
GQA: 62.0%
MM-Vet: 35.4%
MMBench: 64.3%

Limitations

Hallucinations - May describe things not in image
Spatial reasoning - Struggles with precise locations
Small text - Difficulty reading fine print
Object counting - Imprecise for many objects
VRAM requirements - Need powerful GPU
Inference speed - Slower than CLIP

Integration with frameworks

LangChain

python

from langchain.llms.base import LLM
class LLaVALLM(LLM):    def _call(self, prompt, stop=None):        # Custom LLaVA inference        return response
llm = LLaVALLM()

from langchain.llms.base import LLM
class LLaVALLM(LLM):    def _call(self, prompt, stop=None):        # Custom LLaVA inference        return response
llm = LLaVALLM()

Gradio App

python

import gradio as gr
def chat(image, text, history):    response = ask_llava(model, image, text)    return response
demo = gr.ChatInterface(    chat,    additional_inputs=[gr.Image(type="pil")],    title="LLaVA Chat")demo.launch()

import gradio as gr
def chat(image, text, history):    response = ask_llava(model, image, text)    return response
demo = gr.ChatInterface(    chat,    additional_inputs=[gr.Image(type="pil")],    title="LLaVA Chat")demo.launch()

Resources

GitHub: https://github.com/haotian-liu/LLaVA ⭐ 23,000+
Paper: https://arxiv.org/abs/2304.08485
Demo: https://llava.hliu.cc
Models: https://huggingface.co/liuhaotian
License: Apache 2.0

Source and attribution

Source:orchestra-research/ai-research-skillsin18-multimodal/llavaat commit773a529

License: MIT

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

More from orchestra-research/ai-research-skills

Speculative Decoding

orchestra-research

Guides accelerating LLM inference with speculative decoding, Medusa multi-head prediction, and lookahead decoding.

AI & Agents13Kupdated 3 months ago

Systems Paper Writing

orchestra-research

Guides paragraph-level structure and page budgets for writing systems conference papers such as OSDI, SOSP, ASPLOS and NSDI.

Writing & Content13Kupdated 3 months ago

Moe Training

orchestra-research

Guides training and inference of Mixture of Experts models with DeepSpeed or HuggingFace, covering routing, load balancing and expert parallelism.

AI & Agents13Kupdated 3 months ago

Presenting Conference Talks

orchestra-research

Generates conference presentation slides (Beamer LaTeX PDF and editable PPTX) from a compiled paper with speaker notes and talk script. Use when preparing oral talks, spotlight presentations, or invited talks for ML and systems conferences.

Awaiting classification13Kupdated 3 months ago

Model Pruning

orchestra-research

Guides LLM pruning with Wanda, SparseGPT, magnitude and N:M methods to cut model size and speed inference.

AI & Agents13Kupdated 3 months ago

Outlines

orchestra-research

Guides use of the Outlines library for structured, schema-constrained text generation with local models.

AI & Agents13Kupdated 3 months ago

Model	Parameters	VRAM	Quality
LLaVA-v1.5-7B	7B	~14 GB	Good
LLaVA-v1.5-13B	13B	~28 GB	Better
LLaVA-v1.6-34B	34B	~70 GB	Best

Model	VRAM (FP16)	VRAM (4-bit)	Speed (tokens/s)
7B	~14 GB	~4 GB	~20
13B	~28 GB	~8 GB	~12
34B	~70 GB	~18 GB	~5

Llava

LLaVA - Large Language and Vision Assistant

When to use LLaVA

Quick start

Installation

Basic usage

Available models

CLI usage

Web UI (Gradio)

Multi-turn conversations

Common tasks

Image captioning

Visual question answering

Object detection (textual)

Scene understanding

Document understanding

Training custom model

Quantization (reduce VRAM)

Best practices

Performance

Benchmarks

Limitations

Integration with frameworks

LangChain

Gradio App

Resources

Source and attribution

More from orchestra-research/ai-research-skills

Speculative Decoding

Systems Paper Writing

Moe Training

Presenting Conference Talks

Model Pruning

Outlines

More in AI & Agents

Ppt Template Creator

Skill Development

Plugin Structure

Command Development

Claude Md Improver

Find Skills