Gemma Tuner Multimodal

by reason-machines2384a003145aNo licenseListed Oct 8, 2026Updated Oct 8, 2026

Fine-tune Gemma 4 and 3n models with audio, images, and text on Apple Silicon using PyTorch and Metal Performance Shaders.

Instructions onlyAI & Agents

AI-generated overview

Fine-tunes Gemma 3n and Gemma 4 models with text, image, or audio data on Apple Silicon using LoRA and MPS.

What it does: Guides fine-tuning of Gemma 3n and Gemma 4 models on Apple Silicon via PyTorch and Metal Performance Shaders, covering text instruction or completion tuning, image captioning and VQA, and audio transcription. It uses LoRA adapters configured through a hierarchical INI file, supports streaming large datasets from GCS or BigQuery, and produces training runs with checkpoints, metrics, and exportable merged Hugging Face SafeTensors weights. It also documents CLI commands for system checks, dataset preparation, evaluation, and troubleshooting.
When to use it: Use it when you want to fine-tune Gemma multimodal models locally on a Mac without an NVIDIA GPU. It suits LoRA training on local CSV datasets of text, images, or audio, and large cloud-hosted datasets streamed from GCS or BigQuery.
Requirements: macOS 12.3+ on Apple Silicon (arm64), Python 3.10+ installed natively, PyTorch and torchaudio, the gemma-tuner-multimodal package, and a Hugging Face account with Gemma access (HF_TOKEN or huggingface-cli login). Gemma 4 support needs an additional requirements file and a separate virtual environment. GCS/BigQuery streaming needs Google application credentials and network access. The skill is instructions only and ships no scripts.

Gemma Multimodal Fine-Tuner

Skill by ara.so — Daily 2026 Skills collection.

Fine-tune Gemma 4 and Gemma 3n models on text, images, and audio data entirely on Apple Silicon (MPS), with support for streaming large datasets from GCS/BigQuery without filling local storage.

What It Does

Text LoRA: instruction-tuning or completion fine-tuning from local CSV
Image + Text LoRA: captioning and VQA from local CSV
Audio + Text LoRA: the only Apple-Silicon-native path for this modality
Cloud streaming: train on terabytes from GCS/BigQuery without local copy
MPS-native: no NVIDIA GPU required — runs on MacBook Pro/Air/Mac Studio

Installation

Prerequisites

macOS 12.3+ with Apple Silicon (arm64)
Python 3.10+ (native arm64, not Rosetta)
Hugging Face account with Gemma access

bash

# Install Python 3.12 if neededbrew install [email protected]
# Create venvpython3.12 -m venv .venvsource .venv/bin/activate
# Verify arm64 (must show arm64, not x86_64)python -c "import platform; print(platform.machine())"
# Install PyTorchpip install torch torchaudio
# Clone and installgit clone https://github.com/mattmireles/gemma-tuner-multimodalcd gemma-tuner-multimodalpip install -e .
# For Gemma 4 support (separate venv recommended)pip install -r requirements/requirements-gemma4.txt

# Install Python 3.12 if neededbrew install [email protected]
# Create venvpython3.12 -m venv .venvsource .venv/bin/activate
# Verify arm64 (must show arm64, not x86_64)python -c "import platform; print(platform.machine())"
# Install PyTorchpip install torch torchaudio
# Clone and installgit clone https://github.com/mattmireles/gemma-tuner-multimodalcd gemma-tuner-multimodalpip install -e .
# For Gemma 4 support (separate venv recommended)pip install -r requirements/requirements-gemma4.txt

Authenticate with Hugging Face

CLI Commands

bash

# Check system is readygemma-macos-tuner system-check
# Guided setup wizard (recommended for first run)gemma-macos-tuner wizard
# Prepare datasetgemma-macos-tuner prepare <dataset-profile>
# Fine-tune a modelgemma-macos-tuner finetune <profile> --json-logging
# Evaluate a rungemma-macos-tuner evaluate <profile-or-run>
# Export merged HF/SafeTensors (merges LoRA when adapter_config.json present)gemma-macos-tuner export <run-dir-or-profile>
# Blacklist bad samples from errorsgemma-macos-tuner blacklist <profile>
# List training runsgemma-macos-tuner runs list

# Check system is readygemma-macos-tuner system-check
# Guided setup wizard (recommended for first run)gemma-macos-tuner wizard
# Prepare datasetgemma-macos-tuner prepare <dataset-profile>
# Fine-tune a modelgemma-macos-tuner finetune <profile> --json-logging
# Evaluate a rungemma-macos-tuner evaluate <profile-or-run>
# Export merged HF/SafeTensors (merges LoRA when adapter_config.json present)gemma-macos-tuner export <run-dir-or-profile>
# Blacklist bad samples from errorsgemma-macos-tuner blacklist <profile>
# List training runsgemma-macos-tuner runs list

Configuration (`config/config.ini`)

The config is hierarchical INI: defaults → groups → models → datasets → profiles.

ini

[defaults]output_dir = outputbatch_size = 2gradient_accumulation_steps = 8learning_rate = 2e-4num_train_epochs = 3
[model:gemma-3n-e2b-it]group = gemmabase_model = google/gemma-3n-E2B-it
[model:gemma-4-e2b-it]group = gemmabase_model = google/gemma-4-E2B-it
[dataset:my-audio-dataset]data_dir = data/datasets/my-audio-datasetaudio_column = audio_pathtext_column = transcript
[profile:my-audio-profile]model = gemma-3n-e2b-itdataset = my-audio-datasetmodality = audiolora_r = 16lora_alpha = 32lora_dropout = 0.05max_seq_length = 512

[defaults]output_dir = outputbatch_size = 2gradient_accumulation_steps = 8learning_rate = 2e-4num_train_epochs = 3
[model:gemma-3n-e2b-it]group = gemmabase_model = google/gemma-3n-E2B-it
[model:gemma-4-e2b-it]group = gemmabase_model = google/gemma-4-E2B-it
[dataset:my-audio-dataset]data_dir = data/datasets/my-audio-datasetaudio_column = audio_pathtext_column = transcript
[profile:my-audio-profile]model = gemma-3n-e2b-itdataset = my-audio-datasetmodality = audiolora_r = 16lora_alpha = 32lora_dropout = 0.05max_seq_length = 512

Use GEMMA_TUNER_CONFIG env var to point to config outside repo root:

Modality Configuration

Text-Only Fine-Tuning

Instruction tuning (user/assistant pairs):

ini

[profile:text-instruction]model = gemma-3n-e2b-itdataset = my-text-datasetmodality = texttext_sub_mode = instructionprompt_column = prompttext_column = responsemax_seq_length = 2048lora_r = 16lora_alpha = 32

[profile:text-instruction]model = gemma-3n-e2b-itdataset = my-text-datasetmodality = texttext_sub_mode = instructionprompt_column = prompttext_column = responsemax_seq_length = 2048lora_r = 16lora_alpha = 32

Completion tuning (full sequence trained):

ini

[profile:text-completion]model = gemma-3n-e2b-itdataset = my-text-datasetmodality = texttext_sub_mode = completiontext_column = textmax_seq_length = 2048

[profile:text-completion]model = gemma-3n-e2b-itdataset = my-text-datasetmodality = texttext_sub_mode = completiontext_column = textmax_seq_length = 2048

CSV format for instruction tuning (data/datasets/my-text-dataset/train.csv):

csv

prompt,response"What is photosynthesis?","Photosynthesis is the process by which plants...""Explain LoRA fine-tuning","LoRA (Low-Rank Adaptation) is a parameter-efficient..."

prompt,response"What is photosynthesis?","Photosynthesis is the process by which plants...""Explain LoRA fine-tuning","LoRA (Low-Rank Adaptation) is a parameter-efficient..."

Image Fine-Tuning

ini

[profile:image-caption]model = gemma-3n-e2b-itdataset = my-image-datasetmodality = imageimage_sub_mode = captioningimage_token_budget = 256prompt_column = prompttext_column = captionmax_seq_length = 512

[profile:image-caption]model = gemma-3n-e2b-itdataset = my-image-datasetmodality = imageimage_sub_mode = captioningimage_token_budget = 256prompt_column = prompttext_column = captionmax_seq_length = 512

CSV format (data/datasets/my-image-dataset/train.csv):

csv

image_path,prompt,caption/data/images/img1.jpg,Describe this image,A dog sitting on a green lawn.../data/images/img2.jpg,What is shown here,A bar chart showing quarterly revenue...

image_path,prompt,caption/data/images/img1.jpg,Describe this image,A dog sitting on a green lawn.../data/images/img2.jpg,What is shown here,A bar chart showing quarterly revenue...

Audio Fine-Tuning

ini

[profile:audio-asr]model = gemma-3n-e2b-itdataset = my-audio-datasetmodality = audioaudio_column = audio_pathtext_column = transcriptmax_seq_length = 512lora_r = 16lora_alpha = 32lora_dropout = 0.05

[profile:audio-asr]model = gemma-3n-e2b-itdataset = my-audio-datasetmodality = audioaudio_column = audio_pathtext_column = transcriptmax_seq_length = 512lora_r = 16lora_alpha = 32lora_dropout = 0.05

CSV format (data/datasets/my-audio-dataset/train.csv):

csv

audio_path,transcript/data/audio/recording1.wav,The patient presents with acute respiratory symptoms/data/audio/recording2.wav,Counsel objects to the characterization of the evidence

audio_path,transcript/data/audio/recording1.wav,The patient presents with acute respiratory symptoms/data/audio/recording2.wav,Counsel objects to the characterization of the evidence

Supported Models

Model Key	Hugging Face ID	Notes
`gemma-3n-e2b-it`	`google/gemma-3n-E2B-it`	Default, ~2B instruct
`gemma-3n-e4b-it`	`google/gemma-3n-E4B-it`	~4B instruct
`gemma-4-e2b-it`	`google/gemma-4-E2B-it`	Needs requirements-gemma4.txt
`gemma-4-e4b-it`	`google/gemma-4-E4B-it`	Needs requirements-gemma4.txt
`gemma-4-e2b`	`google/gemma-4-E2B`	Base, needs Gemma 4 stack
`gemma-4-e4b`	`google/gemma-4-E4B`	Base, needs Gemma 4 stack

Add custom models with a [model:your-name] section using group = gemma.

Dataset Directory Layout

data/└── datasets/    └── <dataset-name>/        ├── train.csv       # required        ├── validation.csv  # optional        └── test.csv        # optional

data/└── datasets/    └── <dataset-name>/        ├── train.csv       # required        ├── validation.csv  # optional        └── test.csv        # optional

Output Layout

output/└── {run-id}-{profile}/    ├── metadata.json    ├── metrics.json    ├── checkpoint-*/    └── adapter_model/      # LoRA artifacts

output/└── {run-id}-{profile}/    ├── metadata.json    ├── metrics.json    ├── checkpoint-*/    └── adapter_model/      # LoRA artifacts

Python API Examples

Running Fine-Tuning Programmatically

python

from gemma_tuner.core.config import load_configfrom gemma_tuner.core.ops import run_finetune
# Load configconfig = load_config("config/config.ini")
# Run fine-tuning for a profilerun_finetune(profile="my-audio-profile", config=config, json_logging=True)

from gemma_tuner.core.config import load_configfrom gemma_tuner.core.ops import run_finetune
# Load configconfig = load_config("config/config.ini")
# Run fine-tuning for a profilerun_finetune(profile="my-audio-profile", config=config, json_logging=True)

Using Device Utilities

python

from gemma_tuner.utils.device import get_device, memory_hint
device = get_device()   # Returns "mps", "cuda", or "cpu"print(f"Training on: {device}")
hint = memory_hint(model_key="gemma-3n-e2b-it")print(hint)

from gemma_tuner.utils.device import get_device, memory_hint
device = get_device()   # Returns "mps", "cuda", or "cpu"print(f"Training on: {device}")
hint = memory_hint(model_key="gemma-3n-e2b-it")print(hint)

Loading and Inspecting Datasets

python

from gemma_tuner.utils.dataset_utils import load_csv_dataset
train_df, val_df = load_csv_dataset(    data_dir="data/datasets/my-text-dataset",    text_column="response",    prompt_column="prompt")print(f"Train samples: {len(train_df)}, Val samples: {len(val_df)}")

from gemma_tuner.utils.dataset_utils import load_csv_dataset
train_df, val_df = load_csv_dataset(    data_dir="data/datasets/my-text-dataset",    text_column="response",    prompt_column="prompt")print(f"Train samples: {len(train_df)}, Val samples: {len(val_df)}")

Custom LoRA Config

python

from peft import LoraConfig, get_peft_modelfrom transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(    "google/gemma-3n-E2B-it",    torch_dtype="auto",    device_map="mps")
lora_config = LoraConfig(    r=16,    lora_alpha=32,    lora_dropout=0.05,    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],    task_type="CAUSAL_LM")
model = get_peft_model(model, lora_config)model.print_trainable_parameters()

from peft import LoraConfig, get_peft_modelfrom transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(    "google/gemma-3n-E2B-it",    torch_dtype="auto",    device_map="mps")
lora_config = LoraConfig(    r=16,    lora_alpha=32,    lora_dropout=0.05,    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],    task_type="CAUSAL_LM")
model = get_peft_model(model, lora_config)model.print_trainable_parameters()

Common Patterns

Full Workflow: Text Instruction Tuning

bash

# 1. Prepare your datamkdir -p data/datasets/my-datasetcp train.csv data/datasets/my-dataset/cp validation.csv data/datasets/my-dataset/
# 2. Add profile to config/config.inicat >> config/config.ini << 'EOF'[dataset:my-dataset]data_dir = data/datasets/my-dataset
[profile:my-text-run]model = gemma-3n-e2b-itdataset = my-datasetmodality = texttext_sub_mode = instructionprompt_column = prompttext_column = responsemax_seq_length = 2048lora_r = 16lora_alpha = 32EOF
# 3. Prepare datasetgemma-macos-tuner prepare my-dataset
# 4. Fine-tunegemma-macos-tuner finetune my-text-run --json-logging
# 5. Export merged weightsgemma-macos-tuner export my-text-run

# 1. Prepare your datamkdir -p data/datasets/my-datasetcp train.csv data/datasets/my-dataset/cp validation.csv data/datasets/my-dataset/
# 2. Add profile to config/config.inicat >> config/config.ini << 'EOF'[dataset:my-dataset]data_dir = data/datasets/my-dataset
[profile:my-text-run]model = gemma-3n-e2b-itdataset = my-datasetmodality = texttext_sub_mode = instructionprompt_column = prompttext_column = responsemax_seq_length = 2048lora_r = 16lora_alpha = 32EOF
# 3. Prepare datasetgemma-macos-tuner prepare my-dataset
# 4. Fine-tunegemma-macos-tuner finetune my-text-run --json-logging
# 5. Export merged weightsgemma-macos-tuner export my-text-run

GCS Streaming for Large Datasets

ini

[dataset:large-audio-gcs]source = gcsgcs_bucket = my-bucketgcs_prefix = audio-training-data/audio_column = audio_pathtext_column = transcript
[profile:large-audio-run]model = gemma-3n-e4b-itdataset = large-audio-gcsmodality = audiolora_r = 32lora_alpha = 64

[dataset:large-audio-gcs]source = gcsgcs_bucket = my-bucketgcs_prefix = audio-training-data/audio_column = audio_pathtext_column = transcript
[profile:large-audio-run]model = gemma-3n-e4b-itdataset = large-audio-gcsmodality = audiolora_r = 32lora_alpha = 64

Set credentials:

bash

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.jsongemma-macos-tuner finetune large-audio-run

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.jsongemma-macos-tuner finetune large-audio-run

Add a Custom Gemma Checkpoint

ini

[model:my-custom-gemma]group = gemmabase_model = my-org/my-gemma-checkpoint
[profile:custom-run]model = my-custom-gemmadataset = my-datasetmodality = texttext_sub_mode = instruction

[model:my-custom-gemma]group = gemmabase_model = my-org/my-gemma-checkpoint
[profile:custom-run]model = my-custom-gemmadataset = my-datasetmodality = texttext_sub_mode = instruction

Troubleshooting

Wrong architecture (x86_64 instead of arm64)

bash

python -c "import platform; print(platform.machine())"# Must be arm64 — if x86_64, reinstall Python natively:brew install [email protected]python3.12 -m venv .venv && source .venv/bin/activate

python -c "import platform; print(platform.machine())"# Must be arm64 — if x86_64, reinstall Python natively:brew install [email protected]python3.12 -m venv .venv && source .venv/bin/activate

MPS out of memory

Reduce batch_size (try 1)
Increase gradient_accumulation_steps to compensate
Use a smaller model (e2b instead of e4b)
Reduce max_seq_length

Gemma 4 model not loading

bash

# Gemma 4 requires the updated Transformers stackpip install -r requirements/requirements-gemma4.txt# Use a separate venv if you also need Gemma 3n

# Gemma 4 requires the updated Transformers stackpip install -r requirements/requirements-gemma4.txt# Use a separate venv if you also need Gemma 3n

Config not found outside repo root

bash

export GEMMA_TUNER_CONFIG=/absolute/path/to/config/config.inigemma-macos-tuner finetune my-profile

export GEMMA_TUNER_CONFIG=/absolute/path/to/config/config.inigemma-macos-tuner finetune my-profile

Hugging Face auth errors

bash

huggingface-cli login# Or:export HF_TOKEN=your_hf_token# Accept Gemma license at: https://huggingface.co/google/gemma-3n-E2B-it

huggingface-cli login# Or:export HF_TOKEN=your_hf_token# Accept Gemma license at: https://huggingface.co/google/gemma-3n-E2B-it

System check before debugging anything else

Audio tower loaded even for text-only runs

This is a known v1 issue — USM audio tower weights stay in memory even for modality = text. See README/KNOWN_ISSUES.md. Workaround: use a smaller model variant to stay within RAM budget.

Architecture Reference

File	Role
`gemma_tuner/cli_typer.py`	Main CLI entrypoint (`gemma-macos-tuner`)
`gemma_tuner/core/ops.py`	Dispatches prepare/finetune/evaluate/export
`gemma_tuner/scripts/finetune.py`	Router: Gemma models → `models/gemma/finetune.py`
`gemma_tuner/models/gemma/finetune.py`	Core training loop with LoRA
`gemma_tuner/scripts/export.py`	Merges LoRA → HF/SafeTensors tree
`gemma_tuner/utils/device.py`	MPS/CUDA/CPU selection and memory hints
`gemma_tuner/utils/dataset_utils.py`	CSV loading, blacklist/protection semantics
`gemma_tuner/wizard/`	Interactive CLI wizard (questionary + Rich)
`config/config.ini`	Hierarchical INI configuration

Source and attribution

Source:reason-machines/trending-skillsinskills/gemma-tuner-multimodalat commit2384a00

License: No license

Content belongs to its original authors. SourceWeft indexes it from a public repository.

Report or request removal

Gemma Tuner Multimodal

Gemma Multimodal Fine-Tuner

What It Does

Installation

Prerequisites

Authenticate with Hugging Face

CLI Commands

Configuration (config/config.ini)

Modality Configuration

Text-Only Fine-Tuning

Image Fine-Tuning

Audio Fine-Tuning

Supported Models

Dataset Directory Layout

Output Layout

Python API Examples

Running Fine-Tuning Programmatically

Using Device Utilities

Loading and Inspecting Datasets

Custom LoRA Config

Common Patterns

Full Workflow: Text Instruction Tuning

GCS Streaming for Large Datasets

Add a Custom Gemma Checkpoint

Troubleshooting

Wrong architecture (x86_64 instead of arm64)

MPS out of memory

Gemma 4 model not loading

Config not found outside repo root

Hugging Face auth errors

System check before debugging anything else

Audio tower loaded even for text-only runs

Architecture Reference

Source and attribution

More from reason-machines/trending-skills

Tailslayer Dram Hedged Reads

Taiwan Equity Research Coverage

Sheets Terminal Spreadsheet

Slug Font Rendering

Rattles Terminal Spinners

See Through Anime Layer Decomposition

More in AI & Agents

Skill Development

Plugin Structure

Command Development

Claude Md Improver

Find Skills

Plugin Creator

Configuration (`config/config.ini`)