Tribev2 Brain Encoding

reason-machines/trending-skills/skills/tribev2-brain-encoding

作者 reason-machines2384a003145a無授權條款83 個星標收錄於 2026年10月8日更新於 2026年10月8日儲存庫3 個月前更新

Use TRIBE v2, Meta's multimodal foundation model for predicting fMRI brain responses to video, audio, and text stimuli

僅含說明Research & AnalysisData & Analytics
AI 產生的概覽

使用 Meta 的 TRIBE v2 多模態模型預測影片、音訊與文字刺激引發的 fMRI 腦部反應。

功能
此技能說明如何使用 TRIBE v2 這個多模態基礎模型，將影片、音訊與文字表徵對應到皮質表面（fsaverage5，約 2 萬個頂點）。內容涵蓋載入預訓練權重、從媒體檔案建立事件資料框、執行推論以產生預測陣列，以及在腦部表面視覺化結果。它也說明如何使用 PyTorch Lightning 與 Slurm 網格搜尋從頭訓練，以及 ROI 擷取與表面投影輔助函式。
適用情境
當你需要根據影片、音訊或旁白等自然刺激預測皮質活動時使用。它適合計算神經科學實驗、fMRI 編碼建模，以及針對特定腦區預測反應的分析。
執行需求
需要安裝 tribev2 套件的 Python 環境、PyTorch，以及用於下載預訓練權重的 HuggingFace 存取權；LLaMA 3.2 需要 HuggingFace 讀取權杖。選用套件提供繪圖（PyVista、Nilearn）與訓練（PyTorch Lightning、W&B）功能。訓練使用 Slurm 以及資料、輸出與分割區路徑的環境變數。下載權重需要網路存取。此技能未附指令碼，僅為說明文件。

TRIBE v2 Brain Encoding Model

Skill by ara.so — Daily 2026 Skills collection

TRIBE v2 is Meta's multimodal foundation model that predicts fMRI brain responses to naturalistic stimuli (video, audio, text). It combines LLaMA 3.2 (text), V-JEPA2 (video), and Wav2Vec-BERT (audio) encoders into a unified Transformer architecture that maps multimodal representations onto the cortical surface (fsaverage5, ~20k vertices).

Installation

bash
# Inference onlypip install -e .
# With brain visualization (PyVista & Nilearn)pip install -e ".[plotting]"
# Full training dependencies (PyTorch Lightning, W&B, etc.)pip install -e ".[training]"

Quick Start — Inference

Load pretrained model and predict from video

python
from tribev2 import TribeModel
# Load from HuggingFace (downloads weights to cache)model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
# Build events dataframe from a video filedf = model.get_events_dataframe(video_path="path/to/video.mp4")
# Predict brain responsespreds, segments = model.predict(events=df)print(preds.shape)  # (n_timesteps, n_vertices) on fsaverage5

Multimodal input — video + audio + text

python
from tribev2 import TribeModel
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
# All modalities together (text is auto-converted to speech and transcribed)df = model.get_events_dataframe(    video_path="path/to/video.mp4",    audio_path="path/to/audio.wav",   # optional, overrides video audio    text_path="path/to/script.txt",   # optional, auto-timed)
preds, segments = model.predict(events=df)print(preds.shape)  # (n_timesteps, n_vertices)

Text-only prediction

python
from tribev2 import TribeModel
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
df = model.get_events_dataframe(text_path="path/to/narration.txt")preds, segments = model.predict(events=df)

Brain Visualization

python
from tribev2 import TribeModelfrom tribev2.plotting import plot_brain_surface
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")df = model.get_events_dataframe(video_path="path/to/video.mp4")preds, segments = model.predict(events=df)
# Plot a single timepoint on the cortical surfaceplot_brain_surface(preds[0], backend="nilearn")   # or backend="pyvista"

Training a Model from Scratch

1. Set environment variables

bash
export DATAPATH="/path/to/studies"export SAVEPATH="/path/to/output"export SLURM_PARTITION="your_slurm_partition"

2. Authenticate with HuggingFace (required for LLaMA 3.2)

bash
huggingface-cli login# Paste a HuggingFace read token when prompted# Request access at: https://huggingface.co/meta-llama/Llama-3.2-3B

3. Local test run

bash
python -m tribev2.grids.test_run

4. Full grid search on Slurm

bash
# Cortical surface modelpython -m tribev2.grids.run_cortical
# Subcortical regionspython -m tribev2.grids.run_subcortical

Key API — TribeModel

python
from tribev2 import TribeModel
# Load pretrained weightsmodel = TribeModel.from_pretrained(    "facebook/tribev2",    cache_folder="./cache"  # local cache for HuggingFace weights)
# Build events dataframe (word-level timings, chunking, etc.)df = model.get_events_dataframe(    video_path=None,   # str path to .mp4    audio_path=None,   # str path to .wav    text_path=None,    # str path to .txt)
# Run predictionpreds, segments = model.predict(events=df)# preds: np.ndarray of shape (n_timesteps, n_vertices)# segments: list of segment metadata dicts

Project Structure

tribev2/├── main.py              # Experiment pipeline: Data, TribeExperiment├── model.py             # FmriEncoder: Transformer multimodal→fMRI model├── pl_module.py         # PyTorch Lightning training module├── demo_utils.py        # TribeModel and inference helpers├── eventstransforms.py  # Event transforms (word extraction, chunking)├── utils.py             # Multi-study loading, splitting, subject weighting├── utils_fmri.py        # Surface projection (MNI / fsaverage) and ROI analysis├── grids/│   ├── defaults.py      # Full default experiment configuration│   └── test_run.py      # Quick local test entry point├── plotting/            # Brain visualization backends└── studies/             # Dataset definitions (Algonauts2025, Lahner2024, …)

Configuration — Defaults

Edit tribev2/grids/defaults.py or set environment variables:

python
# tribev2/grids/defaults.py (key fields){    "datapath": "/path/to/studies",       # override with DATAPATH env var    "savepath": "/path/to/output",        # override with SAVEPATH env var    "slurm_partition": "learnfair",       # override with SLURM_PARTITION env var    "model": "FmriEncoder",    "modalities": ["video", "audio", "text"],    "surface": "fsaverage5",              # ~20k vertices}

Custom Experiment with PyTorch Lightning

python
from tribev2.main import Data, TribeExperimentfrom tribev2.pl_module import TribePLModuleimport pytorch_lightning as pl
# Configure experimentexperiment = TribeExperiment(    datapath="/path/to/studies",    savepath="/path/to/output",    modalities=["video", "audio", "text"],)
data = Data(experiment)module = TribePLModule(experiment)
trainer = pl.Trainer(    max_epochs=50,    accelerator="gpu",    devices=4,)trainer.fit(module, data)

Working with fMRI Surfaces

python
from tribev2.utils_fmri import project_to_fsaverage, get_roi_mask
# Project MNI coordinates to fsaverage5 surfacesurface_data = project_to_fsaverage(mni_data, target="fsaverage5")
# Get a specific ROI mask (e.g., early visual cortex)roi_mask = get_roi_mask(roi_name="V1", surface="fsaverage5")v1_responses = preds[:, roi_mask]print(v1_responses.shape)  # (n_timesteps, n_v1_vertices)

Common Patterns

Batch prediction over multiple videos

python
from tribev2 import TribeModelimport numpy as np
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
video_paths = ["video1.mp4", "video2.mp4", "video3.mp4"]all_predictions = []
for vp in video_paths:    df = model.get_events_dataframe(video_path=vp)    preds, segments = model.predict(events=df)    all_predictions.append(preds)
# all_predictions: list of (n_timesteps_i, n_vertices) arrays

Extract predictions for specific brain region

python
from tribev2 import TribeModelfrom tribev2.utils_fmri import get_roi_mask
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")df = model.get_events_dataframe(video_path="video.mp4")preds, segments = model.predict(events=df)
# Focus on auditory cortexac_mask = get_roi_mask("auditory_cortex", surface="fsaverage5")auditory_responses = preds[:, ac_mask]  # (n_timesteps, n_ac_vertices)

Access segment timing metadata

python
preds, segments = model.predict(events=df)
for i, seg in enumerate(segments):    print(f"Segment {i}: onset={seg['onset']:.2f}s, duration={seg['duration']:.2f}s")    print(f"  Brain response shape: {preds[i].shape}")

Troubleshooting

LLaMA 3.2 access denied

bash
# Must request access at https://huggingface.co/meta-llama/Llama-3.2-3B# Then authenticate:huggingface-cli login# Use a HuggingFace token with read permissions

CUDA out of memory during inference

python
# Use CPU for inference on smaller machinesimport torchmodel = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")model.to("cpu")

Missing visualization dependencies

bash
pip install -e ".[plotting]"# Installs pyvista and nilearn backends

Slurm training not submitting

bash
# Check env vars are setecho $DATAPATH $SAVEPATH $SLURM_PARTITION# Or edit tribev2/grids/defaults.py directly

Video without audio track causes error

python
# Provide audio separately or use text-only modedf = model.get_events_dataframe(    video_path="silent_video.mp4",    audio_path="separate_audio.wav",)

Citation

bibtex
@article{dAscoli2026TribeV2,  title={A foundation model of vision, audition, and language for in-silico neuroscience},  author={d'Ascoli, St{\'e}phane and Rapin, J{\'e}r{\'e}my and Benchetrit, Yohann and Brookes, Teon          and Begany, Katelyn and Raugel, Jos{\'e}phine and Banville, Hubert and King, Jean-R{\'e}mi},  year={2026}}

Resources

來源與署名

來源：reason-machines/trending-skills位於skills/tribev2-brain-encoding提交2384a00

授權條款： 無授權條款

內容歸原作者所有。SourceWeft 從公開儲存庫中收錄這些內容。

檢舉或申請下架

更多來自 reason-machines/trending-skills 的技能

Yourvpndead Vpn Detection

reason-machines

介紹一款 Android 應用程式，用來偵測 VPN 用戶端並探測本機 SOCKS5 與 Clash API 連接埠是否外洩。

Security833 個月前更新

X Mentor Skill Nuwa

reason-machines

提供 X（Twitter）內容策略指引：推文與長推撰寫、鉤子評論、成長策略與帳號診斷。

Marketing & Sales833 個月前更新

Wildworld Dataset

reason-machines

介紹 WildWorld 動作條件 ARPG 世界建模資料集，並提供預期的載入、篩選與基準評測程式碼。

Data & Analytics833 個月前更新

Understand Anything Knowledge Graph

reason-machines

為程式碼庫建立可互動的知識圖譜，並支援視覺化探索、搜尋與提問。

Software Development833 個月前更新

Webphysics Avbd Engine

reason-machines

指導將實驗性 WebGPU AVBD 剛體與軟體物理引擎整合到瀏覽器專案中。

Software Development833 個月前更新

Ui Ux Pro Max Skill

reason-machines

依產品描述產生 UI/UX 設計系統，包含頁面結構、風格、配色、字體搭配與檢查清單。

Design & Creative833 個月前更新

更多Research & Analysis技能

Equity Research

anthropics

精選

根據分析師共識預估、公司基本面、歷史價格與總體經濟資料，產出結構化的股票研究快照。

Research & Analysis39K2 週前更新

Sector Overview

anthropics

精選

產生產業與類股全景報告，涵蓋市場規模、競爭定位、估值與投資啟示。

Research & Analysis39K2 週前更新

Idea Generation

anthropics

精選

透過量化篩選、主題研究與模式辨識，系統性地篩選股票並發掘投資想法。

Research & Analysis39K2 週前更新

Competitive Analysis

anthropics

精選

建立競爭格局簡報：市場定位、競爭對手深入分析、比較表格與策略綜合。

Research & Analysis39K2 週前更新

Notion Research Documentation

openai

精選

檢索 Notion 頁面並將發現綜整為附引用的簡報、比較或報告，再發佈回 Notion。

Research & Analysis27K4 週前更新

Notion Meeting Intelligence

openai

精選

運用 Notion 脈絡與研究準備會議議程和會前讀物，並依與會者和會議類型調整內容。

Research & Analysis27K4 週前更新