Tribev2 Brain Encoding

作者 reason-machines2384a003145a无许可证83 个星标收录于 2026年10月8日更新于 2026年10月8日仓库3个月前更新

Use TRIBE v2, Meta's multimodal foundation model for predicting fMRI brain responses to video, audio, and text stimuli

仅含说明Research & AnalysisData & Analytics
AI 生成的概览

使用 Meta 的 TRIBE v2 多模态模型预测视频、音频和文本刺激引发的 fMRI 脑响应。

功能
该技能说明如何使用 TRIBE v2 这一多模态基础模型，将视频、音频和文本表示映射到皮层表面（fsaverage5，约 2 万个顶点）。内容涵盖加载预训练权重、从媒体文件构建事件数据框、运行推理以生成预测数组，以及在脑表面上可视化结果。它还介绍使用 PyTorch Lightning 和 Slurm 网格搜索从零训练，以及 ROI 提取和表面投影辅助函数。
适用场景
当你需要根据视频、音频或旁白等自然刺激预测皮层活动时使用。它适用于计算神经科学实验、fMRI 编码建模，以及针对特定脑区预测响应的分析。
运行要求
需要安装 tribev2 包的 Python 环境、PyTorch，以及用于下载预训练权重的 HuggingFace 访问权限；LLaMA 3.2 需要 HuggingFace 读取令牌。可选依赖提供绘图（PyVista、Nilearn）和训练（PyTorch Lightning、W&B）功能。训练使用 Slurm 以及数据、输出和分区路径的环境变量。下载权重需要网络访问。该技能不附带脚本，仅为说明文档。

TRIBE v2 Brain Encoding Model

Skill by ara.so — Daily 2026 Skills collection

TRIBE v2 is Meta's multimodal foundation model that predicts fMRI brain responses to naturalistic stimuli (video, audio, text). It combines LLaMA 3.2 (text), V-JEPA2 (video), and Wav2Vec-BERT (audio) encoders into a unified Transformer architecture that maps multimodal representations onto the cortical surface (fsaverage5, ~20k vertices).

Installation

bash
# Inference onlypip install -e .
# With brain visualization (PyVista & Nilearn)pip install -e ".[plotting]"
# Full training dependencies (PyTorch Lightning, W&B, etc.)pip install -e ".[training]"

Quick Start — Inference

Load pretrained model and predict from video

python
from tribev2 import TribeModel
# Load from HuggingFace (downloads weights to cache)model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
# Build events dataframe from a video filedf = model.get_events_dataframe(video_path="path/to/video.mp4")
# Predict brain responsespreds, segments = model.predict(events=df)print(preds.shape)  # (n_timesteps, n_vertices) on fsaverage5

Multimodal input — video + audio + text

python
from tribev2 import TribeModel
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
# All modalities together (text is auto-converted to speech and transcribed)df = model.get_events_dataframe(    video_path="path/to/video.mp4",    audio_path="path/to/audio.wav",   # optional, overrides video audio    text_path="path/to/script.txt",   # optional, auto-timed)
preds, segments = model.predict(events=df)print(preds.shape)  # (n_timesteps, n_vertices)

Text-only prediction

python
from tribev2 import TribeModel
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
df = model.get_events_dataframe(text_path="path/to/narration.txt")preds, segments = model.predict(events=df)

Brain Visualization

python
from tribev2 import TribeModelfrom tribev2.plotting import plot_brain_surface
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")df = model.get_events_dataframe(video_path="path/to/video.mp4")preds, segments = model.predict(events=df)
# Plot a single timepoint on the cortical surfaceplot_brain_surface(preds[0], backend="nilearn")   # or backend="pyvista"

Training a Model from Scratch

1. Set environment variables

bash
export DATAPATH="/path/to/studies"export SAVEPATH="/path/to/output"export SLURM_PARTITION="your_slurm_partition"

2. Authenticate with HuggingFace (required for LLaMA 3.2)

bash
huggingface-cli login# Paste a HuggingFace read token when prompted# Request access at: https://huggingface.co/meta-llama/Llama-3.2-3B

3. Local test run

bash
python -m tribev2.grids.test_run

4. Full grid search on Slurm

bash
# Cortical surface modelpython -m tribev2.grids.run_cortical
# Subcortical regionspython -m tribev2.grids.run_subcortical

Key API — TribeModel

python
from tribev2 import TribeModel
# Load pretrained weightsmodel = TribeModel.from_pretrained(    "facebook/tribev2",    cache_folder="./cache"  # local cache for HuggingFace weights)
# Build events dataframe (word-level timings, chunking, etc.)df = model.get_events_dataframe(    video_path=None,   # str path to .mp4    audio_path=None,   # str path to .wav    text_path=None,    # str path to .txt)
# Run predictionpreds, segments = model.predict(events=df)# preds: np.ndarray of shape (n_timesteps, n_vertices)# segments: list of segment metadata dicts

Project Structure

tribev2/├── main.py              # Experiment pipeline: Data, TribeExperiment├── model.py             # FmriEncoder: Transformer multimodal→fMRI model├── pl_module.py         # PyTorch Lightning training module├── demo_utils.py        # TribeModel and inference helpers├── eventstransforms.py  # Event transforms (word extraction, chunking)├── utils.py             # Multi-study loading, splitting, subject weighting├── utils_fmri.py        # Surface projection (MNI / fsaverage) and ROI analysis├── grids/│   ├── defaults.py      # Full default experiment configuration│   └── test_run.py      # Quick local test entry point├── plotting/            # Brain visualization backends└── studies/             # Dataset definitions (Algonauts2025, Lahner2024, …)

Configuration — Defaults

Edit tribev2/grids/defaults.py or set environment variables:

python
# tribev2/grids/defaults.py (key fields){    "datapath": "/path/to/studies",       # override with DATAPATH env var    "savepath": "/path/to/output",        # override with SAVEPATH env var    "slurm_partition": "learnfair",       # override with SLURM_PARTITION env var    "model": "FmriEncoder",    "modalities": ["video", "audio", "text"],    "surface": "fsaverage5",              # ~20k vertices}

Custom Experiment with PyTorch Lightning

python
from tribev2.main import Data, TribeExperimentfrom tribev2.pl_module import TribePLModuleimport pytorch_lightning as pl
# Configure experimentexperiment = TribeExperiment(    datapath="/path/to/studies",    savepath="/path/to/output",    modalities=["video", "audio", "text"],)
data = Data(experiment)module = TribePLModule(experiment)
trainer = pl.Trainer(    max_epochs=50,    accelerator="gpu",    devices=4,)trainer.fit(module, data)

Working with fMRI Surfaces

python
from tribev2.utils_fmri import project_to_fsaverage, get_roi_mask
# Project MNI coordinates to fsaverage5 surfacesurface_data = project_to_fsaverage(mni_data, target="fsaverage5")
# Get a specific ROI mask (e.g., early visual cortex)roi_mask = get_roi_mask(roi_name="V1", surface="fsaverage5")v1_responses = preds[:, roi_mask]print(v1_responses.shape)  # (n_timesteps, n_v1_vertices)

Common Patterns

Batch prediction over multiple videos

python
from tribev2 import TribeModelimport numpy as np
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")
video_paths = ["video1.mp4", "video2.mp4", "video3.mp4"]all_predictions = []
for vp in video_paths:    df = model.get_events_dataframe(video_path=vp)    preds, segments = model.predict(events=df)    all_predictions.append(preds)
# all_predictions: list of (n_timesteps_i, n_vertices) arrays

Extract predictions for specific brain region

python
from tribev2 import TribeModelfrom tribev2.utils_fmri import get_roi_mask
model = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")df = model.get_events_dataframe(video_path="video.mp4")preds, segments = model.predict(events=df)
# Focus on auditory cortexac_mask = get_roi_mask("auditory_cortex", surface="fsaverage5")auditory_responses = preds[:, ac_mask]  # (n_timesteps, n_ac_vertices)

Access segment timing metadata

python
preds, segments = model.predict(events=df)
for i, seg in enumerate(segments):    print(f"Segment {i}: onset={seg['onset']:.2f}s, duration={seg['duration']:.2f}s")    print(f"  Brain response shape: {preds[i].shape}")

Troubleshooting

LLaMA 3.2 access denied

bash
# Must request access at https://huggingface.co/meta-llama/Llama-3.2-3B# Then authenticate:huggingface-cli login# Use a HuggingFace token with read permissions

CUDA out of memory during inference

python
# Use CPU for inference on smaller machinesimport torchmodel = TribeModel.from_pretrained("facebook/tribev2", cache_folder="./cache")model.to("cpu")

Missing visualization dependencies

bash
pip install -e ".[plotting]"# Installs pyvista and nilearn backends

Slurm training not submitting

bash
# Check env vars are setecho $DATAPATH $SAVEPATH $SLURM_PARTITION# Or edit tribev2/grids/defaults.py directly

Video without audio track causes error

python
# Provide audio separately or use text-only modedf = model.get_events_dataframe(    video_path="silent_video.mp4",    audio_path="separate_audio.wav",)

Citation

bibtex
@article{dAscoli2026TribeV2,  title={A foundation model of vision, audition, and language for in-silico neuroscience},  author={d'Ascoli, St{\'e}phane and Rapin, J{\'e}r{\'e}my and Benchetrit, Yohann and Brookes, Teon          and Begany, Katelyn and Raugel, Jos{\'e}phine and Banville, Hubert and King, Jean-R{\'e}mi},  year={2026}}

Resources

来源与署名

来源：reason-machines/trending-skills位于skills/tribev2-brain-encoding提交2384a00

许可证： 无许可证

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

更多来自 reason-machines/trending-skills 的技能

Yourvpndead Vpn Detection

reason-machines

介绍一款 Android 应用，用于检测 VPN 客户端并探测本机 SOCKS5 与 Clash API 端口是否泄露。

Security833个月前更新

X Mentor Skill Nuwa

reason-machines

提供 X（Twitter）内容策略指导：推文与长推写作、钩子点评、增长策略与账号诊断。

Marketing & Sales833个月前更新

Wildworld Dataset

reason-machines

介绍 WildWorld 动作条件 ARPG 世界建模数据集，并给出预期的加载、筛选与基准评测代码。

Data & Analytics833个月前更新

Understand Anything Knowledge Graph

reason-machines

为代码库构建可交互的知识图谱，并支持可视化探索、搜索和提问。

Software Development833个月前更新

Webphysics Avbd Engine

reason-machines

指导将实验性 WebGPU AVBD 刚体与软体物理引擎集成到浏览器项目中。

Software Development833个月前更新

Ui Ux Pro Max Skill

reason-machines

根据产品描述生成 UI/UX 设计系统，包括页面结构、风格、配色、字体搭配与检查清单。

Design & Creative833个月前更新

更多Research & Analysis技能

Equity Research

anthropics

精选

根据分析师一致预期、公司基本面、历史价格与宏观数据，生成结构化的股票研究快照。

Research & Analysis39K2周前更新

Sector Overview

anthropics

精选

生成行业与板块全景报告，涵盖市场规模、竞争定位、估值与投资启示。

Research & Analysis39K2周前更新

Idea Generation

anthropics

精选

通过量化筛选、主题研究和模式识别，系统性地筛选股票并挖掘投资想法。

Research & Analysis39K2周前更新

Competitive Analysis

anthropics

精选

构建竞争格局演示文稿：市场定位、竞争对手深度分析、对比表格与战略综合。

Research & Analysis39K2周前更新

Notion Research Documentation

openai

精选

检索 Notion 页面并将发现综合为带引用的简报、对比或报告，再发布回 Notion。

Research & Analysis27K4周前更新

Notion Meeting Intelligence

openai

精选

借助 Notion 上下文与研究准备会议议程和预读材料，并针对参会者和会议类型进行调整。

Research & Analysis27K4周前更新