Gemma Gem Browser Ai

作者 reason-machines2384a003145a无许可证83 个星标收录于 2026年10月8日更新于 2026年10月8日仓库3个月前更新

Build and extend Gemma Gem, an on-device AI browser assistant Chrome extension running Google's Gemma 4 model via WebGPU with no cloud dependencies.

仅含说明Software Development AI & Agents

AI 生成的概览

指导构建和扩展 Gemma Gem——一个基于 WebGPU、在设备端运行 Gemma 4 的 Chrome 扩展。

功能: 该技能说明 Gemma Gem 的架构、构建步骤与接口。Gemma Gem 是一个通过 WebGPU 在设备端运行 Google Gemma 4 模型的 Chrome 扩展。内容涵盖离屏文档、Service Worker 与内容脚本的分工，ModelBackend 与 ToolExecutor 接口、代理循环、内置 DOM 工具、消息路由、模型配置、设置持久化、Shadow DOM 聊天界面以及调试方法。它还演示如何通过定义 schema、注册并实现执行来新增工具。
适用场景: 适用于扩展 Gemma Gem、为其代理循环新增工具，或研究其 Chrome 扩展架构的场景。适合开发或调试设备端 LLM 浏览器助手的开发者。
运行要求: 需要 Node.js 18+ 与 pnpm 来安装和构建扩展，并需要 Chrome 113+ 且启用 WebGPU 与硬件加速。模型在首次使用时从 Hugging Face 下载并缓存在本地。该技能仅为说明文档，不附带脚本。

Gemma Gem Browser AI

Skill by ara.so — Daily 2026 Skills collection.

Gemma Gem is a Chrome extension that runs Google's Gemma 4 model entirely on-device via WebGPU. It injects a chat overlay into every page and exposes a tool-calling agent loop that can read pages, click elements, fill forms, execute JavaScript, and take screenshots — all without sending data to any server.

Architecture Overview

Offscreen Document          Service Worker           Content Script(Gemma 4 + Agent Loop)  <-> (Message Router)    <-> (Chat UI + DOM Tools)       |                         |  WebGPU inference          Screenshot capture  Token streaming           JS execution

Offscreen Document          Service Worker           Content Script(Gemma 4 + Agent Loop)  <-> (Message Router)    <-> (Chat UI + DOM Tools)       |                         |  WebGPU inference          Screenshot capture  Token streaming           JS execution

Offscreen document (offscreen/): Loads the ONNX model via @huggingface/transformers, runs the agent loop, streams tokens.
Service worker (background/): Routes messages, handles take_screenshot and run_javascript.
Content script (content/): Injects shadow DOM chat UI, executes DOM tools.
agent/: Zero-dependency module defining ModelBackend and ToolExecutor interfaces — extractable as a standalone library.

Install & Build

bash

# Prerequisites: Node.js 18+, pnpmpnpm install
# Development build (logging active, source maps)pnpm build
# Production build (errors only, minified)pnpm build:prod

# Prerequisites: Node.js 18+, pnpmpnpm install
# Development build (logging active, source maps)pnpm build
# Production build (errors only, minified)pnpm build:prod

Load the extension:

Open chrome://extensions
Enable Developer mode
Click Load unpacked → select .output/chrome-mv3-dev/

Model download happens automatically on first chat open:

onnx-community/gemma-4-E2B-it-ONNX — ~500 MB (default)
onnx-community/gemma-4-E4B-it-ONNX — ~1.5 GB

Models are cached in the browser's cache storage after the first run.

Key Interfaces (`agent/`)

ModelBackend

typescript

// agent/types.tsexport interface ModelBackend {  generate(    messages: ChatMessage[],    tools: ToolDefinition[],    options: GenerateOptions  ): AsyncGenerator<StreamChunk>;}
export interface ToolDefinition {  name: string;  description: string;  parameters: JSONSchema;}
export interface GenerateOptions {  maxNewTokens?: number;  thinking?: boolean;}

// agent/types.tsexport interface ModelBackend {  generate(    messages: ChatMessage[],    tools: ToolDefinition[],    options: GenerateOptions  ): AsyncGenerator<StreamChunk>;}
export interface ToolDefinition {  name: string;  description: string;  parameters: JSONSchema;}
export interface GenerateOptions {  maxNewTokens?: number;  thinking?: boolean;}

ToolExecutor

typescript

// agent/types.tsexport interface ToolExecutor {  execute(toolName: string, args: Record<string, unknown>): Promise<unknown>;}

// agent/types.tsexport interface ToolExecutor {  execute(toolName: string, args: Record<string, unknown>): Promise<unknown>;}

Agent Loop

typescript

// agent/loop.ts — simplified illustrationexport async function* runAgentLoop(  userMessage: string,  history: ChatMessage[],  model: ModelBackend,  tools: ToolExecutor,  toolDefs: ToolDefinition[],  maxIterations: number): AsyncGenerator<AgentEvent> {  const messages = [...history, { role: "user", content: userMessage }];
  for (let i = 0; i < maxIterations; i++) {    for await (const chunk of model.generate(messages, toolDefs, {})) {      if (chunk.type === "token") yield { type: "token", token: chunk.token };      if (chunk.type === "tool_call") {        yield { type: "tool_start", name: chunk.name };        const result = await tools.execute(chunk.name, chunk.args);        yield { type: "tool_result", name: chunk.name, result };        messages.push({ role: "tool", name: chunk.name, content: String(result) });      }      if (chunk.type === "done") return;    }  }}

// agent/loop.ts — simplified illustrationexport async function* runAgentLoop(  userMessage: string,  history: ChatMessage[],  model: ModelBackend,  tools: ToolExecutor,  toolDefs: ToolDefinition[],  maxIterations: number): AsyncGenerator<AgentEvent> {  const messages = [...history, { role: "user", content: userMessage }];
  for (let i = 0; i < maxIterations; i++) {    for await (const chunk of model.generate(messages, toolDefs, {})) {      if (chunk.type === "token") yield { type: "token", token: chunk.token };      if (chunk.type === "tool_call") {        yield { type: "tool_start", name: chunk.name };        const result = await tools.execute(chunk.name, chunk.args);        yield { type: "tool_result", name: chunk.name, result };        messages.push({ role: "tool", name: chunk.name, content: String(result) });      }      if (chunk.type === "done") return;    }  }}

Built-in Tools

Tool	Location	Description
`read_page_content`	Content script	Read page text/HTML or a CSS selector
`take_screenshot`	Service worker	Capture visible tab as PNG
`click_element`	Content script	Click by CSS selector
`type_text`	Content script	Type into input by CSS selector
`scroll_page`	Content script	Scroll by pixel amount
`run_javascript`	Service worker	Execute JS in page context

Adding a New Tool

Tools live in two places: the definition (in the offscreen agent) and the executor (in content script or service worker).

Step 1 — Define the tool schema

typescript

// offscreen/tools/definitions.tsexport const MY_TOOL_DEFINITION: ToolDefinition = {  name: "get_page_title",  description: "Returns the document title of the current page.",  parameters: {    type: "object",    properties: {},    required: [],  },};

// offscreen/tools/definitions.tsexport const MY_TOOL_DEFINITION: ToolDefinition = {  name: "get_page_title",  description: "Returns the document title of the current page.",  parameters: {    type: "object",    properties: {},    required: [],  },};

Step 2 — Register in the tool list

typescript

// offscreen/tools/index.tsimport { MY_TOOL_DEFINITION } from "./definitions";
export const ALL_TOOLS: ToolDefinition[] = [  // ...existing tools  MY_TOOL_DEFINITION,];

// offscreen/tools/index.tsimport { MY_TOOL_DEFINITION } from "./definitions";
export const ALL_TOOLS: ToolDefinition[] = [  // ...existing tools  MY_TOOL_DEFINITION,];

Step 3 — Implement execution in the content script

typescript

// content/tools/executor.tsexport async function executeContentTool(  name: string,  args: Record<string, unknown>): Promise<unknown> {  switch (name) {    case "get_page_title":      return document.title;
    case "read_page_content": {      const selector = args.selector as string | undefined;      if (selector) {        return document.querySelector(selector)?.textContent ?? "Not found";      }      return document.body.innerText;    }
    case "click_element": {      const el = document.querySelector(args.selector as string) as HTMLElement;      if (!el) throw new Error(`Element not found: ${args.selector}`);      el.click();      return "clicked";    }
    case "type_text": {      const input = document.querySelector(args.selector as string) as HTMLInputElement;      if (!input) throw new Error(`Input not found: ${args.selector}`);      input.focus();      input.value = args.text as string;      input.dispatchEvent(new Event("input", { bubbles: true }));      input.dispatchEvent(new Event("change", { bubbles: true }));      return "typed";    }
    default:      throw new Error(`Unknown content tool: ${name}`);  }}

// content/tools/executor.tsexport async function executeContentTool(  name: string,  args: Record<string, unknown>): Promise<unknown> {  switch (name) {    case "get_page_title":      return document.title;
    case "read_page_content": {      const selector = args.selector as string | undefined;      if (selector) {        return document.querySelector(selector)?.textContent ?? "Not found";      }      return document.body.innerText;    }
    case "click_element": {      const el = document.querySelector(args.selector as string) as HTMLElement;      if (!el) throw new Error(`Element not found: ${args.selector}`);      el.click();      return "clicked";    }
    case "type_text": {      const input = document.querySelector(args.selector as string) as HTMLInputElement;      if (!input) throw new Error(`Input not found: ${args.selector}`);      input.focus();      input.value = args.text as string;      input.dispatchEvent(new Event("input", { bubbles: true }));      input.dispatchEvent(new Event("change", { bubbles: true }));      return "typed";    }
    default:      throw new Error(`Unknown content tool: ${name}`);  }}

Step 4 — Handle service-worker-side tools

typescript

// background/tools.tsexport async function executeSwTool(  name: string,  args: Record<string, unknown>,  tabId: number): Promise<unknown> {  switch (name) {    case "take_screenshot": {      const dataUrl = await chrome.tabs.captureVisibleTab({ format: "png" });      return dataUrl;    }
    case "run_javascript": {      const results = await chrome.scripting.executeScript({        target: { tabId },        func: new Function(args.code as string) as () => unknown,      });      return results[0]?.result ?? null;    }
    default:      return null; // not a SW tool — forward to content script  }}

// background/tools.tsexport async function executeSwTool(  name: string,  args: Record<string, unknown>,  tabId: number): Promise<unknown> {  switch (name) {    case "take_screenshot": {      const dataUrl = await chrome.tabs.captureVisibleTab({ format: "png" });      return dataUrl;    }
    case "run_javascript": {      const results = await chrome.scripting.executeScript({        target: { tabId },        func: new Function(args.code as string) as () => unknown,      });      return results[0]?.result ?? null;    }
    default:      return null; // not a SW tool — forward to content script  }}

Message Routing Pattern

The service worker acts as a message bus. All communication uses chrome.runtime.sendMessage.

typescript

// Message types (shared/messages.ts)export type ExtMessage =  | { type: "TOOL_CALL"; name: string; args: Record<string, unknown>; tabId: number }  | { type: "TOOL_RESULT"; name: string; result: unknown }  | { type: "TOKEN"; token: string }  | { type: "AGENT_DONE" }  | { type: "AGENT_ERROR"; error: string };
// Offscreen → SWchrome.runtime.sendMessage<ExtMessage>({  type: "TOOL_CALL",  name: "click_element",  args: { selector: "#submit-btn" },  tabId: currentTabId,});
// SW → Content scriptchrome.tabs.sendMessage<ExtMessage>(tabId, {  type: "TOOL_CALL",  name: "click_element",  args: { selector: "#submit-btn" },  tabId,});

// Message types (shared/messages.ts)export type ExtMessage =  | { type: "TOOL_CALL"; name: string; args: Record<string, unknown>; tabId: number }  | { type: "TOOL_RESULT"; name: string; result: unknown }  | { type: "TOKEN"; token: string }  | { type: "AGENT_DONE" }  | { type: "AGENT_ERROR"; error: string };
// Offscreen → SWchrome.runtime.sendMessage<ExtMessage>({  type: "TOOL_CALL",  name: "click_element",  args: { selector: "#submit-btn" },  tabId: currentTabId,});
// SW → Content scriptchrome.tabs.sendMessage<ExtMessage>(tabId, {  type: "TOOL_CALL",  name: "click_element",  args: { selector: "#submit-btn" },  tabId,});

Model Configuration

typescript

// offscreen/model.ts — loading with transformers.jsimport { pipeline, TextGenerationPipeline } from "@huggingface/transformers";
const MODEL_IDS = {  E2B: "onnx-community/gemma-4-E2B-it-ONNX",  E4B: "onnx-community/gemma-4-E4B-it-ONNX",} as const;
export type ModelSize = keyof typeof MODEL_IDS;
export async function loadModel(  size: ModelSize,  onProgress: (progress: number) => void): Promise<TextGenerationPipeline> {  return pipeline("text-generation", MODEL_IDS[size], {    dtype: "q4f16",    device: "webgpu",    progress_callback: (p: { progress: number }) => onProgress(p.progress),  });}

// offscreen/model.ts — loading with transformers.jsimport { pipeline, TextGenerationPipeline } from "@huggingface/transformers";
const MODEL_IDS = {  E2B: "onnx-community/gemma-4-E2B-it-ONNX",  E4B: "onnx-community/gemma-4-E4B-it-ONNX",} as const;
export type ModelSize = keyof typeof MODEL_IDS;
export async function loadModel(  size: ModelSize,  onProgress: (progress: number) => void): Promise<TextGenerationPipeline> {  return pipeline("text-generation", MODEL_IDS[size], {    dtype: "q4f16",    device: "webgpu",    progress_callback: (p: { progress: number }) => onProgress(p.progress),  });}

Settings & Persistence

Settings are stored via chrome.storage.sync:

typescript

export interface GemmaGemSettings {  modelSize: "E2B" | "E4B";  thinking: boolean;  maxIterations: number;  disabledHosts: string[];}
const DEFAULT_SETTINGS: GemmaGemSettings = {  modelSize: "E2B",  thinking: false,  maxIterations: 10,  disabledHosts: [],};
export async function getSettings(): Promise<GemmaGemSettings> {  const stored = await chrome.storage.sync.get("settings");  return { ...DEFAULT_SETTINGS, ...(stored.settings ?? {}) };}
export async function saveSettings(patch: Partial<GemmaGemSettings>): Promise<void> {  const current = await getSettings();  await chrome.storage.sync.set({ settings: { ...current, ...patch } });}
// Disable extension on current hostasync function disableOnCurrentSite() {  const host = new URL(location.href).hostname;  const settings = await getSettings();  if (!settings.disabledHosts.includes(host)) {    await saveSettings({ disabledHosts: [...settings.disabledHosts, host] });  }}

export interface GemmaGemSettings {  modelSize: "E2B" | "E4B";  thinking: boolean;  maxIterations: number;  disabledHosts: string[];}
const DEFAULT_SETTINGS: GemmaGemSettings = {  modelSize: "E2B",  thinking: false,  maxIterations: 10,  disabledHosts: [],};
export async function getSettings(): Promise<GemmaGemSettings> {  const stored = await chrome.storage.sync.get("settings");  return { ...DEFAULT_SETTINGS, ...(stored.settings ?? {}) };}
export async function saveSettings(patch: Partial<GemmaGemSettings>): Promise<void> {  const current = await getSettings();  await chrome.storage.sync.set({ settings: { ...current, ...patch } });}
// Disable extension on current hostasync function disableOnCurrentSite() {  const host = new URL(location.href).hostname;  const settings = await getSettings();  if (!settings.disabledHosts.includes(host)) {    await saveSettings({ disabledHosts: [...settings.disabledHosts, host] });  }}

Shadow DOM Chat UI Pattern

The content script injects a shadow DOM to isolate styles:

typescript

// content/ui.tsexport function injectChatOverlay(): ShadowRoot {  const host = document.createElement("div");  host.id = "gemma-gem-host";  // Prevent page styles from leaking in  const shadow = host.attachShadow({ mode: "closed" });
  // Inject styles  const style = document.createElement("style");  style.textContent = CHAT_STYLES; // imported CSS string  shadow.appendChild(style);
  // Inject chat container  const container = document.createElement("div");  container.id = "gemma-gem-container";  shadow.appendChild(container);
  document.body.appendChild(host);  return shadow;}

// content/ui.tsexport function injectChatOverlay(): ShadowRoot {  const host = document.createElement("div");  host.id = "gemma-gem-host";  // Prevent page styles from leaking in  const shadow = host.attachShadow({ mode: "closed" });
  // Inject styles  const style = document.createElement("style");  style.textContent = CHAT_STYLES; // imported CSS string  shadow.appendChild(style);
  // Inject chat container  const container = document.createElement("div");  container.id = "gemma-gem-container";  shadow.appendChild(container);
  document.body.appendChild(host);  return shadow;}

Debugging

All logs use [Gemma Gem] prefix. Development builds log info/debug/warn; production only logs errors.

# Service worker logschrome://extensions → Gemma Gem → "Inspect views: service worker"
# Offscreen document (most useful: model loading, prompts, tool calls)chrome://extensions → Gemma Gem → "Inspect views: offscreen.html"
# Content script logsDevTools on any page → Console (filter: [Gemma Gem])
# All extension contextschrome://inspect#other

# Service worker logschrome://extensions → Gemma Gem → "Inspect views: service worker"
# Offscreen document (most useful: model loading, prompts, tool calls)chrome://extensions → Gemma Gem → "Inspect views: offscreen.html"
# Content script logsDevTools on any page → Console (filter: [Gemma Gem])
# All extension contextschrome://inspect#other

Key things to check in offscreen document logs:

Model download progress
Full prompt construction
Token counts per turn
Raw model output (before tool call parsing)
Tool execution results

Common Patterns & Gotchas

WebGPU availability check:

typescript

if (!navigator.gpu) {  throw new Error("WebGPU not supported. Use Chrome 113+ with hardware acceleration enabled.");}const adapter = await navigator.gpu.requestAdapter();if (!adapter) throw new Error("No WebGPU adapter found.");

if (!navigator.gpu) {  throw new Error("WebGPU not supported. Use Chrome 113+ with hardware acceleration enabled.");}const adapter = await navigator.gpu.requestAdapter();if (!adapter) throw new Error("No WebGPU adapter found.");

Offscreen document lifecycle — Chrome may suspend the offscreen document. Ping it before sending messages:

typescript

async function ensureOffscreen() {  const existing = await chrome.offscreen.hasDocument();  if (!existing) {    await chrome.offscreen.createDocument({      url: "offscreen.html",      reasons: [chrome.offscreen.Reason.WORKERS],      justification: "Run Gemma 4 inference via WebGPU",    });  }}

async function ensureOffscreen() {  const existing = await chrome.offscreen.hasDocument();  if (!existing) {    await chrome.offscreen.createDocument({      url: "offscreen.html",      reasons: [chrome.offscreen.Reason.WORKERS],      justification: "Run Gemma 4 inference via WebGPU",    });  }}

Context window management — Gemma 4 supports 128K tokens but inference slows with long contexts. Clear history per-page with clear_context or limit stored turns:

typescript

const MAX_HISTORY_TURNS = 20;function trimHistory(messages: ChatMessage[]): ChatMessage[] {  if (messages.length <= MAX_HISTORY_TURNS * 2) return messages;  return messages.slice(-MAX_HISTORY_TURNS * 2);}

const MAX_HISTORY_TURNS = 20;function trimHistory(messages: ChatMessage[]): ChatMessage[] {  if (messages.length <= MAX_HISTORY_TURNS * 2) return messages;  return messages.slice(-MAX_HISTORY_TURNS * 2);}

Tool call parsing — Gemma 4 emits tool calls in a structured format. If adding custom parsing, guard against partial/streamed JSON:

typescript

function safeParseToolCall(raw: string): { name: string; args: Record<string, unknown> } | null {  try {    return JSON.parse(raw);  } catch {    return null; // still streaming  }}

function safeParseToolCall(raw: string): { name: string; args: Record<string, unknown> } | null {  try {    return JSON.parse(raw);  } catch {    return null; // still streaming  }}

CSS selector safety for DOM tools:

typescript

function safeQuerySelector(selector: string): Element | null {  try {    return document.querySelector(selector);  } catch {    return null; // invalid selector from model  }}

function safeQuerySelector(selector: string): Element | null {  try {    return document.querySelector(selector);  } catch {    return null; // invalid selector from model  }}

来源与署名

来源：reason-machines/trending-skills位于skills/gemma-gem-browser-ai提交2384a00

许可证：无许可证

内容归原作者所有。SourceWeft 从公开仓库中收录这些内容。

举报或申请下架

Gemma Gem Browser Ai

Gemma Gem Browser AI

Architecture Overview

Install & Build

Key Interfaces (`agent/`)

ModelBackend

ToolExecutor

Agent Loop

Built-in Tools

Adding a New Tool

Step 1 — Define the tool schema

Step 2 — Register in the tool list

Step 3 — Implement execution in the content script

Step 4 — Handle service-worker-side tools

Message Routing Pattern

Model Configuration

Settings & Persistence

Shadow DOM Chat UI Pattern

Debugging

Common Patterns & Gotchas

来源与署名

更多来自 reason-machines/trending-skills 的技能

Yourvpndead Vpn Detection

X Mentor Skill Nuwa

Wildworld Dataset

Understand Anything Knowledge Graph

Webphysics Avbd Engine

Ui Ux Pro Max Skill

更多Software Development技能

Playground

M5 Onboard

Cardputer Buddy

Web Design Guidelines

Vercel React Native Skills

Vercel React Best Practices

Gemma Gem Browser Ai

Gemma Gem Browser AI

Architecture Overview

Install & Build

Key Interfaces (agent/)

ModelBackend

ToolExecutor

Agent Loop

Built-in Tools

Adding a New Tool

Step 1 — Define the tool schema

Step 2 — Register in the tool list

Step 3 — Implement execution in the content script

Step 4 — Handle service-worker-side tools

Message Routing Pattern

Model Configuration

Settings & Persistence

Shadow DOM Chat UI Pattern

Debugging

Common Patterns & Gotchas

来源与署名

更多来自 reason-machines/trending-skills 的技能

Yourvpndead Vpn Detection

X Mentor Skill Nuwa

Wildworld Dataset

Understand Anything Knowledge Graph

Webphysics Avbd Engine

Ui Ux Pro Max Skill

更多Software Development技能

Playground

M5 Onboard

Cardputer Buddy

Web Design Guidelines

Vercel React Native Skills

Vercel React Best Practices

Key Interfaces (`agent/`)