Token Compressor

io.github.base76-research-labPublished by GitHub account base76-research-labv0.1.1Updated Oct 24, 2026

Compress prompts 40-60% using local LLM + embedding validation. Preserves all conditionals.

VerifiedSTDIODesktop onlyDeveloper ToolsAI & ML

Overview

AI-generated overview

A local MCP server that compresses prompts before they reach an LLM, using a local Ollama model plus embedding validation to cut token counts while keeping the

What you can do
Compress a prompt to its semantic minimum with a local Ollama model, preserving conditionals and negations, then validate the result by cosine similarity against the original and fall back to the original text if it falls below the threshold. Return the compressed text with a stats footer covering mode, coverage and estimated tokens in, out and saved. Skip compression for short inputs below a configurable token floor.
When to use it
Useful when long prompts are sent repeatedly to an LLM and token cost or context length matters, and you want compression done locally with a safety check against meaning loss. Less useful for short prompts, which are skipped anyway.
Requirements
Runs as a local Python package (Python 3.10+) via uvx or python3. Needs Ollama running locally with the llama3.2:1b and nomic-embed-text models pulled, plus the ollama and numpy Python dependencies. No accounts, API keys or headers declared.
Before you install
Prompts are sent to the local Ollama models for rewriting and embedding, so they leave the MCP client process and are processed by those models. Compression can still alter wording; the similarity threshold is the only safeguard, and lowering it makes compression more aggressive. The README also suggests a Claude Code hook that runs on every prompt submission and writes to temp files.

Installation

In SourceWeft

  1. Open Token Compressor in the dashboard and add it to a workspace.
  2. Enable the server for the chats that should use its tools.

Desktop only via STDIO. STDIO servers start a local process, so they need the SourceWeft desktop host.

Other MCP clients

Follow the launch instructions in the repository.

README

token-compressor

Reduce LLM prompt tokens by 30–70% while preserving semantic meaning.

mcp-name: io.github.base76-research-lab/token-compressor

Semantic prompt compression for LLM workflows. Reduce token usage by 40–60% without losing meaning.

[License: MIT] [Requires: Ollama] [MCP Compatible]

Built by Base76 Research Lab — research into epistemic AI architecture.

Live demo

Intent Compiler MVP is now live and uses this project as part of the idea -> spec -> compressed output flow:

What it does

token-compressor is a two-stage pipeline that compresses prompts before they reach an LLM:

  1. LLM compression — a local model (llama3.2:1b via Ollama) rewrites the prompt to its semantic minimum, preserving all conditionals and negations
  2. Embedding validation — cosine similarity between original and compressed embeddings must exceed a threshold (default: 0.85) — if not, the original is sent unchanged

The result: shorter prompts, lower costs, same intent.

Input prompt (300 tokens)        ↓  LLM compresses        ↓  Embedding validates (cosine ≥ 0.85?)        ↓  Pass → compressed (120 tokens)   Fail → original (300 tokens)

Key design principle: conditionality is never sacrificed. If your prompt says "only do X if Y", that constraint survives compression.

Requirements

  • Python 3.10+
  • Ollama running locally
  • Two models pulled:
bash
ollama pull llama3.2:1bollama pull nomic-embed-text
  • Python dependencies:
bash
pip install ollama numpy

Quick start

python
from compressor import LLMCompressEmbedValidate
pipeline = LLMCompressEmbedValidate()result = pipeline.process("Your prompt text here...")
print(result.output_text)   # compressed (or original if validation failed)print(result.report())      # MODE / COVERAGE / TOKENS saved

Result object:

FieldDescription
output_textText to send to your LLM
modecompressed / raw_fallback / skipped
coverageCosine similarity (0.0–1.0)
tokens_inEstimated input tokens
tokens_outEstimated output tokens
tokens_savedDifference

CLI usage

bash
echo "Your long prompt here..." | python3 cli.py

Output: compressed text on stdout, stats on stderr.

Claude Code hook (recommended setup)

Add to your ~/.claude/settings.json under hooks → UserPromptSubmit:

json
{  "type": "command",  "command": "echo \"${CLAUDE_USER_PROMPT:-}\" | python3 /path/to/token-compressor/cli.py > /tmp/compressed_prompt.txt 2>/tmp/compress.log || true"}

This runs on every prompt submission and writes the compressed version to a temp file, which can be injected back into context via a second hook or MCP server.

MCP server

The MCP server exposes compression as a tool callable from Claude Code and any MCP-compatible client.

Install:

bash
pip install token-compressor-mcp

Tool: compress_prompt

  • Input: text (string)
  • Output: compressed text + stats footer

Claude Code MCP config (~/.claude/settings.json):

json
{  "mcpServers": {    "token-compressor": {      "command": "uvx",      "args": ["token-compressor-mcp"]    }  }}

Or from source:

json
{  "mcpServers": {    "token-compressor": {      "command": "python3",      "args": ["-m", "token_compressor_mcp"],      "cwd": "/path/to/token-compressor"    }  }}

Configuration

python
pipeline = LLMCompressEmbedValidate(    threshold=0.85,          # cosine similarity floor (lower = more aggressive)    min_tokens=80,           # skip pipeline below this (not worth compressing)    compress_model="llama3.2:1b",    embed_model="nomic-embed-text",)

How it works

Stage 1 — LLM compression

The compression prompt instructs the model to:

  • Preserve all conditionals (if, only if, unless, when, but only)
  • Preserve all negations
  • Remove filler, hedging, redundancy
  • Target 40–60% of original length

Stage 2 — Embedding validation

Computes cosine similarity between the original and compressed text using nomic-embed-text. If similarity falls below threshold, the original is returned unchanged. This prevents silent meaning loss.

Results

Tested across Swedish and English prompts, technical and natural language:

InputTokens inTokens outSaved
Research abstract (EN)893857%
Session intent (SV)321844%
Technical instruction472253%
Short command (<80t)——skipped

Research background

This tool implements the architecture from:

Wikström, B. (2026). When Alignment Reduces Uncertainty: Epistemic Variance Collapse and Its Implications for Metacognitive AI. DOI: 10.5281/zenodo.18731535

Part of the Base76 Research Lab toolchain for epistemic AI infrastructure.

License

MIT — Base76 Research Lab, Sweden

Source: README.md at commit 1f5e39e

Tools

0
SourceWeft has not started this package in its sandbox yet, so its tools are not known.

Version history

1
  1. v0.1.1LatestSep 16, 2026

Score

50% of the applicable checks met

How complete this listing is, computed from the registry entry, the endpoint and public sources. Not a rating of the server and not a statement that it is safe or working today: reachability is a dated check, and no installs or reviews go into the score.

  • Reached in the last 30 days25 pts

    Keep the endpoint answering the MCP handshake; SourceWeft probes it every 30 days.

  • Auth declared10 pts

    Declare the credential headers in the registry entry, or keep the endpoint answering the probe so its auth can be classified.

  • Tools listed15 pts

    Answer tools/list without a credential, or ship a package that starts in a sandbox without one.

  • DescriptionRequired15 pts
  • Logo10 pts
  • Documentation15 pts
  • License10 pts