Grepai Storage Gob

GrepAI Storage with GOB

This skill covers using GOB (Go Binary) as the storage backend for GrepAI, the default and simplest option.

When to Use This Skill

• Single developer projects
• Small to medium codebases
• Simple setup without external dependencies
• Local development environments

What is GOB Storage?

GOB is Go's native binary serialization format. GrepAI uses it to store:

• Vector embeddings
• File metadata
• Chunk information

Everything is stored in a single local file.

Advantages

Limitations

Configuration

Default Configuration

GOB is the default backend. Minimal config:

# .grepai/config.yamlstore:  backend: gob

Explicit Configuration

store:  backend: gob  # Index stored in .grepai/index.gob (automatic)

Storage Location

GOB storage creates files in your project's .grepai/ directory:

.grepai/├── config.yaml    # Configuration├── index.gob      # Vector embeddings└── symbols.gob    # Symbol index for trace

File Sizes

Approximate .grepai/index.gob sizes:

Operations

Creating the Index

# Initialize projectgrepai init
# Start indexing (creates index.gob)grepai watch

Checking Index Status

grepai status
# Output:# Index: .grepai/index.gob# Files: 245# Chunks: 1,234# Size: 12.5 MB# Last updated: 2025-01-28 10:30:00

Backing Up the Index

# Simple file copycp .grepai/index.gob .grepai/index.gob.backup

Clearing the Index

# Delete and re-indexrm .grepai/index.gobgrepai watch

Moving to a New Machine

# Copy entire .grepai directorycp -r .grepai /path/to/new/location/
# Note: Only works if using same embedding model

Performance Considerations

Memory Usage

GOB loads the entire index into RAM for searches:

Search Speed

GOB provides fast searches for typical codebases:

When to Upgrade

Consider PostgreSQL or Qdrant when:

• Index exceeds 1 GB
• Need concurrent access
• Want to share index across team
• Codebase has 50K+ files

.gitignore Configuration

Add .grepai/ to your .gitignore:

# GrepAI (machine-specific index).grepai/

Why: The index is machine-specific because:

• Contains binary embeddings
• Tied to the embedding model used
• Each machine should generate its own

Sharing Index (Not Recommended)

While you can copy the index file, it's not recommended because:

1. Must use identical embedding model
2. File paths are absolute
3. Different machines may have different code versions

Better approach: Each developer runs their own grepai watch.

Migrating to Other Backends

To PostgreSQL

1. Update config:

store:  backend: postgres  postgres:    dsn: postgres://user:pass@localhost:5432/grepai

2. Re-index:

rm .grepai/index.gobgrepai watch

To Qdrant

1. Update config:

store:  backend: qdrant  qdrant:    endpoint: localhost    port: 6334

2. Re-index:

rm .grepai/index.gobgrepai watch

Common Issues

❌ Problem: Index file too large ✅ Solution: Add more ignore patterns or migrate to PostgreSQL/Qdrant

❌ Problem: Slow searches on large codebase ✅ Solution: Migrate to Qdrant for better performance

❌ Problem: Corrupted index ✅ Solution: Delete and re-index:

rm .grepai/index.gob .grepai/symbols.gobgrepai watch

❌ Problem: "Index not found" error ✅ Solution: Run grepai watch to create the index

Best Practices

1. Use for small/medium projects: Up to ~10K files
2. Add to .gitignore: Don't commit the index
3. Backup before major changes: Copy index.gob before experiments
4. Re-index after model changes: If you change embedding models
5. Monitor file size: Migrate if index exceeds 1GB

Output Format

GOB storage status:

✅ GOB Storage Configured
   Backend: GOB (local file)   Index: .grepai/index.gob   Size: 12.5 MB
   Contents:   - Files: 245   - Chunks: 1,234   - Vectors: 1,234 × 768 dimensions
   Performance:   - Search latency: <100ms   - Memory usage: ~25 MB