Local & Open Source Providers

🦙 Ollama

The easiest way to run local LLMs. Supports Llama 3, Mistral, Gemma, and 50+ other models with automatic model management and GPU acceleration.

Installation

# macOS/Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from https://ollama.com/download

# Pull a model
ollama pull llama3.2
ollama pull mistral
ollama pull gemma2

Basic Usage

import { Agent, OllamaProvider, ToolRegistry } from '@lov3kaizen/agentsea-core';

// Initialize Ollama provider
const provider = new OllamaProvider({
  baseUrl: 'http://localhost:11434',
  model: 'llama3.2' // or 'mistral', 'gemma2', etc.
});

// Create agent with local model
const agent = new Agent(
  {
    name: 'local-assistant',
    model: 'llama3.2',
    provider: 'ollama',
    systemPrompt: 'You are a helpful AI assistant running locally.',
    tools: [],
    temperature: 0.7
  },
  provider,
  new ToolRegistry()
);

// Execute locally
const response = await agent.execute(
  'What are the benefits of running AI models locally?'
);

console.log(response.content);

Recommended Models

• llama3.2:3b - Fast, efficient, great for chat (3GB RAM)
• llama3.1:8b - Balanced performance and quality (8GB RAM)
• mistral:7b - Excellent instruction following (7GB RAM)
• gemma2:9b - Google's efficient model (9GB RAM)
• qwen2.5:7b - Strong multilingual support (7GB RAM)

🚀 llama.cpp

High-performance C++ inference engine with Metal (Mac), CUDA (NVIDIA), and CPU support. Provides the fastest local inference with quantized models.

Installation

# Build from source
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# macOS (with Metal acceleration)
make LLAMA_METAL=1

# Linux with CUDA
make LLAMA_CUDA=1

# Start server
./llama-server \
  -m models/llama-3.2-3b-q4_k_m.gguf \
  --port 8080 \
  --n-gpu-layers 35

Basic Usage

import { Agent, LlamaCppProvider, ToolRegistry } from '@lov3kaizen/agentsea-core';

// Initialize llama.cpp provider
const provider = new LlamaCppProvider({
  baseUrl: 'http://localhost:8080',
  model: 'llama-3.2-3b-q4_k_m'
});

const agent = new Agent(
  {
    name: 'fast-agent',
    model: 'llama-3.2-3b-q4_k_m',
    provider: 'llama-cpp',
    systemPrompt: 'You are a fast, efficient AI assistant.',
    tools: [],
    temperature: 0.7,
    maxTokens: 2048
  },
  provider,
  new ToolRegistry()
);

// Stream responses for real-time output
const stream = await agent.stream('Explain quantum computing');

for await (const chunk of stream) {
  if (chunk.type === 'content') {
    process.stdout.write(chunk.content);
  }
}

Download Quantized Models

Get pre-quantized GGUF models from HuggingFace:

• bartowski's collection - Wide variety of quantized models
• TheBloke's collection - Popular GGUF conversions
• QuantFactory - High-quality quantizations

🌐 GPT4All

Privacy-focused local LLM platform with easy-to-use desktop app and Python/TypeScript bindings. Great for beginners and non-technical users.

Installation

# Install GPT4All package
npm install gpt4all

# Or download desktop app
# https://gpt4all.io/

Basic Usage

import { Agent, GPT4AllProvider, ToolRegistry } from '@lov3kaizen/agentsea-core';

// Initialize GPT4All provider
const provider = new GPT4AllProvider({
  model: 'orca-mini-3b-gguf2-q4_0',
  modelPath: './models/' // Optional: custom model directory
});

const agent = new Agent(
  {
    name: 'gpt4all-agent',
    model: 'orca-mini-3b-gguf2-q4_0',
    provider: 'gpt4all',
    systemPrompt: 'You are a helpful assistant.',
    tools: []
  },
  provider,
  new ToolRegistry()
);

const response = await agent.execute('What is machine learning?');
console.log(response.content);

🤗 HuggingFace Transformers

Access thousands of open source models via HuggingFace Inference API or self-hosted endpoints. Supports both cloud and local deployment.

Using Inference API (Free)

import { Agent, HuggingFaceProvider, ToolRegistry } from '@lov3kaizen/agentsea-core';

// Free tier with rate limits
const provider = new HuggingFaceProvider({
  apiKey: process.env.HUGGINGFACE_API_KEY, // Get from hf.co
  model: 'meta-llama/Meta-Llama-3-8B-Instruct'
});

const agent = new Agent(
  {
    name: 'hf-agent',
    model: 'meta-llama/Meta-Llama-3-8B-Instruct',
    provider: 'huggingface',
    systemPrompt: 'You are a helpful AI assistant.',
    tools: []
  },
  provider,
  new ToolRegistry()
);

const response = await agent.execute('Explain neural networks');
console.log(response.content);

Self-Hosted with Text Generation Inference

# Run TGI server locally
docker run -p 8080:80 \
  -v ./models:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Meta-Llama-3-8B-Instruct

# Connect to local endpoint
const provider = new HuggingFaceProvider({
  baseUrl: 'http://localhost:8080',
  model: 'meta-llama/Meta-Llama-3-8B-Instruct'
});

Popular Open Source Models

• meta-llama/Meta-Llama-3.1-8B-Instruct - Meta's latest
• mistralai/Mistral-7B-Instruct-v0.3 - Efficient instruct model
• google/gemma-2-9b-it - Google's open model
• Qwen/Qwen2.5-7B-Instruct - Multilingual support
• microsoft/Phi-3-mini-4k-instruct - Small but capable (3.8B)

💻 LM Studio

User-friendly desktop app for running local LLMs with a beautiful UI. Includes model discovery, automatic quantization selection, and OpenAI-compatible API server.

Setup

1. Download LM Studio from lmstudio.ai
2. Browse and download models from the UI
3. Start the local server (OpenAI-compatible endpoint)
4. Connect AgentSea to the server

Basic Usage

import { Agent, OpenAIProvider, ToolRegistry } from '@lov3kaizen/agentsea-core';

// LM Studio uses OpenAI-compatible API
const provider = new OpenAIProvider({
  baseUrl: 'http://localhost:1234/v1',
  apiKey: 'lm-studio', // Any value works for local
  model: 'local-model'
});

const agent = new Agent(
  {
    name: 'lm-studio-agent',
    model: 'local-model',
    provider: 'openai', // Uses OpenAI interface
    systemPrompt: 'You are a helpful assistant.',
    tools: []
  },
  provider,
  new ToolRegistry()
);

const response = await agent.execute('Hello!');
console.log(response.content);

🌪️ Mistral AI (Self-Hosted)

Mistral offers open-weight models that can be self-hosted. Use their official Docker images or deploy via vLLM for production workloads.

Using vLLM (Recommended for Production)

# Deploy Mistral 7B with vLLM
docker run -p 8000:8000 \
  --gpus all \
  vllm/vllm-openai:latest \
  --model mistralai/Mistral-7B-Instruct-v0.3 \
  --dtype float16

# Connect AgentSea agent
const provider = new OpenAIProvider({
  baseUrl: 'http://localhost:8000/v1',
  apiKey: 'none',
  model: 'mistralai/Mistral-7B-Instruct-v0.3'
});