5 AutoGPT Memory Types (Vector, Redis, File, Conversation)

Memory is what separates a one-shot LLM call from a genuine autonomous agent. Without memory, every step AutoGPT takes starts from scratch — the agent cannot build on what it learned, cannot reference prior context, and cannot avoid repeating the same mistakes.

AutoGPT supports five distinct memory backends, each with different characteristics around speed, persistence, and setup complexity. The right choice depends on your task type, infrastructure, and tolerance for setup overhead.

This guide covers all five backends with configuration examples, benchmarks, and an honest recommendation for each use case.

How AutoGPT Uses Memory

Before comparing backends, it helps to understand what AutoGPT is actually storing. Every time the agent produces a meaningful output — a web search result, a written document, an observation from a tool — it converts that content into an embedding and stores it in memory.

When planning future steps, AutoGPT queries memory using the current context as a search vector, retrieving the most semantically similar past entries. This is how it avoids re-doing work and builds on prior discoveries.

The embedding model used is OpenAI's text-embedding-ada-002 by default, which produces 1,536-dimensional vectors. This means every memory entry is a vector of 1,536 floating point numbers plus the original text.

For the broader context on why this pattern matters, see our AI agent memory and planning guide.

The Memory Backend Comparison Table

Here is the full comparison before we go into detail:

1. Local File Memory

MEMORY_BACKEND=local

The default. AutoGPT writes memory entries to a JSON file in the local workspace. Simple, portable, requires nothing extra.

The implementation is a flat list of records. Every memory read scans the entire list and computes cosine similarity in Python — which works fine for 50 entries and starts to feel slow around 500.

# What the local backend looks like internally
{
  "memories": [
    {
      "data": "The refund policy at Acme Store is 30 days for unused items.",
      "embedding": [0.023, -0.041, 0.118, ...],  # 1,536 dimensions
      "timestamp": "2026-05-31T10:23:41Z"
    },
    ...
  ]
}

When to use it:

• Learning AutoGPT or running tutorials
• Single-session tasks that do not need persistence after the run
• Debugging — the file is human-readable JSON

When not to use it:

• Tasks expected to generate more than 200 memory entries
• Any production deployment
• Tasks where sub-100ms memory lookup matters

Configuration:

MEMORY_BACKEND=local
# Memory file location (relative to workspace)
MEMORY_INDEX=auto-gpt-memory

2. Redis Memory

MEMORY_BACKEND=redis
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=

Redis with the RedisSearch module gives you in-memory vector similarity search. Lookups are dramatically faster than the local backend — typically under 5ms versus 200-500ms for local on a 500-entry store.

Setting up Redis for AutoGPT requires the redis-stack distribution, which bundles RedisSearch:

# Docker (quickest)
docker run -d \
  --name redis-stack \
  -p 6379:6379 \
  -p 8001:8001 \
  redis/redis-stack:latest

# Verify the module is loaded
redis-cli module list
# Should show search module

For persistence across restarts, configure Redis AOF (Append Only File) persistence:

# redis.conf
appendonly yes
appendfsync everysec

Or set via environment:

REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=your_password_here

# If using Redis Cloud
REDIS_HOST=redis-12345.c123.us-east-1-1.ec2.cloud.redislabs.com
REDIS_PORT=12345
REDIS_PASSWORD=cloud_password_here

Performance profile:

• Write: ~2ms per entry
• Read (exact): ~1ms
• Read (vector search, 1000 entries): ~5ms
• Read (vector search, 100k entries): ~20ms

When to use it:

• Tasks running for hours where speed matters
• Agentic pipelines with fast iteration cycles
• When you want simple setup with good performance

When not to use it:

• You need guaranteed persistence without ops overhead
• Memory should survive a Redis restart without configuration

3. Pinecone Memory

MEMORY_BACKEND=pinecone
PINECONE_API_KEY=your-pinecone-api-key
PINECONE_ENV=us-east-1-aws

Pinecone is a managed vector database in the cloud. It requires no infrastructure on your end — you call an API, Pinecone stores and indexes the vectors, and you query them later. Data persists indefinitely without any configuration.

Setup:

1. Create an account at pinecone.io
2. Create an index named auto-gpt-memory with dimension=1536 and metric=cosine
3. Copy the API key and environment string

# Verifying Pinecone connection before running AutoGPT
import pinecone

pinecone.init(api_key="your-key", environment="us-east-1-aws")
index = pinecone.Index("auto-gpt-memory")
print(index.describe_index_stats())
# Should show: {'namespaces': {}, 'dimension': 1536, 'index_fullness': 0.0}

The main downside is cost. Pinecone's starter plan is free up to 100k vectors but the paid plans start at $70/month for the serverless tier at scale. For occasional AutoGPT tasks, the free tier is usually sufficient.

For a detailed comparison of vector database options including Pinecone, see our vector database guide.

When to use it:

• Long-running agents that accumulate large memory stores
• When you need memory to persist between different agent runs spanning days or weeks
• When you do not want to manage any infrastructure

When not to use it:

• Cost-sensitive deployments
• Development and testing (use local or Redis instead)

4. Milvus Memory

MEMORY_BACKEND=milvus
MILVUS_ADDR=localhost:19530

Milvus is an open-source vector database with enterprise-grade performance. It is the best choice for high-volume, on-premise deployments where you want Pinecone-level capability without the cloud dependency or cost.

Setup is more involved:

# Docker Compose (recommended for local)
wget https://github.com/milvus-io/milvus/releases/download/v2.4.0/milvus-standalone-docker-compose.yml -O docker-compose.yml

docker-compose up -d

# Verify
docker-compose ps
# Should show milvus-standalone running

Create the collection before running AutoGPT:

from pymilvus import connections, Collection, CollectionSchema, FieldSchema, DataType

connections.connect("default", host="localhost", port="19530")

fields = [
    FieldSchema(name="pk", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="data", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=1536)
]

schema = CollectionSchema(fields, description="AutoGPT memory")
collection = Collection("auto_gpt_memory", schema)

# Create IVF_FLAT index for similarity search
index_params = {
    "metric_type": "COSINE",
    "index_type": "IVF_FLAT",
    "params": {"nlist": 128}
}
collection.create_index("embedding", index_params)
print("Milvus collection ready")

Performance profile (1M vectors):

• Write throughput: ~10,000 entries/second
• Query latency: 10-30ms
• Recall accuracy: 99%+ with HNSW index

When to use it:

• Enterprise deployments needing on-premise storage
• High-volume agents generating millions of memory entries
• When you need fine-grained control over indexing strategy

When not to use it:

• Quick projects or experimentation
• Single-developer setups without Docker comfort

5. Weaviate Memory

MEMORY_BACKEND=weaviate
WEAVIATE_HOST=http://localhost:8080
WEAVIATE_USERNAME=
WEAVIATE_PASSWORD=

Weaviate is unique among vector databases because it supports hybrid search — combining vector similarity with traditional keyword (BM25) search. If your agent's memory contains structured data and you sometimes need exact keyword matches alongside semantic search, Weaviate is worth the extra setup.

# Docker setup
docker run -d \
  --name weaviate \
  -p 8080:8080 \
  -p 50051:50051 \
  cr.weaviate.io/semitechnologies/weaviate:1.24.4 \
  --query.defaults.limit 25 \
  --authentication.anonymous_access.enabled true

Verify the connection:

import weaviate

client = weaviate.Client("http://localhost:8080")
print(client.is_ready())
# Should print: True

AutoGPT creates its memory schema in Weaviate automatically on first run. You do not need to create collections manually.

When to use it:

• When your agent needs both semantic and keyword search over its memory
• Building agents that work with structured documents alongside unstructured content
• When you want Weaviate's built-in vectorization modules to reduce embedding costs

When not to use it:

• Pure vector similarity workloads (Redis or Milvus are faster)
• Simple setups where the hybrid search capability is not needed

Choosing the Right Backend

Here is a decision tree:

Are you experimenting or learning AutoGPT?
  → Use local

Are you running a task that completes in one session?
  → Use Redis (fast, simple cleanup)

Do you need memory to persist across multiple agent runs?
  → Do you have cloud budget? → Pinecone
  → Do you want self-hosted?  → Milvus (high volume) or Weaviate (hybrid search)

Are you building a long-running production agent?
  → Pinecone (low ops) or Milvus (control)

Memory Configuration Best Practices

Regardless of backend, a few practices apply universally:

Limit memory size: Large memory stores slow down retrieval and increase embedding costs. Set a maximum number of entries and implement a forgetting policy — remove entries older than N days or with low relevance scores.

Namespace by task: If you run multiple agents, use separate namespaces or collections per agent. Mixing memories from different task contexts degrades retrieval quality.

# Separate namespaces for different agents
MEMORY_INDEX=research_agent_v2

Monitor embedding costs: Every memory write calls OpenAI's embedding API. At $0.0001 per 1K tokens, a research task creating 500 memory entries of 200 tokens each costs about $0.01 in embeddings alone — but this adds up in production.

Back up Pinecone data: Despite being a managed service, Pinecone does not offer automatic backups on the free tier. Export your index data periodically if it represents significant agent work.

For context on how these memory patterns compare to LangChain's memory implementation, see Build AI agent with LangChain and LangChain tutorial 2025.

If you are comparing AutoGPT against BabyAGI specifically on memory architecture, AutoGPT vs BabyAGI covers that comparison in depth.

Memory Performance Benchmarks

Testing with a standard 500-entry memory store on a local machine (M2 MacBook Pro):

Redis wins on pure speed for local deployments. Pinecone's network latency is offset by zero infrastructure management. For anything over 10,000 entries, Milvus is the performance leader.

Frequently Asked Questions

Can AutoGPT use multiple memory backends at the same time? No, AutoGPT uses a single memory backend configured via MEMORY_BACKEND in your .env file. If you need hybrid storage (e.g., fast lookup plus persistent storage), you would need to implement a custom memory provider.

What happens to AutoGPT's memory when I restart the agent? It depends on your backend. Local file and Redis (with persistence) retain memory across restarts. In-memory Redis without AOF/RDB persistence loses all data on restart. Pinecone, Milvus, and Weaviate always retain data since they are external services.

How much memory does AutoGPT typically use? Typical AutoGPT tasks generate 50–500 memory entries. At 1,536 dimensions per embedding (OpenAI ada-002), each entry takes about 6KB in a vector store. A full research task with 200 entries uses roughly 1.2MB of vector storage.