Docs
✨ Features
Agents

Agents: Build Custom AI Assistants

LibreChat’s AI Agents feature provides a flexible framework for creating custom AI assistants powered by various model providers.

This feature is similar to OpenAI’s Assistants API and ChatGPT’s GPTs, but with broader model support and a no-code implementation, letting you build sophisticated assistants with specialized capabilities.

Getting Started

To create a new agent, select “Agents” from the endpoint menu and open the Agent Builder panel found in the Side Panel.

Agents - Endpoints Menu

The creation form includes:

  • Avatar: Upload a custom avatar to personalize your agent
  • Name: Choose a distinctive name for your agent
  • Description: Optional details about your agent’s purpose
  • Instructions: System instructions that define your agent’s behavior
  • Model: Select from available providers and models

Existing agents can be selected from the top dropdown of the Side Panel.

  • Also by mention with ”@” in the chat input.

Agents - Mention

Model Configuration

The model parameters interface allows fine-tuning of your agent’s responses:

  • Temperature (0-1 scale for response creativity)
  • Max context tokens
  • Max output tokens
  • Additional provider-specific settings

Agent Capabilities

Note: All capabilities can be toggled via the librechat.yaml configuration file. See docs/configuration/librechat_yaml/object_structure/agents#capabilities for more information.

Code Interpreter

When enabled, the Code Interpreter capability allows your agent to:

  • Execute code in multiple languages, including:
    • Python, JavaScript, TypeScript, Go, C, C++, Java, PHP, Rust, and Fortran
  • Process files securely through the LibreChat Code Interpreter API
  • Run code without local setup, configuration, or sandbox deployment
  • Handle file uploads and downloads seamlessly
  • More info about the Code Interpreter API

File Search

The File Search capability enables:

  • RAG (Retrieval-Augmented Generation) functionality
  • Semantic search across uploaded documents
  • Context-aware responses based on file contents
  • File attachment support at both agent and chat thread levels

File Context (using Optical Character Recognition)

The File Context (OCR) capability allows your agent to extract and process text from images and documents:

  • Extract text while maintaining document structure and formatting
  • Process complex layouts including multi-column text and mixed content
  • Handle tables, equations, and other specialized content
  • Work with multilingual content
  • More info about OCR
    • Currently uses Mistral OCR API which may incur costs

Artifacts

The Artifacts capability enables your agent to generate and display interactive content:

  • Create React components, HTML code, and Mermaid diagrams
  • Display content in a separate UI window for clarity and interaction
  • Configure artifact-specific instructions at the agent level
  • More info about Artifacts

When enabled, additional instructions specific to the use of artifacts are added by default. Options include:

  • Enable shadcn/ui instructions: Adds instructions for using shadcn/ui components (a collection of re-usable components built using Radix UI and Tailwind CSS)
  • Custom Prompt Mode: When enabled, the default artifacts system prompt will not be included, allowing you to provide your own custom instructions

Configuring artifacts at the agent level is the preferred approach, as it allows for more granular control compared to the legacy app-wide configuration.

If you enable Custom Prompt Mode, you should include at minimum the basic artifact format in your instructions. Here’s a simple example of the minimum instructions needed:

When creating content that should be displayed as an artifact, use the following format:

:::artifact{identifier="unique-identifier" type="mime-type" title="Artifact Title"}

Your artifact content here

:::

For the type attribute, use one of:
- "text/html" for HTML content
- "application/vnd.mermaid" for Mermaid diagrams
- "application/vnd.react" for React components
- "image/svg+xml" for SVG images

Tools

Agents can also be enhanced with various built-in tools:

  • DALL-E-3: Image generation from text descriptions

  • Tavily Search: Advanced search API with diverse data source integration

  • Calculator: Mathematical calculations

  • Google Search: Access to web search functionality

  • Stable Diffusion: Text-to-image generation

  • Azure AI Search: Information retrieval

  • Traversaal: A robust search API for LLM Agents

  • Wolfram: Computational and mathematical capabilities

  • Tools can be disabled using the librechat.yaml configuration file:

Actions

With the Actions capability, you can dynamically create tools from OpenAPI specs to add to your Agents.

Agents - Endpoints Menu

Clicking the button above will open a form where you can input the OpenAPI spec URL and create an action:

Agents - Endpoints Menu

Note that you can add add the ‘x-strict’: true flag at operation-level in the OpenAPI spec for actions. If using an OpenAI model supporting it, this will automatically generate function calls with ‘strict’ mode enabled. Strict mode supports only a partial subset of json. Read https://platform.openai.com/docs/guides/structured-outputs/some-type-specific-keywords-are-not-yet-supported for details.

Agent Chain

The Agent Chain capability enables a Mixture-of-Agents (MoA) approach, allowing you to create a sequence of agents that work together:

  • Create chains of specialized agents for complex tasks
  • Each agent in the chain can access outputs from previous agents
  • Configure the maximum number of steps for the agent chain
  • Note: Access this feature from the Advanced Settings panel in the Agent Builder
  • Note: This feature is currently in beta and may be subject to change
    • The current maximum of agents that can be chained is 10, but this may be configurable in the future
Agent Chain

This feature introduces a layered Mixture-of-Agents architecture to LibreChat, where each agent takes all the outputs from agents in the previous layer as auxiliary information in generating its response, as described in the eponymous “Mixture-of-Agents” paper.

Advanced Settings

Advanced settings for your agent (found in the Advanced view of the Agent form) outside of “capabilities.”

Max Agent Steps

This setting allows you to limit the number of steps an agent can take in a “run,” which refers to the agent loop before a final response is given.

If left unconfigured, the default is 25 steps, but you can adjust this to suit your needs. For admins, you can set a global default as well as a global maximum in the librechat.yaml file.

A “step” refers to either an AI API request or a round of tool usage (1 or many tools, depending on how many tool calls the LLM provides from a single request).

A single, non-tool response is 1 step. A singular round of tool usage is usually 3 steps:

  1. API Request -> 2. Tool Usage (1 or many tools) -> 3. Follow-up API Request

Model Context Protocol (MCP)

MCP is an open protocol that standardizes how applications provide context to Large Language Models (LLMs), acting like a universal adapter for AI tools and data sources.

Imagine MCP as the “USB-C of AI” - just as USB-C provides a universal connection standard for electronic devices, MCP offers a standardized way to connect AI models to diverse tools, data sources, and services.

By configuring mcpServers in the librechat.yaml file, you can:

  • Add custom tools from various sources
  • Integrate specialized data access tools
  • Extend AI capabilities beyond default offerings

Here are some tools added by configuring the “Filesystem” MCP server: Agents - MCP

Key Features:

  • Upon app restart, tools are available in the “Add Tools” button within the Agent Builder panel
  • Seamlessly connect custom tools and data sources
  • Leverage a growing ecosystem of MCP-compatible servers and integrations

Learn More:

⚠️ MCP Limitations

Most existing MCP servers use stdio processes, which work well for local, single-user environments but pose significant challenges for multi-user and production platforms.

The Challenges

  • STDIO servers are designed for local, single-user contexts
  • Not scalable for remote or cloud deployments
  • Limited multi-user support (concurrency, authentication, security)

Emerging Solutions

Server-Sent Events (SSE) are being explored as a more scalable transport mechanism. MCP developers are actively working on remote connection support, addressing complexities around deployment, authentication, and security.

Current Status: Promising, but still a work in progress.

LibreChat is at the forefront of implementing flexible, scalable MCP server integrations to support diverse usage scenarios.

File Management

Agents support four distinct file upload categories:

  1. Image Upload: For visual content processing
  2. File Search Upload: Documents for RAG capabilities
  3. Code Interpreter Upload: Files for code processing
  4. File Context (OCR): Documents processed with OCR and added to the agent’s instructions

Files can be attached directly to the agent configuration or within individual chat threads.

File Context using OCR for agents

Files uploaded as “File Context” are processed using OCR to extract text, which is then added to the Agent’s instructions. This is ideal for documents, images with text, or PDFs where you need the full text content of a file to be available to the agent. Note, the OCR is performed at the time of upload and is not stored as a separate file, rather purely as text in the database.

Sharing and Permissions

Administrator Controls

Administrators have access to global permission settings:

  • Enable/disable agent sharing across all users
  • Control agent usage permissions
  • Manage agent creation rights
  • Configure platform-wide settings

The use of agents for all users can also be disabled via config, more info.

User-Level Sharing

Individual users can:

  • Share their agents with all users (if enabled)
  • Control editing permissions for shared agents
  • Manage access to their created agents

Notes

  • Instructions, model parameters, attached files, and tools are only exposed to the user if they have editing permissions
    • An agent may leak any attached data, whether instructions or files, through conversation—make sure your instructions are robust against this
  • Only original authors and administrators can delete shared agents
  • Agents are private to authors unless shared

Optional Configuration

LibreChat allows admins to configure the use of agents via the librechat.yaml file:

  • Disable Agents for all users (including admins): more info
  • Customize agent capabilities using: more info

Best Practices

  • Provide clear, specific instructions for your agent
  • Carefully consider which tools are necessary for your use case
  • Organize files appropriately across the four upload categories
  • Review permission settings before sharing agents
  • Test your agent thoroughly before deploying to other users

Recap

  1. Select “Agents” from the endpoint dropdown menu
  2. Open the Agent Builder panel
  3. Fill out the required agent details
  4. Configure desired capabilities (Code Interpreter, File Search, File Context or OCR)
  5. Add necessary tools and files
  6. Set sharing permissions if desired
  7. Create and start using your agent

What’s next?

LibreChat Agents usher in a new era for the app where future pipelines can be streamlined via Agents for specific tasks and workflows across your experience in LibreChat.

Future updates will include:

  • General improvements to the current Agent experience
  • Multi-agent orchestration for complex workflows
  • Ability to customize agents for various functions: titling (chat thread naming), memory management (user context/history), and prompt enhancement (input assistance/predictions)
  • More tools, configurable tool parameters, dynamic tool creation.

Furthermore, the update introduces a new paradigm for LibreChat, as its underlying architecture provides a much needed refresh for the app, optimizing both the user experience and overall app performance.

To highlight one notable optimization, an AI generation of roughly 1000 tokens will transfer about 1 MB of data using traditional endpoints (at the time of writing, any endpoint option besides Agents and AWS Bedrock).

Using an agent, the same generation will transfer about about 52 kb of data, a 95% reduction in data transfer, which is that much less of a load on the server and the user’s device.

AI Agents in LibreChat provide a powerful way to create specialized assistants without coding knowledge while maintaining the flexibility to work with your preferred AI models and providers.

#LibreChat #AIAssistants #NoCode #OpenSource

OverviewCode Interpreter API