OCR Config Object Structure

Overview

The ocr object allows you to configure Optical Character Recognition (OCR) settings for the application, enabling the extraction of text from images. This section provides a detailed breakdown of the ocr object structure.

There are 5 fields under ocr:

• mistralModel
• apiKey
• baseURL
• strategy
• allowedAddresses

Notes:

• OCR is only active when librechat.yaml has an ocr section. Setting OCR_API_KEY alone does not turn it on. For Mistral's hosted API, strategy: "mistral_ocr" plus OCR_API_KEY in .env is enough, since apiKey and baseURL default to ${OCR_API_KEY} and ${OCR_BASEURL} (or https://api.mistral.ai/v1 when that is unset).
• OCR runs on files added as text: Upload as Text in chat and File Context in the Agent Builder. Agents also need the ocr capability in endpoints.agents.capabilities, which is included by default. Without it, a configured OCR strategy rejects the upload with "OCR capability is not enabled for Agents".
• Which file types go to OCR is controlled by fileConfig.ocr.supportedMimeTypes.
• The default strategy is mistral_ocr. When mistralModel is empty, mistral-ocr-latest is used.
• Azure-deployed Mistral OCR models can be used by setting the strategy to azure_mistral_ocr.
• Google Vertex AI-deployed Mistral OCR models can be used by setting the strategy to vertexai_mistral_ocr.
  • Requires the GOOGLE_SERVICE_KEY_FILE environment variable to be set with service account credentials
  • The service key can be provided as: file path, URL, base64 encoded JSON, or raw JSON string
  • Project ID and location are automatically extracted from the service account credentials
• Local text extraction is available via document_parser, which extracts text from PDF, DOCX, XLS/XLSX, ODT, and ODS files up to 15 MB without any external API.
  • Uses pdfjs-dist, mammoth, and SheetJS locally, no API key or base URL needed
  • Only the strategy field is required; apiKey, baseURL, and mistralModel are ignored
• custom_ocr is accepted by the schema but has no implementation yet.
• Environment variable parsing is supported for apiKey, baseURL, and mistralModel parameters.

Automatic Document Parsing (No Configuration Required)

The built-in document_parser runs automatically for agent file uploads even when no ocr block is configured in your librechat.yaml. This means PDF, DOCX, XLS/XLSX, and ODS files are parsed out of the box without any setup.

The resolution logic works as follows:

1. No ocr config exists: When an agent context file is uploaded and its MIME type matches a supported document type (PDF, DOCX, Excel, ODS), the document_parser is used directly. No OCR capability check is required for the agent.

2. ocr config exists: The configured strategy (e.g., mistral_ocr) is tried first. If the configured strategy fails at runtime, the document_parser is used as a fallback so text extraction still succeeds for supported document types.

3. Neither succeeds: If both the configured strategy and the document parser fail (e.g., the file is an image-only PDF with no embedded text), an error is returned suggesting that an OCR service is needed.

Example

ocr:
  mistralModel: "mistral-ocr-latest"
  apiKey: "your-mistral-api-key"
  strategy: "mistral_ocr"

Example with Azure Mistral OCR:

ocr:
  mistralModel: "deployed-mistral-ocr-2503" # should match deployment name on Azure
  apiKey: "${AZURE_MISTRAL_OCR_API_KEY}" # arbitrary .env var reference
  baseURL: "https://your-deployed-endpoint.models.ai.azure.com/v1" # hardcoded, can also be .env var reference
  strategy: "azure_mistral_ocr"

Example with Google Vertex AI Mistral OCR:

ocr:
  mistralModel: "mistral-ocr-2505" # model name as deployed in Vertex AI
  strategy: "vertexai_mistral_ocr"

For Vertex AI, deploy a Mistral OCR model in your Google Cloud project, create a service account that can call the Vertex AI endpoint, and point GOOGLE_SERVICE_KEY_FILE at its key. The variable accepts a file path, a URL to fetch the key from, base64-encoded JSON, or the raw JSON string:

GOOGLE_SERVICE_KEY_FILE=/path/to/service-account-key.json

Example with local document parser (no external API needed):

ocr:
  strategy: "document_parser"

mistralModel

ocr:
  mistralModel: "mistral-ocr-latest"

For Azure deployments:

ocr:
  mistralModel: "deployed-mistral-ocr-2503" # Your Azure deployment name

For Google Vertex AI deployments:

ocr:
  mistralModel: "mistral-ocr-2505" # Your Vertex AI model name

apiKey

ocr:
  apiKey: "your-ocr-api-key"

baseURL

ocr:
  baseURL: "https://your-ocr-service.com/api"

allowedAddresses

OCR requests block private, loopback, link-local, and cloud-metadata destinations at connect time and disable redirects. allowedAddresses is an exemption list for exact trusted private host:port, private.ip:port, or [ipv6]:port targets; it does not restrict access to public destinations.

ocr:
  strategy: "mistral_ocr"
  baseURL: "http://ocr.internal:8080/v1"
  apiKey: "${INTERNAL_OCR_KEY}"
  allowedAddresses:
    - "ocr.internal:8080"

strategy

ocr:
  strategy: "document_parser"

Available Strategies:

• mistral_ocr: Uses Mistral's OCR through the standard Mistral API.
• azure_mistral_ocr: Uses Mistral OCR models deployed on Azure AI Foundry. Set mistralModel to your deployment name and baseURL to the deployment endpoint.
• vertexai_mistral_ocr: Uses Mistral OCR models deployed on Google Cloud Vertex AI, authenticated with GOOGLE_SERVICE_KEY_FILE.
• document_parser: Uses local text extraction for PDF, DOCX, XLS/XLSX, and OpenDocument files. No external API needed. Also runs automatically for agent file uploads when no ocr config is present, and as a fallback when a configured OCR strategy fails.
• custom_ocr: Reserved for a custom OCR service at baseURL. Not implemented yet.