RAG
How AI turns uploaded files into retrieval-ready context for chat and assistants.
Files & RAG in AI turns uploaded documents and standalone images into retrieval context for chat, assistants, and other retrieval-enabled workflows. This page explains the ingestion lifecycle and gives AI engineers the API steps, configuration prerequisites, and failure handling for image ingestion.
At a high level, the flow is:
- a user uploads a file through the platform
- the backend stores the source file and creates a library record
- a durable background workflow parses a document or describes an image, then splits the resulting text into chunks
- embeddings are generated and written to
Milvusin bounded batches - chat and assistant flows retrieve matching chunks later through RAG
What This Capability Covers
This capability is broader than file upload alone.
It includes:
- the user-facing file library in the
Portal - file metadata and status tracking in the backend
- source-file storage in the configured
S3-compatible object store - background indexing through
Temporal - document chunking and normalization through
Docling - standalone image descriptions and text transcription through a tenant-selected vision model
- vector storage and similarity search in
Milvus - retrieval use inside chat and assistant flows
User-Facing Entry Points
The main user-facing entry points are:
- the
Filessection in thePortal - the chat Files picker in the portal composer
The Files area is where users upload, organize, rename, delete, reprocess, and share content.
The chat Files control is where users decide which files or folders should be used as retrieval context for a conversation.
Core Processing Flow
The backend implementation currently spans multiple runtime layers.
Where The Data Lives
Different parts of the capability live in different systems.
| Data type | Main system |
|---|---|
| Original uploaded file | The organization's own object storage bucket; in the Global organization, the configured platform bucket. Both are S3-compatible storage, either in-cluster or external |
| File library metadata and status | PostgreSQL |
| Chunk lists and derived artifacts | The organization's own object storage bucket; in the Global organization, the {bucket}-artifacts bucket |
| Searchable vectors and retrieval payloads | Milvus |
A file keeps one chunk list for each chunk size and chunking type it was built with since it was last processed in full, so a reindex back to earlier values reuses the list. An image file has one list with its description. A file whose chunks were saved in the earlier single-list layout keeps that one list, which a reindex reuses only for the chunk size and chunking type the file was built with; other values parse the file again. Lists hold chunks without overlap; the overlap is added when the chunks are embedded.
This boundary matters.
The file row shown in the portal is not the retrieval index itself. The portal library is driven by database metadata, while actual retrieval depends on vectorized chunks in Milvus.
Tenant-scoped vector collections
BullSequana AI uses one shared Milvus service and separate platform-managed collections for each tenant. It does not deploy a separate Milvus instance for every tenant.
The backend names each collection with the tenant's prefix and a generated suffix, and records the embedding model and dimension with the tenant. A tenant's first embedding-model save creates its collection. Changing the embedding model creates a new tenant-scoped target collection through the reindex workflow. The BSQAI API measures the new model's dimension with one embedding before the reindex starts.
Collection listing, statistics, document inspection, and deletion operations expose only names in the active tenant's collection namespace. Collection search and Responses API file search accept only the tenant's active collection. A foreign or inactive collection identifier is returned as not found, and Responses API file search requires exactly one vector store identifier.
A completed reindex deletes the tenant-scoped collection that search used before it. A tenant whose active collection is outside its own collection namespace, such as a collection shared before collections were tenant-scoped, keeps using it until a reindex completes. The reindex then switches the tenant to its own collection and leaves the old one in place, because another tenant may still use it.
Hybrid retrieval and collection compatibility
Retrieval defaults to Hybrid: Milvus ranks chunks by both embedding similarity and BM25 keyword matching, then combines the rankings using reciprocal rank fusion. This helps queries containing exact identifiers, product names, or years while retaining semantic matches. Dense mode uses embedding similarity alone. File access checks and selected document/folder scope apply in either mode.
Administrators select the mode, returned chunk count, and ranking merge constant in tenant retrieval settings. The optional reranker runs after retrieval; it is not the BM25 keyword index.
Recover a collection without BM25 fields
collection_missing_bm25_field means the collection predates the sparse BM25 field. Milvus cannot add that field to the existing collection. Changing to Hybrid or repeating a same-model, same-dimension reindex does not recreate its schema.
For platform operators:
- Record the active tenant, embedding model, dimension, and collection name. Preserve original files, cached chunks, library metadata, and the relevant backup before changing the collection.
- Select Dense temporarily if existing searches need to continue while a migration is prepared. This is a search compatibility measure, not a fix for ingestion into an incompatible collection.
- Plan a reindex into a fresh collection using an intentionally changed, supported model or document processing setting. Collection names are generated by the platform, not entered in the Portal.
- The active collection cannot be deleted, and saving the model and settings search already uses only adds missing files to it. Such a reindex does not rebuild the collection's schema.
- Start embedding reindexing with the intended model and settings through the tenant's General settings or
POST /v1/config/embedding/reindex/start. The new collection receives the current schema. When the reindex completes, the platform processes files with no reusable chunks and failed files again from their preserved originals, with the tenant's document processing settings or the values a file was uploaded with, and default OCR settings. The progress counts them asreprocess_total,reprocessed, andreprocess_failed. Inspect those counts and file stages, not only the completion status, and reprocess files that failed again. - After successful indexing, select Hybrid, check an exact-keyword query and a semantic query against known documents, and confirm unauthorized files remain excluded. Verify that a new upload becomes searchable.
While the new collection is built, search stays on the old active collection until the reindex completes. If the reindex fails, repair the reported error and save the same settings again; the reindex continues with the files it could not rebuild.
Status Lifecycle
Each file records its ingestion stage in upload_stage and whether the run failed in upload_failed. Stages run in this order:
pending: the upload or reprocess request was accepted and the workflow is queuedparsing: the worker downloads the source file and checks its content against the type stored at upload. Docling converts a document and splits it into chunks with metadata; this is usually the longest step. A standalone image is validated. On reprocess, the file's older vectors are removed first.extracting_artifacts: Docling extracts the figures and tables of a document, when the tenant's Extract figures and tables setting is on.describing_pictures: the vision model describes a standalone image, or the pictures inside a document. Files with nothing to describe skip this stage.embedding: the worker saves the chunk list, adds the chunk overlap, embeds the chunks in batches, and inserts them intoMilvussaving: the platform records the collections the file is in, the document processing values each was built with, and the file's artifacts, and marks the file ready once it is in every collection the tenant's embedding state requires; the embedding model belongs to the tenant, not to the fileavailable: indexing is complete
upload_failed is true when the run failed, timed out, or was cancelled. The stage stays at the step where it failed. Reprocessing starts over at pending and clears the flag.
upload_warning names content that a successful run skipped. It is null when nothing was skipped, or one of these values:
artifacts_not_extracted: the Docling conversion that extracts tables and figures failed. The document text is indexed and searchable, but the file has no table or figure artifacts.pictures_not_described: an image description model is configured, the file is a PDF, Word, PowerPoint, or Excel file, and at least one of its pictures has no description. The description request failed, the picture is larger than 20,000,000 pixels, or its image could not be stored.
The warning belongs to the file's build in the collection that search uses. Processing the file again replaces it: a reprocess, or the retry of failed files after a reindex. A reindex with figures and tables on retries a failed extraction and the missing picture descriptions, and the file's warning changes when search switches to the new collection. A file with a warning stays ready for search.
A file is ready for search when its stage is available and upload_failed is false. GET /v1/files/children, GET /v1/files/search, and POST /v1/files/move return upload_stage, upload_failed, and upload_warning for each file. For a ready file they also return built_with: the chunking_type, max_tokens, chunk_overlap_tokens, and extract_artifacts it is searched with, and has_overrides, which is true when those values were passed through the API on upload or reprocess.
This is why upload success is not the same as retrieval readiness.
A file can appear in the portal library before it is usable in grounded chat.
How Uploads Become Retrieval-Ready
Upload through POST /v1/files/upload-stream with a multipart/form-data body containing the file field. Optional upload settings, such as folder_id, are query parameters. The Portal uses this endpoint too. See Access and tenant context for authentication and tenant selection.
The endpoint does more than store bytes. It:
- checks the file's name and the first 8 KiB of its content before storing anything
- streams the file directly to its permanent storage location, chosen by the platform; clients do not pass a storage bucket
- creates the library record and its ownership and folder relationships
- sets the initial stage to
pending - starts a
Temporalworkflow onfile-events-queue
The endpoint decides the file's type once, from its name and content, and ignores the Content-Type of the multipart part:
- A name that ends in one of the extensions that
GET /v1/files/upload-typesreturns, in any letter case, gets that extension's type when the content matches it. A file whose content does not match, such as a PDF namednotes.txtor a text file namedreport.pdf, gets400with the error codefile_type_mismatchand the message "This file's content does not match its extension." - A name without one of these extensions is accepted when its content is UTF-8 text, a PNG, JPEG, or WebP image, or a PDF. A PDF uploaded without its extension is processed as a PDF, and its chunks carry the file's own name as
document_name. - Any other file gets
400with the error codeunsupported_file_format.
A refused file stores nothing. The stored type is returned as content_type in the upload response and as mime_type in file listings, and every later processing step uses it.
The file is processed with the tenant's document processing settings. The query parameters chunking_type (hybrid or hierarchical), max_tokens, chunk_overlap_tokens, and extract_artifacts override them for this file. An overlap above half the chunk size it resolves to gets 422 with the error code chunk_overlap_too_large. A file uploaded with any of these values keeps them through later reindexes.
A successful response is 202 Accepted and includes the file id and workflow_id. Track the workflow with GET /v1/files/workflow/{workflow_id} and wait for the file to reach stage available without upload_failed before using it for retrieval. The status response also carries the file's upload_warning. The actual indexing work does not happen inside the upload request itself.
To rename a file, call POST /v1/files/move with new_name. A rename that changes the type derived from the file name, such as report.pdf to report.txt or report, gets 400 with the error code file_bad_request, and the file stays unchanged. Dots in the name, a change of the extension's letter case, and extensions of the same type are allowed: .jpg and .jpeg, or .txt and code or configuration extensions such as .py, .ts, and .yaml, which are all plain text.
To download a file, call GET /v1/files/download?file_id={id}. It returns a time-limited download_url (one hour by default, set with expires_in_seconds) that saves the file under its current name.
To download a table or figure extracted from a file, add its artifact_id: GET /v1/files/download?file_id={id}&artifact_id={artifact_id}. The same read check as for the file applies. An artifact_id that does not belong to that file returns 404 with the error code file_artifact_not_found. Search results and the chunk listing carry the artifact_id of table and figure chunks.
Instead, a separate file-event worker performs the expensive background steps later. If that worker is not running, files can remain stuck in pending even though the upload returned success.
Standalone image ingestion
Standalone images use the same upload endpoint and Temporal workflow as documents. A validated image bypasses Docling parsing, Docling OCR settings, and artifact extraction. The tenant-selected vision model describes visible content and transcribes readable text. The service keeps that output as one chunk, whatever the chunk size, caches it, embeds it, and replaces the file's existing vectors. Retrieval searches the generated text, not image embeddings.
Supported formats and limits
| Check | Requirement |
|---|---|
| Format | PNG (image/png), JPEG (image/jpeg), or WebP (image/webp). |
| Frames | Exactly one frame. Animated PNG and WebP, GIF, TIFF, SVG, and other image formats are unsupported. |
| Pixel count | Width × height must be at most 20,000,000 pixels. |
| Integrity | The image must fully decode; corrupt and truncated data fail validation. |
| Upload size | The tenant's file_handling.max_file_size_mb limit also applies. The default is 100 MiB, calculated as 100 × 1,024 × 1,024 bytes. |
The worker inspects the image bytes; changing a filename or MIME type does not convert an unsupported image. Image validation happens during background ingestion, so a successful upload can still fail at the parsing stage.
The vision model reads a copy of the image whose longer side is at most 2,048 pixels, in the image's own format; smaller images are sent unchanged, and the stored file keeps its size.
Configure and upload an image
Before uploading, ask a tenant administrator to configure vision.image_description_model. Upload and reprocess requests cannot choose a different model. The selected model route must accept image input, and its connection must be reachable from the file-event worker. The setting defaults to null; without a model, standalone image ingestion fails at the describing_pictures stage.
Use a JWT and tenant context as described in API access. Upload requires storage.object.create and write access to the destination folder. Workflow status requires storage.object.read and read access to the file. Reprocessing requires storage.object.update and write access to the file.
With BSQAI_BASE_URL set to https://api.<platform-domain>/v1, upload a supported local image:
curl --fail-with-body "$BSQAI_BASE_URL/files/upload-stream" \
-H "Authorization: Bearer $BSQAI_TOKEN" \
-H "X-Tenant-Id: $BSQAI_TENANT_ID" \
-F "file=@diagram.png;type=image/png"Keep the returned id and workflow_id. Use the workflow ID to check processing:
export WORKFLOW_ID="<workflow_id-from-upload>"
curl --fail-with-body "$BSQAI_BASE_URL/files/workflow/$WORKFLOW_ID" \
-H "Authorization: Bearer $BSQAI_TOKEN" \
-H "X-Tenant-Id: $BSQAI_TENANT_ID"Wait for the file to reach stage available with upload_failed set to false, then use its ID in your application's file-search scope. Verify retrieval with a question about a visible label or object. Descriptions are model-generated and can contain errors; compare important details with the original image.
To obtain a presigned download_url for the original image, use GET /v1/files/download?file_id={file_id}; file read permission is required.
Image ingestion failures
GET /v1/files/workflow/{workflow_id} returns the Temporal workflow status, such as RUNNING, COMPLETED, or FAILED, its start and close times, upload_stage, the file's current stage, and upload_warning. Use the workflow_id from the upload response, for example file-event:<file-id>:created:none; a reprocess returns file-event:<file-id>:created:reprocess. A bare file ID is rejected with HTTP 400 and error code file_bad_request. The response has no failure code or failure reason.
To see whether a file failed, read its upload_stage and upload_failed through GET /v1/files/children or GET /v1/files/search. The stage of a failed file shows the step that failed:
| Failed stage | Likely cause | Recovery |
|---|---|---|
parsing | The image cannot be decoded, is truncated, exceeds the pixel limit, has an unsupported format, or has multiple frames. | Convert it to a valid static PNG, JPEG, or WebP within the limit and upload the corrected file. |
describing_pictures | No description model is configured, the provider rejects the image request, the model is unavailable, or it returns empty text. | Check the tenant model alias, its image input, credentials, quotas, and worker connectivity, then reprocess the file. |
These are asynchronous ingestion failures; the earlier upload response remains 202 Accepted. Saving an invalid description model in the tenant configuration is a separate, synchronous error: HTTP 400 with invalid_image_description_config.
Docling, Chunking, And Embeddings
Supported file types include documents (pdf, docx, md, pptx, txt, vtt), data files (json, csv, xlsx), code files (py, ts, js, java, go, rs, and others), config files (yaml, yml, toml, ini), and images (png, jpg, jpeg, webp). GET /v1/files/upload-types returns the full list. OCR is enabled by default with automatic engine selection.
For standard document formats, the worker uses Docling-based processing to chunk and normalize the source material.
That processing can also produce additional artifacts such as chunk JSON and, where applicable, metadata, images, or tables.
Files are chunked with the tenant's document processing settings: its chunking type and its chunk size in tokens. The chunk size ranges from 1 to 8,192 tokens. Each figure, table, and image description is one chunk, never split by the chunk size. A figure, table, image description, or text chunk over 60,000 bytes of UTF-8 text, about 60,000 characters, is stored in parts, because Milvus holds at most 65,535 bytes per chunk. Each part of a table repeats its caption, header row, and separator line.
When the tenant sets a chunk overlap, each text chunk starts with the last tokens of the text chunk before it, cut at a word boundary. Figures, tables, and image descriptions get no overlap and give none. The overlap is added when the chunks are embedded: the saved chunk lists stay without it, while the text that search returns and chat receives includes it. A large chunk size with an overlap, such as 4,096 tokens, can exceed the embedding model's input limit; such a chunk is embedded in pieces combined into one vector.
The processing path includes several safeguards for large documents, busy queues, and transient service failures:
- Docling conversion runs as a queued job, so processing is not tied to one API request or backend pod
- the worker sends liveness heartbeats while it polls Docling and while it embeds batches
- queue wait is treated as normal load, with separate execution and end-to-end workflow budgets
- missing Docling task IDs fail immediately, while a small number of transient polling failures are tolerated before the job fails
- OCR selection is resolved once per document and reused for parsing and artifact extraction
- the chunk tokenizer follows the active embedding model, avoiding tokenizer drift after an administrator changes models
- text beyond the embedding model's input limit is split safely and combined into one vector for the source chunk
- embeddings and vector inserts run in bounded batches to reduce peak memory use on large or scanned files
If the chunk tokenizer for the active embedding model cannot be loaded, the document is still chunked, with a default tokenizer or by document structure, and chunk sizes are approximate.
The embedding model's input limit is the Max input tokens of its model route. The BSQAI API counts each chunk's tokens before embedding it, and splits a chunk longer than three quarters of that limit, or longer than 2,048 tokens when the route sets no limit. Retrieval queries are split the same way.
Artifact extraction is best-effort: a failed image or table extraction does not discard otherwise valid text chunks. When the Docling conversion for artifacts fails, the file becomes ready for search with upload_warning set to artifacts_not_extracted.
Each stored table and figure gets a record with its own ID, and the chunk that holds its text carries that ID and the artifact's file name. A table whose file could not be stored keeps its text as a table chunk without an artifact reference; the document's text chunks hold its cells as well. An artifact's ID comes from its file and its name, so reprocessing the file keeps the ID of each table or figure it extracts again. Reprocessing replaces the artifact records, and deleting the file removes them. Files processed before artifact records existed get them when they are reprocessed.
Document pictures
When the tenant's Extract figures and tables setting (document_processing.extract_artifacts) is on, the default, Docling decides which pictures a document has, and the worker stores each picture once as img_0.png, img_1.png, and so on, in reading order. For a PDF, the stored image is Docling's render of the picture at 144 dpi, twice its size on the page in points. For a Word, PowerPoint, or Excel file, it is the image file inserted into the document, at its own size. Charts, shapes, and diagrams drawn natively in Office files are not pictures to Docling, so they are not stored.
A picture with extracted text or a caption becomes a figure chunk with its page and an artifact reference. A picture without text is stored and can be downloaded by its artifact_id, but adds no chunk unless a description is generated for it. When its image could not be stored, a picture keeps its text as a figure chunk without an artifact reference.
A picture that Docling returns without a usable image is left out, because processing the file again returns the same result. Pictures in Markdown files are always left out: Docling returns no image for a linked or an embedded Markdown image. A caption stays in the document's text chunks.
Optional document picture descriptions
When vision.image_description_model is also configured, the worker describes each stored picture from PDF, Word (.docx), PowerPoint (.pptx), and Excel (.xlsx) files. It adds the description to the picture's extracted text or caption, or uses the description alone when there is none. The original document text chunks remain intact.
The model reads a copy of the picture whose longer side is at most 2,048 pixels; smaller pictures are sent unchanged, and the stored image keeps its size. A picture larger than 20,000,000 pixels, the same limit as for standalone images, is not described.
The vision model is optional for documents. Without it, the pipeline retains available extracted text and captions. If a picture cannot be loaded, validated, or described, processing falls back to that text. A picture with no remaining text is not searchable; the file keeps it, so a later reindex with an image description model set describes it. A picture-description failure alone does not fail otherwise valid document ingestion.
When a model is configured and a picture gets no description, the file becomes ready for search with upload_warning set to pictures_not_described. This includes pictures over the pixel limit and pictures whose image could not be stored. Without a model, documents get no picture warning.
Embedding the resulting chunks
After chunking, embeddings are generated for the chunks and inserted into Milvus. If ingestion ultimately fails, times out, or is cancelled, the platform marks the file failed and attempts to remove partial vectors from that attempt; a cleanup that cannot reach the vector store does not delay the failed mark. A run stops ingestion 10 minutes before its 3-hour timeout so that it can still mark the file failed. Removal is best-effort: if the vector store itself is unavailable, some vectors can remain until the file is reprocessed, which clears them first, or deleted. Retrieval never uses them, because it only searches files at stage available that are not marked failed. A document that produces no chunks also fails without repeatedly rerunning the same conversion.
Those vectors are what later power retrieval.
How Chat Uses Files Later
The portal chat composer has a Files control that searches the file library and lets users pick:
- individual files
- folders
- all available retrieval-ready content
Only files that are ready for search, at stage available and not marked failed, are selectable in that picker.
When a user submits a chat request with retrieval enabled, the UI sends:
fileSearch- optional
documentIds - optional
folderIds
The backend then resolves the effective retrieval scope.
Folder selections are expanded on the backend into the underlying file IDs for that user, and the final retrieval step uses those document IDs when running similarity search.
Whichever scope is used (selected files, all content, or a folder), the backend keeps only files at stage available that are not marked failed. Files that are still in progress or that failed are left out, even when the user can read them. If no eligible file remains, nothing is searched. Such files still appear in Files with their current stage.
This makes the flow user-friendly in the portal while still keeping retrieval scoped to concrete indexed documents under the hood.
The selected files, folders, and file-search state are stored with the conversation. Reopening a chat restores the last RAG selection instead of silently widening or losing its retrieval scope.
Sources in answers
When an answer uses file search, BullSequana AI records the sources it was based on. Each source identifies:
- the file, by the name shown in Files
- the page, for documents with pages such as PDFs and slide decks
- a short excerpt from the start of the matched passage
The sources are saved with the conversation, so reopening a chat shows the same sources for each answer.
The model also receives each retrieved passage labeled with its file name and page, so an answer can refer to a document by name.
Applications receive the same information through the BSQAI API. See Citations from file search.
Retrieval Boundary
The retrieval path is currently centered around the backend RetrievalService.
In practice, it:
- resolves and searches the active tenant's
Milvuscollection - optionally filters by document IDs
- can apply structured metadata filters
- returns chunk text that is assembled into model context, labeled with file name and page
- returns the list of sources used for the answer
The chat-oriented file_search tool uses this retrieval layer so that the assistant can ground responses in the selected content instead of relying only on base-model knowledge.
The same file-and-folder scoping is available to portal chat, the Responses API file_search tool, and collection search. Explicit file and folder selections are checked through batched OpenFGA authorization calls, then intersected with records in the active tenant before retrieval. If no readable documents remain, the result is a deny-all scope rather than an unfiltered search.
Collection search and chunk listing
POST /v1/collections/{collection_name}/search ranks chunks against a query. POST /v1/collections/{collection_name}/documents lists the chunks of the documents the caller can read, without a query. Both need the storage.object.read permission and take the same scope in the request body:
| Field | Meaning | Limit |
|---|---|---|
document_ids | files to include | 1,000 |
folder_ids | folders whose readable files to include | 100 |
filters | metadata conditions, all of which must match | 20 |
Each filter has a field of up to 64 characters, an operator (== with one value, or in with a list of up to 100 values) and a value of strings or integers. A filter only narrows the readable documents; a field that no chunk has matches nothing. Requests over a limit return 422.
Every result carries the chunk id, the file's document_id, the chunk text in page_content, and metadata with the fields that apply to that chunk:
| Field | Meaning |
|---|---|
document_name | the file's current name |
chunk_index | the chunk's position in the file |
page_no | the page, for documents with pages |
artifact_type | table or figure, for chunks holding an extracted table or figure |
artifact_id, artifact_name | the stored table or figure, for downloading it |
image_description_model | the model that described an image |
The listing returns up to limit chunks per page (default 100, at most 1,000) in chunk ID order, with total, the number of matching chunks; total is omitted when the count cannot be read. When next_cursor is present, pass it as the cursor query parameter to read the next page. A caller who can read none of the requested documents gets an empty page with a total of 0.
Files, Folders, And Scope
The platform treats files and folders differently:
- files are the actual indexed retrieval units
- folders are a user-facing grouping and selection mechanism
That means selecting a folder in chat is really a convenient way to include the files contained in that folder hierarchy.
This also explains why the Files area and the chat Files picker are so closely connected.
Sharing And Access Scopes
Files and folders can be shared with other users or Keycloak groups through the portal Files area.
The backend supports three listing scopes:
| Scope | What it returns |
|---|---|
mine | Files and folders owned by the current user |
shared_with_me | Files and folders accessible to the current user but owned by someone else |
all | Combined scope used internally by the chat RAG search |
The all scope ensures that the chat Files picker surfaces both the user's own files and all files shared with them.
Sharing is enforced through OpenFGA. A user reaches a file as its owner, editor, or viewer: assigned directly, through a group, or inherited from a parent folder that is shared with them. Tenant-administrator status alone does not grant access to another user's private files.
Reprocess And Delete
The Files area is not only for initial upload.
It also supports later lifecycle actions.
Reprocess
Reprocess reruns indexing for an already stored source file.
This is useful when:
- indexing previously failed
- processing settings changed
- the vector representation needs to be rebuilt
Reprocess does not require the user to upload the file again. In Portal Files, a failed file offers Retry indexing, and a ready file offers Reprocess in its action menu, after a confirmation: "The file's text is extracted and indexed again with the current document processing settings. During processing the file is not available for search."
Reprocess builds every part of the file again: its chunk lists, figures, tables, and descriptions are replaced. It uses the tenant's document processing settings. Through the API, POST /v1/files/reprocess accepts the same chunking_type, max_tokens, chunk_overlap_tokens, and extract_artifacts overrides as an upload, with the same 422 chunk_overlap_too_large check. A reprocess without overrides, such as the Portal's, drops values the file was given earlier through the API.
Reprocess checks the stored content against the file's type again. A stored file whose content does not match its extension, such as a PDF named notes.txt that was stored before uploads checked content, fails at the parsing stage with "This file's content does not match its extension." A reindex that changes the chunk size or chunking type fails it the same way, and the file leaves search when that reindex completes. Upload it again under a name with the right extension.
Reprocess is refused with 409 reindex_in_progress while the tenant's embedding reindex runs, and with 409 file_ingestion_in_progress while the file's own upload or reprocess runs. A completed reindex processes failed files again itself.
For an image, send POST /v1/files/reprocess with the JSON body {"file_id":"<uploaded-file-id>"} and the same authentication and tenant headers. Track the returned workflow_ids through the workflow-status endpoint. Reprocessing reads the stored original, regenerates its description using the current tenant model and service prompt, replaces the cached chunks, and rebuilds vectors. Document reprocessing also regenerates enabled picture descriptions.
Reindex versus reprocess
| Operation | What it rebuilds | Use it when |
|---|---|---|
| Reindex, model or overlap change | Embeds each file's saved chunks again without parsing their text. With figures and tables on, a file whose figures were never extracted or failed to extract still goes to Docling, and pictures without a description are described when a vision model is set. | Changing the embedding model or the chunk overlap. |
| Reindex, chunk size or chunking type change | Parses each file's text again. Reuses the file's figures, tables, and image descriptions. | Changing how documents are split into chunks. |
| Reindex, figures and tables turned on | Reuses figures and tables the file already has, and extracts them only where they were never extracted or the extraction failed. With a vision model set, describes pictures that have no description. | Adding figures and tables to search. |
| Reindex, figures and tables turned off | Leaves the file's figures and tables out of the new collection. | Removing figures and tables from search. |
| File reprocess | Reads the original, parses it, extracts figures and tables, and generates descriptions again, then replaces the chunk lists and vectors. | Retrying ingestion, or applying a changed description model or prompt to one file. |
After any reindex, files without saved chunks are processed again in full when it completes. Reprocessing replaces the file's upload_warning, and the new run sets it again only if it skips content. A reindex keeps the warning unless it extracts figures or describes pictures for the file.
The reusable cache is the file's processed chunks, not a separate cache keyed by source, description model, and prompt. Changing the description model does not refresh existing chunks automatically.
Files uploaded while a reindex runs are written to both collections and are searchable as soon as they are ready. When a tenant reindex completes, the platform also schedules full processing for files marked upload_failed and for files without cached chunks. These follow-up runs can call the vision model. GET /v1/config/embedding/reindex/progress reports total, the tenant's ready files, which must all be in the new collection; moved, those already there, rebuilt by the reindex or written by their own upload during it, which the Portal shows as Rebuilt; and failed, those the reindex could not rebuild. While the reindex runs, total and moved are counted live, so an upload adds one and a delete removes one; when it ends they are stored. The follow-up runs are counted as reprocess_total, reprocessed, and reprocess_failed. Before a reindex starts, POST /v1/config/embedding/reindex/preview returns rebuild, the ready files not yet in the collection the reindex fills, and reprocess, the failed files its completion processes again. See document processing and reindexing. A reindex never regenerates an existing description.
Delete
Delete removes the file from the library lifecycle and triggers cleanup behavior for the indexed content.
From a user perspective, the important point is that delete is not only a UI metadata operation. It also participates in the broader storage and retrieval cleanup path.
Operational Realities
The current implementation has a few practical behaviors that matter to operators and advanced users:
- indexing is asynchronous
- upload success does not guarantee retrieval success
- ingestion needs
Temporal, object storage, embedding inference, andMilvus; document parsing also needsDocling, while standalone images need the configured vision model - use
POST /v1/files/upload-streamfor library uploads; placing an object directly in storage does not start RAG indexing - long conversions and temporary queue backlogs are tolerated, but bounded timeouts still mark stalled work as failed, at the stage where it stopped
- ingestion, file metadata, folder expansion, and vector operations carry the active tenant context, and new collections include the tenant identity in their generated name
- a failed reprocess can temporarily leave a file without active vectors until indexing succeeds again
For day-to-day portal use, the most important takeaway is simple: wait for Ready for search before expecting grounded chat answers from a newly uploaded document.
Why This Matters In AI
This capability is one of the main reasons AI can support grounded assistants and enterprise document workflows.
Without it, chat would only have direct model generation.
With it, the platform can:
- ground answers in uploaded content
- scope retrieval to specific files or folders
- connect document management with conversational AI
- support more governed and inspectable knowledge workflows
Related Pages
Reuse knowledge in an Agent
Configure an Agent with one file or folder in the Portal to reuse its retrieval scope. The caller still needs access to the source. Live Voicebot does not perform file retrieval.