Skip to content
SentientOne

Knowledge and RAG

Ground your product's AI in your docs, with the sources to prove it

Add PDFs, crawl a website, write snippets and exact Q&A. SentientOne chunks and embeds them, searches by meaning and by keyword on every question, reranks the results and tells your app which sources the answer used.

Start freeHow retrieval works

200 free credits when you sign up. No card needed.

At a glance

Four kinds of source, one search that uses them all

Sources

Four kinds, added on the agent's Knowledge tab. Each agent has its own knowledge; it switches on as soon as anything is indexed.

Knowledge sources
SourceLimits
DocumentsPDF files up to 50 MB each in the app; PDF and plain-text files through the API. Image-only PDFs have no text and are refused
WebsitesA start address, max pages (1 to 1,000, default 20) and depth (1 to 10, default 1), include and exclude paths. Each crawl stops at 200 pages. Only the main content of each page is kept
SnippetsShort facts, rules or definitions, 3 to 300 characters, stored whole
Q&AA question (3 to 200 characters) and its exact answer (3 to 500)

Ingestion

What happens between upload and the first answer that uses it.

  • Text is extracted from the PDF or the crawled page (crawled pages arrive as markdown; very short and duplicate pages are skipped).
  • It is split into 400-word chunks that overlap by 40 words, so a sentence on a boundary is never lost.
  • Each chunk is embedded with OpenAI text-embedding-3-small (1536 dimensions), in batches.
  • Credits are charged as pages are indexed and refunded for any page that fails.
  • Changing an agent's knowledge clears its cached answers and search results, so the next question sees the new content.
PDFUp to 50 MB
WebsiteUp to 200 pages
SnippetA fact or rule
Q&AExact answers
Chunk and embed400-word chunks, 40 overlap, OpenAI text-embedding-3-small (1536 dims)
Vector searchpgvector, HNSW index
Keyword searchPostgres full-text
FuseReciprocal rank fusion (k = 60), 24 candidates
RerankCohere rerank-v3.5
AnswerTop 5 passages to the model, sources streamed to you

The index

Everything lives in Postgres, in one table of chunks, so a single query can do both kinds of search.

Index details
Tablerag_chunks, one row per chunk, each tagged with its agent
Vectorspgvector, 1536 dimensions, HNSW index with cosine distance
Full textA stored tsvector (English) with a GIN index
AccessRow-level security on, no public access; only SentientOne's servers query it

Retrieval

Pure vector search is good at paraphrase and bad at exact strings: SKUs, error codes, names, clause numbers. Keyword search is the opposite. SentientOne runs both and lets a reranker settle it.

  1. 01Two searches

    The question is embedded and matched with pgvector, and also run as a Postgres full-text query over the GIN index. Each side returns its best matches for the agent.

  2. 02Fusion

    The two ranked lists are merged with reciprocal rank fusion (k = 60, equal weights), keeping the best 24 candidates.

  3. 03Rerank

    Cohere rerank-v3.5 scores each candidate against the question. If it doesn't answer within 1.5 seconds the fused order is used.

  4. 04Context

    The top 5 passages go to the model when the best one scores at least 0.3 (0.5 when the agent also has MCP tools, so it looks things up rather than answer from a weak match).

  5. 05Grounding

    A fixed instruction tells the model to answer from these passages and not to make things up. The agent can also search again mid-answer with its built-in search_documents tool.

Sources in the stream

Before the answer starts, the stream sends a sources event listing the passages the model was given: title, URL or file name, type, score and a snippet. Show them as citations, or log them.

Server-sent event
data: {"type": "sources", "sources": [  {"index": 1, "id": "…", "title": "Returns and refunds", "url": "https://example.com/returns",   "filename": null, "source_type": "web", "score": 0.84, "snippet": "Unworn items can be returned within…"},  {"index": 2, "id": "…", "title": "Warranty terms", "url": null,   "filename": "warranty-2026.pdf", "source_type": "document", "score": 0.71, "snippet": "…"}]}

Ingest by API

Everything the Knowledge tab does is also an endpoint, with your secret key and the agent's id: documents, snippets, Q&A, websites, and deleting a source.

Ingest endpoints
# A PDF or a plain-text file (up to 50 MB)curl https://api.sentientone.ai/v1/ingest/document \  -H "X-Api-Key: $SENTIENTONE_API_KEY" \  -H "X-Agent-Id: $AGENT_ID" \  -F "file=@warranty-2026.pdf"# A website (the API crawls at most 200 pages per crawl)curl https://api.sentientone.ai/v1/ingest/website \  -H "Content-Type: application/json" \  -H "X-Api-Key: $SENTIENTONE_API_KEY" \  -H "X-Agent-Id: $AGENT_ID" \  -d '{"url": "https://help.example.com", "max_pages": 100, "max_depth": 3, "include_paths": ["/docs"]}'

Errors use the same envelope as chat: FILE_TOO_LARGE (413), UNSUPPORTED_MEDIA_TYPE (415), EXTRACTION_FAILED (422), OUT_OF_CREDITS (402) and others.

Caching

Retrieval is the slowest part of a grounded reply that you control, so its pieces are cached.

Knowledge caches
Query embeddings7 days: the same question isn't embedded twice
Search results5 minutes per agent, dropped when its knowledge changes
AnswersThe first question of a new conversation, for agents without live tools, is answered from a 15-minute cache. Any knowledge or agent change makes old answers unreachable

Costs

Knowledge costs in credits
A PDF page5 credits, once, when it's added (a text file counts as one page)
A crawled website page5 credits per indexed page, per crawl. A crawl needs at least 5 credits to start and stops adding pages when the balance runs out
A snippet or Q&A pair1 credit
Searching itIncluded in the reply's 1 credit
Deleting itFree, and not refunded

Help centre

Knowledge in the help centre

Step-by-step guides and reference, taken from the product itself. Browse all help

FAQ

Questions about knowledge

Something else? The help centre goes deeper.

Which file types can I add?
In the app, PDFs up to 50 MB. Through the ingest API, PDFs and plain-text formats too: .txt, .md, .csv, .json, .html and similar. Scanned PDFs that are only images have no text to index and are refused.
How do I keep a website up to date?
Re-crawl it from the Websites list. The old pages and their vectors are replaced by the new crawl, which is charged again per page.
Is my knowledge shared between agents?
No. Every chunk belongs to one agent and both searches filter by it, so one agent never sees another's content.
What if the reranker is slow or down?
The fused order is used instead, after 1.5 seconds at most, so a slow reranker never stalls a reply.

Ground your first agent on your own docs

Sign up and you get 200 free credits, no card needed. Create an agent, add your knowledge, try it in the Playground, then call it from your code or put it on your website.

Start freeFollow the quickstart