Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Search Concepts and Configuration

A Search Profile defines which records from a source Object Type to index and how to turn their fields into searchable text. The named indexes for a Search Profile define reusable query behavior. This page explains the main configuration elements, the choices available for each, and how those choices affect indexed content and search results.

Templates

A Search Profile's required body template selects and organizes the source record information that becomes searchable text. Contextual renders the body for each eligible record and splits it into chunks. An optional, separate header template adds context to each chunk before embedding.

Templates use Handlebars syntax and can reference fields from the source Native Object record. Use {{fieldName}} to insert a field's value. For example, {{title}} inserts the record's title value.

Include information that could help answer likely natural-language queries, and label values so their meaning is clear. Use conditional blocks for optional text:

Title: {{title}}
{{#if description}}Description: {{description}}{{/if}}
{{#if resolution}}Resolution: {{resolution}}{{/if}}

Some source records store another record's ID in a field such as ownerId. Including {{ownerId}} in a template adds the ID itself, not the related record's name or description. To make those details searchable, include a field that already holds them on the source record, or add and maintain such a field before indexing.

The rendered header is prepended to each chunk before embedding. A short title or category can give each chunk context. Headers consume part of the chunk size, so a long header leaves less room for the body.

Generate from schema creates a starting body template from renderable top-level fields. It does not establish which fields will produce good matches. Review the result for relevance and readability.

The console preview shows the rendered header and body for one sample record, or placeholders when no sample is available. Chunking happens during indexing; the preview does not show chunks or validate all source records.

Chunking

A document is the text derived from a source record. A chunk is a portion of that text that can be embedded and retrieved. Long documents can produce multiple chunks that refer to the same source record.

These settings control how rendered text is divided into chunks:

Setting
Default
Supported Values

Splitter

plain-text-window

plain-text-window

Size

1,600 characters

100–128,000 characters

Overlap

10%

0–50%

Preserve boundaries

true

true or false

To split text at paragraph, sentence, or word boundaries where possible, leave Strict windows (not recommended) unchecked in the dashboard (preserveBoundaries: true). See the dashboard control map.

Size is measured in characters. Each chunk sent for embedding, including its rendered header and separator, must also fit the embedding model's input token limit. The indexer reserves space for the header and separator within the character budget before splitting the body.

Overlap repeats some body text between adjacent chunks to retain context around a split. With boundary preservation enabled, the indexer considers paragraph, sentence, and word boundaries. It can fall back to split within a sentence or word to stay within the chunk size limit when necessary. Chunk lengths and actual overlap can therefore differ from fixed windows.

Start with the defaults. Change chunking when retrieved passages omit needed context or combine unrelated topics. An empty rendered body produces no chunks.

Embedding

An embedding represents text as a vector used for semantic comparison. Contextual embeds indexed text and the query using the active index's embedding configuration.

Setting
Meaning
Default
Supported Values

Model

The trained AI system that converts text into vectors representing its meaning.

voyage-4

voyage-4-lite, voyage-4, voyage-4-large

Dimensions

The number of numeric values in each embedding vector.

1,024

512, 1,024, 2,048

A model's name (such as "lite" or "large") does not describe how well it will retrieve relevant content from your records. Before changing embedding models, compare search results for questions users are likely to ask.

Changing dimensions changes the vector configuration and requires rebuilding. The dashboard saves the change without a rebuild confirmation prompt. After saving, inspect the Index panel for progress. See Change an Existing Profile and Keep Search Up to Date.

Projections

Projections let a named Search Index narrow searches by structured values from source records, such as category or rating. During indexing, an active projection evaluates an expression against each record to produce a typed attribute. A named index can use that attribute in a static filter, and its value, when present, is returned with search evidence. A projection does not add text to the template; include the information in a template too if you want it to be searchable as text.

Each projection has a unique alias, an expression language, an expression, and a result type. Aliases start with a lowercase letter and contain lowercase letters, numbers, or underscores, up to 63 characters.

Property
Supported Values

Language

jsonpath, jsonata

Type

string, number, integer, boolean, datetime, string[], number[], integer[]

Retired

Optional true to stop writing and returning the projection while keeping its alias and type reserved

For example, suppose a profile indexes equipment records for pumps, valves, and motors. This projection extracts each record's category:

You could configure a named index called pumps with a static filter of category = "Pumps". A search request with indexId: "pumps" then uses that filter to retrieve pump records. The profile still indexes the other categories, and searches that do not select pumps do not use its filter. The filter value is saved in the named index; the search request supplies the index ID, not a category value.

Preview the expression against real records, including missing fields and unexpected values. The indexer converts results to the declared type; absent or unconvertible values can be omitted. A projection is not a substitute for source-data validation.

Projection Types and Stored Columns

The projection type specifies how the expression's result is converted. Projection values are stored in typed columns in the indexed data shared by a profile's named Search Indexes. Each column keeps its type once created. Filters and returned projection values use the projection alias, such as category; the corresponding stored column is named data.category.

Projection Type
Stored as in a New Column
Accepts

string, string[]

text

Any value, converted to text

number, number[]

floating-point number (float)

Whole and fractional numbers

integer, integer[]

integer (int)

Whole numbers only; a fraction, or a whole number too large to store exactly, skips the record

boolean

boolean

true, false, and the strings "true" and "false"

datetime

datetime

Dates and timestamps, stored as ISO 8601

Choose number unless every value is a whole number and you want an integer column. Neither type rounds a value to make it fit: an integer projection that produces 2.5 skips that record and reports it. Because a JSONata expression's result is known only when it runs, an integer projection over computed values, such as an average, can skip records even when the source fields are whole numbers.

Indexes created before explicit column types can hold columns whose type came from the first value written. A number projection whose column holds integers keeps working for whole values. A fraction is not rounded: the indexer skips that record and reports it (see Skipped Records). A datetime projection whose column holds text keeps working. The dashboard shows each column's stored type under Stored columns.

Some conversions happen before storage:

  • An expression that returns null, or matches nothing, produces no value. The indexer omits the attribute rather than storing zero or an empty value.

  • Values are converted to the declared projection type, so a numeric string such as "12" becomes 12 for a number projection. A value that cannot convert is omitted.

  • A JSONata expression can return a different kind of value than the source field holds. For example, $count(tags) returns a number from an array field. Choose the type for what the expression returns. The source field's schema alone does not establish it.

The preview evaluates one sample record. It cannot show that every future value fits the stored column.

Add, Rename, Remove, or Retire a Projection

Changing projections only, with source, template, chunking, and embedding unchanged, updates the existing index in place. A background repair reprocesses records and reuses their embeddings, so unchanged text needs no new embedding.

Change
Result

Add a projection

The new column is created with its declared type before any record is written.

Change an expression, or a type the stored column can hold

The repair rewrites values in place.

Rename a projection

The new alias is a new column. The old column becomes a removed column, described below.

Change to a type the stored column cannot hold, such as text to number

The save does not apply in place. Choose a compatible type or a new alias, or save with a rebuild into a fresh index.

Retire a projection

Writes, filters, and results stop using it right away. The projection stays in the profile, marked "retired": true, and appears in a collapsed Retired group. Stored values stay on each record until that record is next rewritten.

Re-enable a retired projection

Writes, filters, and results use it again, and the repair rewrites its values. It must keep a type the stored column can hold.

Remove a projection

The same as retiring, except the projection leaves the profile. Its stored column shows as removed.

Retiring or removing a projection never frees its column in the current index. The name and type stay reserved there, so re-enabling a retired projection, or reusing a removed alias, needs a compatible type. To change a type, use a new alias, or rebuild. A rebuild starts a fresh index that declares only the active projections; retired projections stay retired in the profile and get no column there.

Indexes

A named Search Index is a query preset within a profile. Multiple presets can use the same indexed records with different retrieval settings.

Setting
Named Index Default
Meaning

Mode

hybrid

Semantic plus keyword retrieval; semantic uses vector similarity.

Top K

10

Retrieval candidate limit, from 1 to 100.

One result per document

true

Group retrieval by source document.

Static filters

None

Restrict candidates using projected fields.

Reranking

Disabled

Optionally reorder retrieved candidates.

To use a named preset, include its ID in the indexId field of your search query request. If you omit indexId, the search uses semantic mode, Top K 10, no document grouping, no preset filters, and no reranking. You can override Top K by including topK in the request.

Top K sets the maximum number of retrieval candidates. A query can return fewer results. Filtering, grouping, and source-record availability affect the result count.

Named index settings apply when you run a search. Saving changes to a named index's retrieval settings does not trigger a rebuild or change stored embeddings. After you save, the next query that selects the named index uses the new values.

Static Filters

Static filters restrict a named index's candidates using projection aliases. All configured static filters must match.

Operators
Value Shape and Purpose

eq, not-eq

Compare a projected value with the supplied value.

lt, lte, gt, gte

Compare ordered values. Use a value compatible with the projected type.

in, not-in

Supply an array of candidate values.

contains

Test membership in an array-valued field with a single value.

contains-any

Supply an array of values to match against an array-valued field.

glob, regex

Supply a string pattern; regex must be a valid regular expression.

Source eligibility and static filters apply at different stages:

  • Source eligibility selects which source records belong in the profile.

  • Static filters restrict searches that use a particular named index.

Reranking

Reranking scores retrieved candidates again against the query and reorders them.

To use reranking, enable it on a named Search Index and select a rerank preset: a saved configuration that identifies the model used for reranking. Available presets depend on your tenant's configuration. Set Top N explicitly to limit how many of the highest-scoring candidates are retained after reranking. Top N must not exceed the configured Top K.

The default scope is document. Each document is reranked once, using the text of its highest-ranked retrieved hit rather than the full source record. The document's other retrieved hits follow it and share its rerank score. Because Top N counts documents in this scope, turning grouping off can return more hit rows than Top N.

With chunk scope, reranking scores each retrieved text chunk separately, including chunks from the same source record. Choose a scope that matches whether the reader needs distinct records or individual passages. Compare it with reranking disabled using the same queries.

Reranking cannot recover a relevant record omitted from the original retrieval.

Last updated

Was this helpful?