> For the complete documentation index, see [llms.txt](https://docs.contextual.io/documentation-and-resources/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.contextual.io/documentation-and-resources/components-and-data/ai-search/concepts-and-configuration.md).

# Search Concepts and Configuration

A Search Profile defines which records from a source Object Type to index and how to turn their fields into searchable text. The named indexes for a Search Profile define reusable query behavior. This page explains the main configuration elements, the choices available for each, and how those choices affect indexed content and search results.

## Templates

A Search Profile's required body template selects and organizes the source record information that becomes searchable text. Contextual renders the body for each eligible record and splits it into chunks. An optional, separate header template adds context to each chunk before embedding.

Templates use [Handlebars](https://handlebarsjs.com/) syntax and can reference fields from the source Native Object record. Use `{{fieldName}}` to insert a field's value. For example, `{{title}}` inserts the record's `title` value.

{% hint style="warning" %}
AI Search does not extract text from files or read attachment contents, including text or Markdown attachments. Extract the text before indexing, save it in a field on the source record, and include that field in the body template. You can keep the original file as an attachment.
{% endhint %}

Include information that could help answer likely natural-language queries, and label values so their meaning is clear. Use conditional blocks for optional text:

```handlebars
Title: {{title}}
{{#if description}}Description: {{description}}{{/if}}
{{#if resolution}}Resolution: {{resolution}}{{/if}}
```

Some source records store another record's ID in a field such as `ownerId`. Including `{{ownerId}}` in a template adds the ID itself, not the related record's name or description. To make those details searchable, include a field that already holds them on the source record, or add and maintain such a field before indexing.

The rendered header is prepended to each chunk before embedding. A short title or category can give each chunk context. Headers consume part of the chunk size, so a long header leaves less room for the body.

**Generate from schema** creates a starting body template from renderable top-level fields. It does not establish which fields will produce good matches. Review the result for relevance and readability.

The console preview shows the rendered header and body for one sample record, or placeholders when no sample is available. Chunking happens during indexing; the preview does not show chunks or validate all source records.

## Chunking

A document is the text derived from a source record. A chunk is a portion of that text that can be embedded and retrieved. Long documents can produce multiple chunks that refer to the same source record.

These settings control how rendered text is divided into chunks:

| Setting             | Default             | Supported Values       |
| ------------------- | ------------------- | ---------------------- |
| Splitter            | `plain-text-window` | `plain-text-window`    |
| Size                | 1,600 characters    | 100–128,000 characters |
| Overlap             | 10%                 | 0–50%                  |
| Preserve boundaries | `true`              | `true` or `false`      |

To split text at paragraph, sentence, or word boundaries where possible, leave **Strict windows (not recommended)** unchecked in the dashboard (`preserveBoundaries: true`). See the [dashboard control map](https://docs.contextual.io/documentation-and-resources/components-and-data/ai-search/manage-search-profiles.md#find-settings-in-the-dashboard).

Size is measured in characters. Each chunk sent for embedding, including its rendered header and separator, must also fit the embedding model's input token limit. The indexer reserves space for the header and separator within the character budget before splitting the body.

Overlap repeats some body text between adjacent chunks to retain context around a split. With boundary preservation enabled, the indexer considers paragraph, sentence, and word boundaries. It can fall back to split within a sentence or word to stay within the chunk size limit when necessary. Chunk lengths and actual overlap can therefore differ from fixed windows.

Start with the defaults. Change chunking when retrieved passages omit needed context or combine unrelated topics. An empty rendered body produces no chunks.

## Embedding

An embedding represents text as a vector used for semantic comparison. Contextual embeds indexed text and the query using the active index's embedding configuration.

| Setting    | Meaning                                                                         | Default    | Supported Values                              |
| ---------- | ------------------------------------------------------------------------------- | ---------- | --------------------------------------------- |
| Model      | The trained AI system that converts text into vectors representing its meaning. | `voyage-4` | `voyage-4-lite`, `voyage-4`, `voyage-4-large` |
| Dimensions | The number of numeric values in each embedding vector.                          | 1,024      | 512, 1,024, 2,048                             |

A model's name (such as "lite" or "large") does not describe how well it will retrieve relevant content from your records. Before changing embedding models, compare search results for questions users are likely to ask.

Changing dimensions changes the vector configuration and requires rebuilding. The dashboard saves the change without a rebuild confirmation prompt. After saving, inspect the **Index** panel for progress. See [Change an Existing Profile](https://docs.contextual.io/documentation-and-resources/components-and-data/ai-search/manage-search-profiles.md#change-an-existing-profile) and [Keep Search Up to Date](https://docs.contextual.io/documentation-and-resources/components-and-data/ai-search/keep-search-up-to-date.md#repair-and-rebuild).

## Projections

Projections let a named Search Index narrow searches by structured values from source records, such as category or rating. During indexing, an active projection evaluates an expression against each record to produce a typed attribute. A named index can use that attribute in a static filter, and its value, when present, is returned with search evidence. A projection does not add text to the template; include the information in a template too if you want it to be searchable as text.

Each projection has a unique alias, an expression language, an expression, and a result type. Aliases start with a lowercase letter and contain lowercase letters, numbers, or underscores, up to 63 characters.

| Property | Supported Values                                                                                       |
| -------- | ------------------------------------------------------------------------------------------------------ |
| Language | `jsonpath`, `jsonata`                                                                                  |
| Type     | `string`, `number`, `integer`, `boolean`, `datetime`, `string[]`, `number[]`, `integer[]`              |
| Retired  | Optional `true` to stop writing and returning the projection while keeping its alias and type reserved |

For example, suppose a profile indexes equipment records for pumps, valves, and motors. This projection extracts each record's category:

```json
{
  "alias": "category",
  "language": "jsonpath",
  "expression": "$.category",
  "type": "string"
}
```

You could configure a named index called `pumps` with a static filter of `category = "Pumps"`. A search request with `indexId: "pumps"` then uses that filter to retrieve pump records. The profile still indexes the other categories, and searches that do not select `pumps` do not use its filter. The filter value is saved in the named index; the search request supplies the index ID, not a category value.

Preview the expression against real records, including missing fields and unexpected values. The indexer converts results to the declared type; absent or unconvertible values can be omitted. A projection is not a substitute for source-data validation.

### Projection Types and Stored Columns

The projection type specifies how the expression's result is converted. Projection values are stored in typed columns in the indexed data shared by a profile's named Search Indexes. Each column keeps its type once created. Filters and returned projection values use the projection alias, such as `category`; the corresponding stored column is named `data.category`.

| Projection Type        | Stored as in a New Column       | Accepts                                                                                        |
| ---------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------- |
| `string`, `string[]`   | text                            | Any value, converted to text                                                                   |
| `number`, `number[]`   | floating-point number (`float`) | Whole and fractional numbers                                                                   |
| `integer`, `integer[]` | integer (`int`)                 | Whole numbers only; a fraction, or a whole number too large to store exactly, skips the record |
| `boolean`              | boolean                         | `true`, `false`, and the strings `"true"` and `"false"`                                        |
| `datetime`             | datetime                        | Dates and timestamps, stored as ISO 8601                                                       |

Choose `number` unless every value is a whole number and you want an integer column. Neither type rounds a value to make it fit: an `integer` projection that produces `2.5` skips that record and reports it. Because a JSONata expression's result is known only when it runs, an `integer` projection over computed values, such as an average, can skip records even when the source fields are whole numbers.

Indexes created before explicit column types can hold columns whose type came from the first value written. A `number` projection whose column holds integers keeps working for whole values. A fraction is not rounded: the indexer skips that record and reports it (see [Skipped Records](https://docs.contextual.io/documentation-and-resources/components-and-data/ai-search/keep-search-up-to-date.md#skipped-records)). A `datetime` projection whose column holds text keeps working. The dashboard shows each column's stored type under **Stored columns**.

Some conversions happen before storage:

* An expression that returns `null`, or matches nothing, produces no value. The indexer omits the attribute rather than storing zero or an empty value.
* Values are converted to the declared projection type, so a numeric string such as `"12"` becomes `12` for a `number` projection. A value that cannot convert is omitted.
* A JSONata expression can return a different kind of value than the source field holds. For example, `$count(tags)` returns a number from an array field. Choose the type for what the expression returns. The source field's schema alone does not establish it.

The preview evaluates one sample record. It cannot show that every future value fits the stored column.

### Add, Rename, Remove, or Retire a Projection

Changing projections only, with source, template, chunking, and embedding unchanged, updates the existing index in place. A background repair reprocesses records and reuses their embeddings, so unchanged text needs no new embedding.

| Change                                                                 | Result                                                                                                                                                                                                                                     |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Add a projection                                                       | The new column is created with its declared type before any record is written.                                                                                                                                                             |
| Change an expression, or a type the stored column can hold             | The repair rewrites values in place.                                                                                                                                                                                                       |
| Rename a projection                                                    | The new alias is a new column. The old column becomes a removed column, described below.                                                                                                                                                   |
| Change to a type the stored column cannot hold, such as text to number | The save does not apply in place. Choose a compatible type or a new alias, or save with a rebuild into a fresh index.                                                                                                                      |
| Retire a projection                                                    | Writes, filters, and results stop using it right away. The projection stays in the profile, marked `"retired": true`, and appears in a collapsed **Retired** group. Stored values stay on each record until that record is next rewritten. |
| Re-enable a retired projection                                         | Writes, filters, and results use it again, and the repair rewrites its values. It must keep a type the stored column can hold.                                                                                                             |
| Remove a projection                                                    | The same as retiring, except the projection leaves the profile. Its stored column shows as removed.                                                                                                                                        |

Retiring or removing a projection never frees its column in the current index. The name and type stay reserved there, so re-enabling a retired projection, or reusing a removed alias, needs a compatible type. To change a type, use a new alias, or rebuild. A rebuild starts a fresh index that declares only the active projections; retired projections stay retired in the profile and get no column there.

## Indexes

A named Search Index is a query preset within a profile. Multiple presets can use the same indexed records with different retrieval settings.

| Setting                 | Named Index Default | Meaning                                                             |
| ----------------------- | ------------------- | ------------------------------------------------------------------- |
| Mode                    | `hybrid`            | Semantic plus keyword retrieval; `semantic` uses vector similarity. |
| Top K                   | 10                  | Retrieval candidate limit, from 1 to 100.                           |
| One result per document | `true`              | Group retrieval by source document.                                 |
| Static filters          | None                | Restrict candidates using projected fields.                         |
| Reranking               | Disabled            | Optionally reorder retrieved candidates.                            |

To use a named preset, include its ID in the `indexId` field of your [search query request](https://docs.contextual.io/documentation-and-resources/components-and-data/ai-search/query-and-use-results.md). If you omit `indexId`, the search uses semantic mode, Top K 10, no document grouping, no preset filters, and no reranking. You can override Top K by including `topK` in the request.

Top K sets the maximum number of retrieval candidates. A query can return fewer results. Filtering, grouping, and source-record availability affect the result count.

Named index settings apply when you run a search. Saving changes to a named index's retrieval settings does not trigger a rebuild or change stored embeddings. After you save, the next query that selects the named index uses the new values.

## Static Filters

Static filters restrict a named index's candidates using projection aliases. All configured static filters must match.

```json
{
  "kind": "static",
  "field": "category",
  "op": "eq",
  "value": "Pumps"
}
```

| Operators                | Value Shape and Purpose                                                 |
| ------------------------ | ----------------------------------------------------------------------- |
| `eq`, `not-eq`           | Compare a projected value with the supplied value.                      |
| `lt`, `lte`, `gt`, `gte` | Compare ordered values. Use a value compatible with the projected type. |
| `in`, `not-in`           | Supply an array of candidate values.                                    |
| `contains`               | Test membership in an array-valued field with a single value.           |
| `contains-any`           | Supply an array of values to match against an array-valued field.       |
| `glob`, `regex`          | Supply a string pattern; `regex` must be a valid regular expression.    |

Source eligibility and static filters apply at different stages:

* **Source eligibility** selects which source records belong in the profile.
* **Static filters** restrict searches that use a particular named index.

{% hint style="warning" %}
Static filters do not enforce access permissions. A query can omit `indexId` or select another named Search Index and bypass a particular index's filters. Use tenant and source read permissions to control access to records.
{% endhint %}

## Reranking

Reranking scores retrieved candidates again against the query and reorders them.

To use reranking, enable it on a named Search Index and select a rerank preset: a saved configuration that identifies the model used for reranking. Available presets depend on your tenant's configuration. Set **Top N** explicitly to limit how many of the highest-scoring candidates are retained after reranking. Top N must not exceed the configured Top K.

The default scope is `document`. Each document is reranked once, using the text of its highest-ranked retrieved hit rather than the full source record. The document's other retrieved hits follow it and share its rerank score. Because Top N counts documents in this scope, turning grouping off can return more hit rows than Top N.

With `chunk` scope, reranking scores each retrieved text chunk separately, including chunks from the same source record. Choose a scope that matches whether the reader needs distinct records or individual passages. Compare it with reranking disabled using the same queries.

{% hint style="info" %}
Reranking cannot recover a relevant record omitted from the original retrieval.
{% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.contextual.io/documentation-and-resources/components-and-data/ai-search/concepts-and-configuration.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.
