> For the complete documentation index, see [llms.txt](https://docs.contextual.io/documentation-and-resources/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.contextual.io/documentation-and-resources/tenants/monitoring/metrics.md). # Metrics Metrics includes multi-panel dashboards at the tenant, individual agent, Object Type, and record levels. ## Where to Find It * Open **Monitoring > Metrics** for tenant-wide dashboards. * Open an individual agent and select the **Metrics** tab for detailed per-agent dashboards. * Open an Object Type whose detail page shows a **Records** tab and select **Metrics** for storage, size, and change activity for that type. This includes Flow, Agent, and Connection types. * Open a native-object record and select **Object Metrics**, next to **Definition**, for its current state and audit activity. ## Dashboard Coverage ### Tenant Metrics (Monitoring > Metrics) The tenant metrics view includes tabs for: * **Event** * **HTTP** * **Cron** * **Internal API** * **AI Gateway** * **Record Level Attachments (RLA)** * **Native Object** The RLA and Native Object tabs report on your solution data, including records, attachments, and change history. The other tabs report runtime and system metrics. For Event/HTTP/Cron tabs, each row can be expanded to show per-instance CPU and memory charts. ### Agent Metrics (Agent > Metrics) * **Event agents** include Event Metrics and System Metrics. * **HTTP agents** include HTTP Metrics and System Metrics. * **Cron agents** show System Metrics. An agent's **Metrics** tab reports runtime activity, while its **Object Metrics** tab reports the agent configuration record's state and audit history. Its **AI Gateway** tab reports provider usage for AI requests made by that agent. ## Time Range, Rollup, and Auto Refresh * The tenant runtime tabs default to **Last 24 Hours**, **1m rollup**, and **max aggregation**. * Rollup options are constrained by selected range to control point density. * Rollup defaults increase as ranges widen (for example: 24h -> 1m, 3d -> 5m, 7d -> 15m). * The RLA and Native Object tabs do not show the **Agg** control. Each panel defines its own calculation, shown in its title, card labels, and table columns. For example, count panels use counts, size panels total bytes, and summary cards identify averages and maximums. * The solution data tabs add **Last 3 Months**, **Last 6 Months**, **Last 12 Months**, and **All Time**. Switching from a solution data tab to a tab that does not support these ranges resets an extended range to **Last 30 Days**. * **All Time** starts at the earliest solution data captured for your tenant or object type. It does not mean that data exists for the tenant's entire lifetime. * Time range, rollup, aggregation, and selected tab are saved in browser local storage. * Time ranges also sync to URL parameters for shareable links (`last`, `from`, `to`). * Auto refresh is off by default. To enable it, click the circular-arrow control beside **Last Refreshed**. The dashboard refreshes immediately and then every 60 seconds for preset ranges. Click the control again to pause auto refresh. **All Time** and custom date ranges turn auto refresh off and disable the control. ## Linked Charts and Presentation Behavior * Charts in the same dashboard are linked for synchronized hover and zoom windows. * Zooming any chart pauses auto refresh and shows a pause state in the refresh control. * **Restore** returns all linked charts to the full selected range. * **Use Zoom Window** converts the current zoom selection into a custom time range. * Full screen mode hides navigation and header chrome for a focused dashboard view. * Use **Expand** on a chart or table to view that panel in full screen. Select **Close** or press **Escape** to exit. * Solution data line charts have a **Period | Cumulative** toggle. **Period** shows the value within each rollup bucket; **Cumulative** shows a running total across the selected range. * Browser back exits full screen first before navigating away. * Layout is responsive across breakpoints (wide multi-column down to single-column). ## Event Agent Dashboard Metrics Event agents include Pulsar-driven panels: * **Backlog Count** * **Storage Backlog Size (KB)** * **Messages Received (Per Sec)** * **Data Received (KB/Sec)** These metrics help correlate backlog growth with message volume and payload throughput. ## System Metrics (All Agent Types) System metrics include: * **Terminated Instances** (including OOMKilled and other stop reasons) * **Instance Count** over time * **Network I/O (KB and KB/Sec)** by receive/transmit direction * **CPU Usage (%)** and **CPU Usage (mCore)** with request/limit references * **Memory Usage (%)** and **Memory Usage (GB)** with Usage, RSS, Working Set, and request/limit references Instance Count overlays termination/watchdog markers. Hovering over markers shows stop reason, exit code, restart count, pod instance, image, start/termination timestamps, and runtime. ## HTTP and Internal API Dashboards * HTTP dashboards include endpoint counts, latency, and status-code trends, with matching aggregate tables. * Internal API dashboards include request volume, latency slices, rate-limit delays, and rate-limit errors by service/user/method/path. ## Interpreting Rollups and Gaps * Larger rollups smooth short spikes because each point represents an aggregated bucket. * A chart can look lower than max summary values when a high spike is averaged within a bucket. * Flat or inactive series can show sparse points or gaps when no new source metrics are emitted. ## AI Gateway Dashboard The AI Gateway tab provides visibility into LLM provider usage, cost, latency, and reliability across all AI routes in the tenant. It adds a **Last 90 Days** preset. Event, HTTP, Cron, and Internal API presets extend to **Last 30 Days**; RLA and Native Object also offer longer ranges and **All Time**. Switching tabs resets an unsupported preset to **Last 30 Days**. ### Summary Cards Six cards at the top of the dashboard display at-a-glance metrics for the selected time period. Each card shows a trend indicator comparing the current period to the prior equivalent window. * **AI Calls** — Total number of provider API calls made. Each call corresponds to a single request sent to an LLM provider endpoint. * **Tokens Processed** — Combined count of input and output tokens consumed across all calls. * **Estimated Cost** — Total spend for the period. An info icon (ⓘ) on the card opens a breakdown of cost by token category: Input (uncached), Cache Read, Cache Write, Text Output, and Reasoning. The tooltip also shows when model rates were last synced and what percentage of calls have pricing data. * A **warning icon** replaces the info icon when the cost estimate may be unreliable. This occurs in two cases: * **Stale pricing data** — model rates have not been synced in the last 24 hours. * **Low coverage** — fewer than 90% of calls in the selected time range have pricing data available. This typically indicates a recently added model whose rates have not yet been configured. * **Avg Latency** — Mean end-to-end response time across all calls. * **Error Rate** — Percentage of calls that resulted in an error. * **Cache Hit Rate** — Percentage of calls served from the provider's prompt cache. ### Time Series Charts Six charts are displayed in a two-column grid below the summary cards. All charts share synchronized hover and zoom interactions. * **Request Volume & Failures** — Call volume shown as bars with a failure count overlay. Use this to spot spikes in traffic or error bursts. * **Token Usage** — Stacked area chart broken down by token type: Input, Cache Read, Cache Write, Output, and Reasoning. A total token line is overlaid for quick trend comparison. * **Latency Trend** — P50 (median) and P95 latency over time. The gap between the two lines indicates how variable response times are — a widening gap suggests intermittent slowdowns even when the median looks healthy. * **Cost Breakdown** — Stacked bars by token category (Input, Cache Read, Cache Write, Output, Reasoning) with a total cost trend line overlaid. * **Route Health** — Horizontal stacked bar chart showing request outcomes per route. See [Route Health Chart](#route-health-chart) below. * **Finish Reasons** — Donut chart showing how requests terminated: `stop`, `tool-calls`, `length`, `content_filter`, or `error`. The center displays total request count. ### Breakdown Table A sortable table below the charts groups metrics by a user-selected dimension. Use the dropdown at the top-right of the table to switch grouping. **Group by options:** | Option | Description | | ---------------- | ------------------------------------------------------- | | Provider / Model | Breaks down metrics by LLM provider and model (default) | | Route | Groups by configured AI route | | Agent | Groups by the calling agent | | Node | Groups by workflow node | | Finish Reason | Groups by how the request terminated | **Columns:** Calls, Input Tokens, Output Tokens, Cache Hit, Avg Latency, P95 Latency, Errors, Error Rate, Est. Cost. All columns are sortable. Rows with errors show a red drill-down icon next to the error count (not available when grouped by Provider / Model). Click it to open the [Error Detail dialog](#error-detail-dialog). ### Tool Calls Table A separate table at the bottom of the page shows metrics for tool invocations made during agent execution. Columns: Tool Name, Calls, Avg Latency, P95 Latency, Errors, Error Rate. *** ### Route Health Chart The Route Health chart gives a per-route breakdown of how requests resolved across the selected time period. Each route appears as a horizontal stacked bar with three segments: * **Succeeded** (green) — Requests that completed successfully on the first attempt, with no fallback needed. * **Recovered** (orange) — Requests that initially failed on the primary model but succeeded after falling back to one or more alternative models. These requests produced a successful response but required extra attempts to do so. * **Failed** (red) — Requests where all attempts were exhausted — including any configured fallbacks — and no successful response was returned. Hovering over a bar segment shows a tooltip with the exact count and percentage for each status, the total request and call counts for the route, the primary model, any configured fallback models, and the total wasted cost if applicable. Clicking any bar segment opens the **Fallback Paths dialog** for that route. #### Fallback Paths Dialog The Fallback Paths dialog shows how requests moved through the model chain for a given route, aggregated into distinct path sequences. **Summary line** at the top shows the number of requests recovered by fallback, the number that ultimately failed, and the total wasted cost. **Fallback paths table** — Each row represents a unique sequence of model attempts: | Column | Description | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Path | The sequence of models attempted, shown as chips connected by arrows (→). Intermediate models are outlined; the final model is filled green if the path succeeded for any request, or red if all requests on that path failed. | | Requests | Number of requests that followed this exact path. | | Outcome | Count of succeeded requests (green checkmark) and failed requests (red error icon) for this path. | | Wasted Cost | Cost incurred on attempts that did not produce a successful response, shown in red. | | Total Cost | Total cost of all attempts across all requests on this path. | Rows where the primary model failed with no fallback attempted appear as a separate entry with the primary model chip in red and the label *(no fallback attempted)*. *** ### Error Detail Dialog The Error Detail dialog opens from the drill-down icon on any row in the breakdown table (except when grouped by Provider / Model). It provides a detailed view of errors for that grouping within the current time range and filters. **Summary bar** at the top shows three stats: total **Errors**, **Error Rate**, and **Est. Cost** for the selected group. #### Failure Breakdown Table Shows which provider/model combinations generated errors: | Column | Description | | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------- | | Provider | LLM provider name | | Model | Model identifier | | Calls | Total attempts made to this model | | Failed | Number of failed attempts (shown in red) | | Failure Rate | Percentage of calls to this model that failed (shown in red) | | Wasted Cost | Cost incurred on failed attempts | | Top Errors | Up to three error message chips showing the most common error messages for this model. Messages longer than 60 characters are truncated. | #### Recent Fallback Chains Below the table, the last 10 fallback chain sequences are shown in full detail. Each chain represents a single end-user request and the sequence of model attempts made to fulfill it. **Chain header:** Request ID (first 12 characters), attempt count, total duration, wasted cost (if any), and an **exhausted** badge if all attempts failed. **Each attempt shows:** * Attempt number * Success or failure icon * Provider and model (e.g. `openai/gpt-4o`) * Finish reason as a colored chip: red for errors, purple for `tool-calls`, green for normal completions * Error message, if the attempt failed * Duration and cost for that attempt ## Solution Data Dashboards Both tabs use one shared set of **Time Range**, **Rollup**, **Auto Refresh**, and full-screen controls at the top of the Metrics page. The selected time range and rollup apply to every panel on the active tab. The **Agg** control is not shown because each panel defines its own calculation. ### Record Level Attachments (RLA) The RLA tab covers attachments across the tenant. It combines composition charts, trends, summary cards, distributions, and tables so you can compare attachment volume and storage use. | Panel | What It Shows | | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | | Attachment Count by Content Type / Attachment Count by Content Type Over Time | The number of attachments for each content type and how those counts change over time. | | Attachment Size by Content Type / Attachment Size by Content Type Over Time | Total attachment size for each content type and how that storage use changes over time. | | Attachment Count by Type ID / Attachment Count by Type ID Over Time | Attachment counts by Object Type and their trends. | | Attachment Size by Type ID / Attachment Size by Type ID Over Time | Attachment storage use by Object Type and its trends. | | Attachment Count / Attachment Counts Over Time | Total attachments, average attachments per instance, maximum attachments per instance, and the count trend. | | Attachment Size / Attachment Size Over Time | Total attachment size, average size per instance, maximum size per instance, and the size trend. | | Attachment Count Distribution (Per Instance) | How attachment counts are distributed across instances. | | Attachment Size Distribution | How individual attachment sizes are distributed. | | Attachment Statistics | Content Type, File Count, % of Files, Total Size, % of Size, Avg Size, Max Size, and Min Size. | | Most Attachments by Count | Object Type, Instance ID, File Count, Avg Size, Max Size, and Total Size for the instances with the most attachments. | | Largest Attachments by Size | Object Type, Instance ID, Filename, Size, and Last Updated for the largest attachments. | In **Most Attachments by Count** and **Largest Attachments by Size**, clicking an Instance ID opens the record's **Attachments** tab. ### Native Object The Native Object tab covers all Native Object Types in the tenant. It combines record size and count metrics with operation and change-history metrics. | Panel | What It Shows | | --------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | Counts by Type / Counts by Type Over Time | Stored record counts by Object Type and their trends. | | Size by Type / Size by Type Over Time | Storage use by Object Type and its trends. | | Operations / Operations Over Time | The distribution and trend of record operations. | | Object Changes by History / Object Changes by History Over Time | Change-history activity by Object Type and its trend. | | Object Type Stats | Object Type, Record Count, % of Records, Total Bytes, % of Bytes, Avg Bytes, Max Bytes, and Min Bytes. | | Most Changed Objects | Object Type, Object ID, Operation, Change Count, First Change, Last Change, and Active Days for frequently changed records. | | Largest Objects by Size | Object Type, Object ID, Size, and Last Updated for the largest records. | Click an Object Type in any table to open that Object Type's **Metrics** tab with the current time range. In **Most Changed Objects** and **Largest Objects by Size**, click an Object ID to open that record's **Object Metrics** tab. ## Object Type Metrics (Object Type > Metrics) ### Where to Find It Open an Object Type and select the **Metrics** tab. This dashboard reports storage, size, and change activity for records of that Object Type only. ### Availability The **Metrics** tab is available on any Object Type whose detail page shows a **Records** tab, including Flow, Agent, and Connection types. Operation, user, path, and change-activity panels use audit history, as do **Avg Changes / Record** and **Max Changes / Record**. Without audit history for the selected range, those panels show no activity. Count and size panels depend on captured record data, which may be unavailable for some system records. ### Controls The Object Type dashboard has its own **Time Range**, **Rollup**, **Auto Refresh**, and full-screen controls. These controls are independent of the controls on **Monitoring > Metrics**. * The default time range is **Last 30 Days**. The rollup uses the recommended default for the selected range. * Time range and rollup selections are saved in the browser for Object Type metrics. * Auto refresh is off by default. **All Time** and custom ranges turn it off and disable the control. Zooming pauses it, and selecting **Use Zoom Window** changes the dashboard to a custom time range. * The dashboard supports the solution data ranges through **All Time**. ### Panels | Panel | What It Shows | | -------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Counts / Counts Over Time | **Records** reports the indexed record count for the selected range. **Avg Changes / Record** and **Max Changes / Record** summarize audit-history activity. The chart has **Records** and **Changes** lines. | | Record Size / Size Over Time | Total, average, and maximum record size and the size trend. | | Operations / Operations Over Time | The distribution and trend of record operations. | | Changes by User / Changes by User Over Time | Change activity by author and its trend, using user emails when available. | | Most Updated Paths | Path, Operation, Updates, % of Updates, and Unique Records for frequently updated fields. | | Updates by Path / Updates by Path Over Time | The distribution and trend of updates by field path. | | Operations Summary / Operations vs Previous Period | Operation counts. When enough prior data exists, the table also shows Current, Previous, and Change values. | | Activity by Hour of Day | Change activity by local hour. For ranges longer than six months, the panel uses the latest six months of data. | | Record Size Distribution | How record sizes are distributed. | | Most Changed Records | Record ID, Changes, Latest Version, Last Changed By (email when available), and Last Changed for frequently changed records. | | Largest Records by Size | Record ID, Size, and Last Updated for the largest records. | Click a Record ID in **Most Changed Records** or **Largest Records by Size** to open that record's **Object Metrics** tab. ## Record Metrics (Record > Object Metrics) Open a native-object record and select **Object Metrics**, next to **Definition**. Record links in the Native Object and Object Type metrics tables also open this tab. A direct link to a record can select it with `?tab=metrics`. ### Controls and Summary Tiles The record dashboard has its own **Time Range**, **Rollup**, **Auto Refresh**, and full-screen controls. It defaults to **Last 30 Days**, reuses saved record-dashboard range and rollup selections, and supports the solution data ranges through **All Time**. Auto refresh starts off; **All Time** and custom ranges disable it. The chart and table controls described above also apply here. When current record data is available, the summary tiles show **Current Size**, **History Version**, **Last Editor**, **Last Updated**, and **Changes** in the selected range. The first four describe the latest captured state, independently of the selected activity range. The tile strip is hidden when no current record data is available. ### Activity Panels | Panel | What It Shows | | -------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | Operations / Operations Over Time | Record operations and their trend. | | Changes by User / Changes by User Over Time | Activity by author, using user emails when available, and its trend. | | Most Updated Paths | Path, Operation, Updates, and % of Updates for frequently updated fields. | | Updates by Path / Updates by Path Over Time | Updates by field path and their trend. | | Operations Summary / Operations vs Previous Period | Operation counts, with Current, Previous, and Change values when enough prior data exists for comparison. | | Activity by Hour of Day | Activity by local hour, using the latest six months for longer ranges. | These panels use audit history. When there is no activity in the selected range, they show an empty state: **No activity in this time range.** Audit trail may be disabled for the Object Type. ## Interpreting Solution Data * **Some Native Object panels can include multiple record revisions.** An updated record may contribute more than once within the selected range. This can affect the count and size panels and **Object Type Stats**, including the Object Type **Counts** panel's **Records** value. These values are not necessarily distinct-record totals. **Largest Objects by Size** and **Largest Records by Size** select the latest available version of each record within the range. * **Author labels use available tenant user data.** A matching user ID displays as an email. Long unresolved IDs are shortened, and a missing ID displays as **Unknown**. * **RLA panels handle versions according to the panel type.** Summary cards, pie charts, and tables select the latest available version of each attachment within the selected range and exclude attachments marked as soft-deleted. Time-series and distribution charts use matching versions from across the range, so an attachment can contribute more than once after updates. * **Tables summarize a limited set of top rows.** Statistics and “most” tables typically return around 20 top rows. A total in one of these tables summarizes the returned rows, not every matching record in the tenant. * **Hourly activity uses recent data.** The **Activity by Hour of Day** panel uses your local timezone and, for longer ranges, up to the latest six months of data. * **Trends need a complete previous period.** Summary cards and tables show period-over-period changes only when data covers the full previous window. Otherwise, they show the current values without a trend. * **History starts when capture starts.** Older data appears only after historical data has been loaded for your tenant. If a long range is empty or shorter than expected, data might not have been captured that far back. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.contextual.io/documentation-and-resources/tenants/monitoring/metrics.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.