AI Generate

Overview

The AI Generate node simplifies AI implementation in your Flows by providing a universal interface to multiple AI service providers. When paired with AI Routes and AI Connections, it handles the complexity of provider-specific APIs, authentication, and failover scenarios automatically.

What It Does

• Calls AI providers using a single, standardized input format regardless of provider

• Returns responses in a consistent, structured format with rich details

• Automatically handles provider failover through AI Routes

• Eliminates the need to refactor flows when switching providers or models

Benefits

• Provider-agnostic - Switch providers or models without refactoring your flows

• Automatic failover - If one provider fails, AI Routes automatically try the next configured provider

• Consistent interface - Same input/output format across all providers

• Rich response details - Access token usage, model information, and execution details

• Production-ready - Works seamlessly with versioned AI Routes packaged in Services

Prerequisites

Before using the AI Generate node, you'll need:

1. AI Connection(s) - AI Connections to one or more AI providers (OpenAI, Anthropic, Azure OpenAI, Google AI, Vertex AI, Vertex AI Anthropic)

2. AI Route - An AI Route that references your AI Connection(s) and specifies which model(s) to use

Basic Setup

1. Drag the AI Generate node from the AI Gateway category in the Flow Editor palette onto your canvas

2. Double-click the node to open the configuration panel

3. Select an AI Route:

   • Choose from the dropdown of AI Routes created in your tenant

   • Or dynamically specify using a msg property, environment variable, or static string

4. Specify the Input - Designate the msg object property that contains your input data (prompt, temperature, file data, expected schema, etc.)

5. Optional: Configure tools (selected tool nodes, max steps, dynamic tool filtering), timeout, Output, and Response property names

Tool Calling with AI Generate

AI Generate can invoke tools defined in the same flow. Tools let the model call specific functions (API calls, lookups, transformations) and then continue generation using the tool output.

Quick Setup

1. From the Flow Editor left palette, add Tool and Tool Response nodes from the AI Gateway category.

2. Configure each Tool node with a unique Tool ID and an Input Schema.

3. Build your tool logic downstream of each Tool node, then end the path with Tool Response to return output.

4. In AI Generate, open Tools and choose Selected Tool Nodes, then select the Tool nodes you want available to the model.

Configuration Details

• Tools -> Selected Tool Nodes - Pick the Tool nodes to expose to the model. Tool IDs must be unique.

• Max steps - Caps the number of tool calls in a single request, so a tool chain does not run indefinitely.

• Enable dynamic tool filtering - Provide a msg property that contains a list of Tool IDs and choose Include only or Exclude to control which tools are available per request.

For detailed Tool configuration, see AI Tool and AI Tool Response.

Execution Behavior

When a provider returns multiple tool calls in one model turn, AI Generate executes them in parallel by default.

For providers that support overrides, you can pass provider-specific options in the request (for example, providerOptions.openai.parallelToolCalls: false) to force one tool call per turn.

• Tool call events are dispatched in the same order returned by the provider, without waiting for earlier tool calls to finish.

• AI Generate waits until all pending tool calls are resolved before replying to the model for the next turn.

Input Format

The AI Generate node uses one universal, standardized input format regardless of which AI provider your AI Route calls. This standardization means changing providers or models requires little to no refactoring of your surrounding flow logic.

Your input msg object might contain:

• One of the following is required:

  • prompt - simple text prompt (use this OR messages array)

  • messages - array of messages (use this OR prompt), which are required in order to pass base64 file content

• schema - JSON Schema for structured output. Providing schema selects structured object generation; if output is omitted, it defaults to "object". When output is "object" or "array", schema is required.

• system - (optional) system instructions to guide the model's behavior

• temperature - (optional) Control randomness in responses

• maxTokens - maximum token to generate

• topP - Nucleus sampling parameter

• topK - Top-K sampling parameter

• frequencyPenalty - Penalty for repeated tokens

• presencePenalty - Penalty for repeated topics

• stopSequences - Array of sequences that stop generation

• seed - Random seed for deterministic output

• Other provider parameters as needed

Retry Behavior

The following retry and failover behavior applies to both text and object generation:

• Malformed responses, internal server errors (500), too many requests (429), and similar transient failures are treated as retryable errors.

• Unknown errors, bad requests (400), and auth issues (401/403) shortcut to the next provider without same-provider retries.

• Retries happen on the same provider first, based on the maxRetries setting on your AI Route.

• If retries are exhausted and fallback providers are configured, the AI Route cascades to the next provider.

• All retries use exponential backoff with random jitter, with no delay when falling back to a new provider.

• Failed attempts are still captured in providerErrors and steps in the final output.

Object Generation Special Case

In object generation mode, the model response must be valid JSON and match your schema. Invalid JSON (or output that fails schema validation) is treated as a retryable generation error.

Input Examples - Text Generation

Example: Simple Prompt

msg.payload = {
  prompt: "What is 2+2? Answer in one sentence.",
  temperature: 0.3,
  maxTokens: 50
};

Example: Messages Array

msg.payload = {
  messages: [
    {
      role: "user",
      content: "Explain photosynthesis in two sentences."
    }
  ],
  temperature: 0.5,
  maxTokens: 100
};

Example: With System Message

msg.payload = {
  system: "You are a concise math tutor. Always show your work.",
  prompt: "What is 15 * 23?",
  temperature: 0.3,
  maxTokens: 100
};

Example: Multi-Turn Conversation

msg.payload = {
  messages: [
    {
      role: "user",
      content: "What's the capital of France?"
    },
    {
      role: "assistant",
      content: "The capital of France is Paris."
    },
    {
      role: "user",
      content: "What's its population?"
    }
  ],
  temperature: 0.3,
  maxTokens: 100
};

Input Examples - Object Generation

Schema Configuration:

• output - Type of output: "object" (default) or "array". If omitted while schema is present, output defaults to "object".

• mode - Generation mode: "auto" (default), "json", or "tool"

• schemaName - Optional name for the schema

• schemaDescription - Optional description of what the schema represents

• schema - Required JSON Schema describing the object shape to generate

Optional Configuration: Common fields include system, temperature, maxTokens, topP, topK, frequencyPenalty, presencePenalty, and seed.

Example: Simple Object Extraction

msg.payload = {
  prompt: "Extract: Sarah Martinez, age 32, works as Senior Software Engineer at TechCorp.",
  schema: {
    type: "object",
    properties: {
      name: { type: "string", description: "Full name" },
      age: { type: "number", description: "Age in years" },
      jobTitle: { type: "string", description: "Job title" },
      company: { type: "string", description: "Employer" }
    },
    required: ["name", "age", "company"]
  },
  temperature: 0.3
};

Example: Array Generation

msg.payload = {
  prompt: "Generate 5 product ideas for smart home devices",
  output: "array",
  schema: {
    type: "object",
    properties: {
      name: { type: "string", description: "Product name" },
      description: { type: "string" },
      targetPrice: { type: "number", description: "Price in USD" }
    },
    required: ["name", "description", "targetPrice"]
  },
  temperature: 0.8
};

Example: File/PDF Input

msg.payload = {
  system: "Extract key information from this invoice PDF",
  messages: [
    {
      role: "user",
      content: [
        {
          type: "file",
          filename: "invoice-2025-001.pdf",
          mediaType: "application/pdf",
          data: msg.pdfData  // base64 encoded PDF
        }
      ]
    }
  ],
  schema: {
    type: "object",
    properties: {
      invoiceNumber: { type: "string" },
      invoiceDate: { type: "string", description: "YYYY-MM-DD" },
      vendorName: { type: "string" },
      total: { type: "number" }
    },
    required: ["invoiceNumber", "vendorName", "total"]
  }
};

Output Format

The node writes a standardized output response and details - the generation result - to msg[outputProperty] (default: msg.payload). This standardization means changing providers or models requires little to no refactoring of your surrounding flow logic.

Text Generation Output

When generating text, the output includes the generated text along with metadata about the generation process.

Main Fields:

• generationType - Always "text" for text generation

• text - The generated text response

• reasoning - Internal reasoning content (if available from reasoning models)

• messages - Array containing the response message

Generation Metadata:

• model - The model that generated the response (e.g., "gemini-2.5-flash")

• providerType - The AI provider ("openai", "anthropic", "vertex", etc.)

• providerId - ID of the provider connection used

• apiId - API connection ID used for the successful generation

• finishReason - Why generation stopped ("stop", "length", etc.)

• steps - Detailed generation steps (including request/response metadata when available)

• providerErrors - Array of errors from failed provider attempts

Step-level warnings are available in steps[n].warnings when provided by the model provider.

Token Usage:

• usage.inputTokens - Tokens in the input

• usage.outputTokens - Tokens in the output

• usage.totalTokens - Total tokens used

• usage.reasoningTokens - Reasoning tokens (for models like GPT-5, Gemini 2.5)

• usage.cachedInputTokens - Cached prompt tokens (if applicable)

Example Output

// msg.payload after node execution
{
  generationType: "text",
  text: "Two plus two equals four.",
  messages: [
    {
      role: "assistant",
      content: "Two plus two equals four."
    }
  ],
  usage: {
    inputTokens: 12,
    outputTokens: 6,
    totalTokens: 18
  },
  model: "gemini-2.5-flash",
  providerType: "vertex",
  finishReason: "stop",
  providerId: "google-vertex-ai",
  apiId: "google-vertex-ai",
  providerErrors: [],
  steps: [...]
}

Object Generation Output

When generating structured objects, the output includes the parsed object or array along with the same metadata as text generation.

If an earlier attempt failed (for example, invalid JSON) and a later retry succeeded, the final output still returns the successful object, while failure details remain available in providerErrors and steps.

Main Fields:

• generationType - Always "object" for object generation

• object - The generated structured data (object or array)

• reasoning - Internal reasoning content (if available from reasoning models)

Generation Metadata: Same fields as text generation (model, providerType, providerId, apiId, finishReason, steps, providerErrors)

Token Usage: Same fields as text generation (inputTokens, outputTokens, totalTokens, reasoningTokens, cachedInputTokens)

Example Output

// msg.payload after node execution
{
  generationType: "object",
  object: {
    name: "Sarah Martinez",
    age: 32,
    jobTitle: "Senior Software Engineer",
    company: "TechCorp"
  },
  usage: {
    inputTokens: 81,
    outputTokens: 50,
    totalTokens: 131
  },
  model: "gemini-2.5-flash",
  providerType: "vertex",
  finishReason: "stop",
  providerId: "google-vertex-ai",
  apiId: "google-vertex-ai",
  providerErrors: [],
  steps: [...]
}

Response Metadata

The node writes response metadata to msg[responseProperty] (default: msg._response).

AI Generate uses a WebSocket connection to the AI Gateway, so this metadata is a normalized success marker rather than the full upstream HTTP response.

Fields:

• statusCode - 200 on successful generation

• headers - A minimal header map (currently includes content-type)

Example

// msg._response after node execution
{
  statusCode: 200,
  headers: {
    "content-type": "application/json"
  }
}