AI Generate

Overview

The AI Generate node simplifies AI implementation in your Flows by providing a universal interface to multiple AI service providers. When paired with AI Routes and AI Connections, it handles the complexity of provider-specific APIs, authentication, and failover scenarios automatically.

What It Does

Calls AI providers using a single, standardized input format regardless of provider
Returns responses in a consistent, structured format with rich details
Automatically handles provider failover through AI Routes
Eliminates the need to refactor flows when switching providers or models

Benefits

Provider-agnostic - Switch providers or models without refactoring your flows
Automatic failover - If one provider fails, AI Routes automatically try the next configured provider
Consistent interface - Same input/output format across all providers
Rich response details - Access token usage, model information, and execution details
Production-ready - Works seamlessly with versioned AI Routes packaged in Services

Prerequisites

Before using the AI Generate node, you'll need:

AI Connection(s) - Connection credentials to one or more AI providers (OpenAI, Anthropic, Azure OpenAI, Google AI, Vertex AI, Vertex AI Anthropic)
AI Route - A configured AI Route that references your AI Connection(s) and specifies which model(s) to use

Basic Setup

Drag the AI Generate node from the AI Gateway category in the Flow Editor palette onto your canvas
Double-click the node to open the configuration panel
Select an AI Route:
- Choose from the dropdown of AI Routes created in your tenant
- Or dynamically specify using a msg property, environment variable, or static string
Specify the Input - Designate the msg object property that contains your input data (prompt, temperature, file data, expected schema, etc.)
Optional: Adjust timeout, Output, and Response property names

Input Format

The AI Generate node uses one universal, standardized input format regardless of which AI provider your AI Route calls. This standardization means changing providers or models requires little to no refactoring of your surrounding flow logic.

Your input msg object might contain:

One of the following is required...
- prompt - simple text prompt (use this OR messages array)
- messages - array of messages (use this OR prompt), which are required in order to pass base64 file content
schema - (optional) specify the JSON schema desired for structured JSON object responses rather than simple text generation
system - (optional) system instructions to guide the model's behavior
temperature - (optional) Control randomness in responses
maxTokens - maximum token to generate
topP - Nucleus sampling parameter
topK - Top-K sampling parameter
frequencyPenalty - Penalty for repeated tokens
presencePenalty - Penalty for repeated topics
stopSequences - Array of sequences that stop generation
seed - Random seed for deterministic output
Other provider parameters as needed

Input Examples - Text Generation

Example: Simple Prompt

msg.payload = {
  prompt: "What is 2+2? Answer in one sentence.",
  temperature: 0.3,
  maxTokens: 50
};

Example: Messages Array

msg.payload = {
  messages: [
    {
      role: "user",
      content: "Explain photosynthesis in two sentences."
    }
  ],
  temperature: 0.5,
  maxTokens: 100
};

Example: With System Message

msg.payload = {
  system: "You are a concise math tutor. Always show your work.",
  prompt: "What is 15 * 23?",
  temperature: 0.3,
  maxTokens: 100
};

Example: Multi-Turn Conversation

msg.payload = {
  messages: [
    {
      role: "user",
      content: "What's the capital of France?"
    },
    {
      role: "assistant",
      content: "The capital of France is Paris."
    },
    {
      role: "user",
      content: "What's its population?"
    }
  ],
  temperature: 0.3,
  maxTokens: 100
};

Input Examples - Object Generation

Schema Configuration:

output - Type of output: "object" (default) or "array"
mode - Generation mode: "auto" (default), "json", or "tool"
schemaName - Optional name for the schema
schemaDescription - Optional description of what the schema represents

Optional Configuration: Same fields as text generation (system, temperature, maxTokens, etc.)

Example: Simple Object Extraction

msg.payload = {
  prompt: "Extract: Sarah Martinez, age 32, works as Senior Software Engineer at TechCorp.",
  schema: {
    type: "object",
    properties: {
      name: { type: "string", description: "Full name" },
      age: { type: "number", description: "Age in years" },
      jobTitle: { type: "string", description: "Job title" },
      company: { type: "string", description: "Employer" }
    },
    required: ["name", "age", "company"]
  },
  temperature: 0.3
};

Example: Array Generation

msg.payload = {
  prompt: "Generate 5 product ideas for smart home devices",
  output: "array",
  schema: {
    type: "object",
    properties: {
      name: { type: "string", description: "Product name" },
      description: { type: "string" },
      targetPrice: { type: "number", description: "Price in USD" }
    },
    required: ["name", "description", "targetPrice"]
  },
  temperature: 0.8
};

Example: File/PDF Input

msg.payload = {
  system: "Extract key information from this invoice PDF",
  messages: [
    {
      role: "user",
      content: [
        {
          type: "file",
          filename: "invoice-2025-001.pdf",
          mediaType: "application/pdf",
          data: msg.pdfData  // base64 encoded PDF
        }
      ]
    }
  ],
  schema: {
    type: "object",
    properties: {
      invoiceNumber: { type: "string" },
      invoiceDate: { type: "string", description: "YYYY-MM-DD" },
      vendorName: { type: "string" },
      total: { type: "number" }
    },
    required: ["invoiceNumber", "vendorName", "total"]
  }
};

Output Format

The node writes a standardized output response and details - the generation result - to msg[outputProperty] (default: msg.payload). This standardization means changing providers or models requires little to no refactoring of your surrounding flow logic.

Text Generation Output

When generating text, the output includes the generated text along with metadata about the generation process.

Main Fields:

generationType - Always "text" for text generation
text - The generated text response
reasoning - Internal reasoning content (if available from reasoning models)
messages - Array containing the response message

Generation Metadata:

model - The model that generated the response (e.g., "gemini-2.5-flash")
providerType - The AI provider ("openai", "anthropic", "vertex", etc.)
providerId - ID of the provider connection used
finishReason - Why generation stopped ("stop", "length", etc.)
steps - Detailed generation steps including full request/response data
providerErrors - Array of errors from failed provider attempts
warnings - warnings from the model provider (e.g. unsupported settings)

Token Usage:

usage.inputTokens - Tokens in the input
usage.outputTokens - Tokens in the output
usage.totalTokens - Total tokens used
usage.reasoningTokens - Reasoning tokens (for models like GPT-5, Gemini 2.5)
usage.cachedInputTokens - Cached prompt tokens (if applicable)

Example Output

// msg.payload after node execution
{
  generationType: "text",
  text: "Two plus two equals four.",
  messages: [
    {
      role: "assistant",
      content: "Two plus two equals four."
    }
  ],
  usage: {
    inputTokens: 12,
    outputTokens: 6,
    totalTokens: 18
  },
  model: "gemini-2.5-flash",
  providerType: "vertex",
  finishReason: "stop",
  providerId: "google-vertex-ai",
  providerErrors: [],
  steps: [...]
}

Object Generation Output

When generating structured objects, the output includes the parsed object or array along with the same metadata as text generation.

Main Fields:

generationType - Always "object" for object generation
object - The generated structured data (object or array)
reasoning - Internal reasoning content (if available from reasoning models)

Generation Metadata: Same fields as text generation (model, providerType, providerId, finishReason, steps, providerErrors, warnings)

Token Usage: Same fields as text generation (inputTokens, outputTokens, totalTokens, reasoningTokens, cachedInputTokens)

Example Output

// msg.payload after node execution
{
  generationType: "object",
  object: {
    name: "Sarah Martinez",
    age: 32,
    jobTitle: "Senior Software Engineer",
    company: "TechCorp"
  },
  usage: {
    inputTokens: 81,
    outputTokens: 50,
    totalTokens: 131
  },
  model: "gemini-2.5-flash",
  providerType: "vertex",
  finishReason: "stop",
  providerId: "google-vertex-ai",
  providerErrors: [],
  steps: [...]
}

Response Metadata

The node writes HTTP response metadata to msg[responseProperty] (default: msg._response).

Fields:

statusCode - HTTP status code (200 for success)
headers - HTTP response headers
responseUrl - The AI Gateway endpoint URL

Example

// msg._response after node execution
{
  statusCode: 200,
  headers: {
    "access-control-allow-origin": "*",
    "content-type": "application/json; charset=utf-8",
    "content-length": "2626",
    "etag": "W/\"a42-j15VhUEcpgnJzva8l/5p7/P6UQ4\"",
    "date": "Thu, 13 Nov 2025 14:29:45 GMT",
    "connection": "keep-alive",
    ...
  },
  responseUrl: "http://ai-gateway-api.contextual.svc.cluster.local/api/v1/generate",
}

PreviousAI Gateway NextCommon

Last updated 3 days ago

Was this helpful?