What Lives in Your AI Agent's Context Window

Your conversation logs are quietly hosting a guest nobody invited. Maybe it's an account number lounging where it has no business being, a token that snuck in through the side door, or some random field that hitched a ride on a tool response because that's what the query happened to cough up. The AI took it in, put it to good use, and now it's sitting cushily in the conversation log, the post-call summary, and whatever debugging session springs up the next time someone pastes that transcript somewhere to figure out what went sideways.

Nobody rolled out the welcome mat for it. It's just what happens when you build the tool the obvious way: query the database, return the result, let the model sort out the rest. It works. It ships fast. And it lounges there undisturbed for a good three months before anyone so much as raises an eyebrow.

When they finally do, the first instinct is to tidy up the logs. Strip the account number before it reaches the pipeline, redact the tokens, sanitize on write. Which helps! Technically. But cleaning the log only ever cleans the log. The data is still tucked into the context window and the summary. It's still in the compliance transcript someone fishes out six months later, and the screenshot a support engineer cheerfully snapped to file a bug report.

By the time you're sanitizing logs, the data has already toured the building and let itself into every room it was never supposed to see.

Context vs. capability

This is a distinction worth drawing carefully as most of the exposure sneaks in right where the line gets blurry: an AI agent doesn't need to know sensitive data to act on it. Those are two completely different jobs, and collapsing them into one is where the wheels tend to come off.

Think of a coat check. You hand over your coat, you get a numbered tag, and that tag is useless to anyone who isn't the coat check. You don't need to know where your coat is hanging to get it back at the end of the night, you just need the tag.

A tool function works the same way: it can look up an account balance with a stored customer ID without the AI ever laying eyes on the ID. The AI can cheerfully tell a caller their balance is current without ever holding the number itself. An auth token can green-light a backend API call without once showing its face in the conversation.

The AI's job is to figure out what the customer needs and call the right functions. The functions' job is to handle the data. Make the AI a pass-through relay for sensitive values and you've handed it access it never needed. Worse, you've handed it a risk it has no way to manage.

The metadata pattern is built on exactly this line. It gives tool functions a private channel to pass sensitive data to each other across the life of a conversation, without a single value ever wandering into the AI's context. The AI stays useful and stays ignorant, which turns out to be exactly the combination you want.

Context accumulation

When a tool function hands a value back to the model, it takes that value in as part of its context, and that context is the raw material for whatever it says next. It's also logged, summarized, sometimes cached, and in plenty of setups shipped off to third-party providers for inference. You usually have a firm grip on what goes into a prompt at the start of a conversation. You have a much looser grip on what piles up over the course of one.

Picture a multi-step support call. The AI verifies the caller, pulls up their account, retrieves an order, then handles a return. If each of those functions dumps its raw data into the AI's response, then by the time the call wraps, the model is quietly holding the customer's account number, order IDs, shipping address, payment method, and auth token. Not one of those values was needed for the AI to say what it said. They were needed for the backend functions to do their jobs. The AI was just the messenger, and now the whole pile is sitting in its memory.

There's a sneakier problem riding on top of the compliance risk: models hallucinate. A model that saw an account number earlier might spit out a slightly mangled version of it later. There's no intent to leak behind it. It's just generating text, and that pattern is right there in its context. This is a well-documented failure mode for models handed more than they need to finish the job. It isn't hypothetical. It shows up in production.

The fix is to keep the data out of the context from the start.

Below the context

At SignalWire, a SignalWire AI Gateway (SWAIG) function does two things when it answers. It returns a response, the text the AI reads and uses to keep the conversation going. And through a set_meta_data action, it can tuck a data payload into the call's metadata, where the platform stores it and hands it to later functions. The model reads the first part. It never lays eyes on the second.

That metadata is scoped by a meta_data_token. Any function sharing the same token can reach what was stored under it, so a customer ID stashed during verification is right there during order modification five functions later, without the AI ever touching it. The conversation log has what the AI said, and it doesn't have what the functions knew.

This works because the metadata lives in the platform's data layer, not in the AI's context. It moves from function to function under that token, and the model plays no part in the handoff. From the model's point of view, it called a function, got an answer, and kept talking. Whatever happened inside that call is simply outside its field of vision.

The metadata pattern in practice

Customer verification is a useful starting point because it generates the most sensitive data at the beginning of a conversation, and that data needs to be available throughout. The contrast between implementations is clearest here.

The common first version returns everything in the response field:

// The version that feels fine until the first compliance review
if (funcName === 'verify_customer') {
  const customer = db.findByPhone(args.phone_number);
  const verified  = db.verifyPin(customer.id, args.pin);
  if (verified) {
    return res.json({
      response: `Verified. Account number ${customer.account_number},
                 auth token ${customer.auth_token}. How can I help?`
    });
  }
}

// The version that feels fine until the first compliance review
if (funcName === 'verify_customer') {
  const customer = db.findByPhone(args.phone_number);
  const verified  = db.verifyPin(customer.id, args.pin);
  if (verified) {
    return res.json({
      response: `Verified. Account number ${customer.account_number},
                 auth token ${customer.auth_token}. How can I help?`
    });
  }
}

// The version that feels fine until the first compliance review
if (funcName === 'verify_customer') {
  const customer = db.findByPhone(args.phone_number);
  const verified  = db.verifyPin(customer.id, args.pin);
  if (verified) {
    return res.json({
      response: `Verified. Account number ${customer.account_number},
                 auth token ${customer.auth_token}. How can I help?`
    });
  }
}

// The version that feels fine until the first compliance review
if (funcName === 'verify_customer') {
  const customer = db.findByPhone(args.phone_number);
  const verified  = db.verifyPin(customer.id, args.pin);
  if (verified) {
    return res.json({
      response: `Verified. Account number ${customer.account_number},
                 auth token ${customer.auth_token}. How can I help?`
    });
  }
}

This works. The AI can pass those values to downstream functions. It also holds them in its context for the rest of the call, includes them in any summarization, and surfaces them in transcripts. The AI didn't need to know the account number or the auth token. It just needed to know that verification succeeded.

The same function, using metadata:

// Verification with metadata: the AI learns the outcome, not the data
if (funcName === 'verify_customer') {
  const customer = db.findByPhone(args.phone_number);
  const verified  = db.verifyPin(customer.id, args.pin);
  if (verified) {
    return res.json({
      // The model receives this
      response: `Thanks for verifying, ${customer.first_name}. What can I help you with?`,
      // The model never sees this — set_meta_data stores it for later functions
      action: [
        {
          set_meta_data: {
            verified:        true,
            customer_id:     customer.id,
            account_number:  customer.account_number,
            auth_token:      generateToken(customer.id),
            customer_tier:   customer.tier
          }
        }
      ]
    });
  }
  return res.json({
    response: "That PIN doesn't match our records. Please try again."
  });
}

// Verification with metadata: the AI learns the outcome, not the data
if (funcName === 'verify_customer') {
  const customer = db.findByPhone(args.phone_number);
  const verified  = db.verifyPin(customer.id, args.pin);
  if (verified) {
    return res.json({
      // The model receives this
      response: `Thanks for verifying, ${customer.first_name}. What can I help you with?`,
      // The model never sees this — set_meta_data stores it for later functions
      action: [
        {
          set_meta_data: {
            verified:        true,
            customer_id:     customer.id,
            account_number:  customer.account_number,
            auth_token:      generateToken(customer.id),
            customer_tier:   customer.tier
          }
        }
      ]
    });
  }
  return res.json({
    response: "That PIN doesn't match our records. Please try again."
  });
}

// Verification with metadata: the AI learns the outcome, not the data
if (funcName === 'verify_customer') {
  const customer = db.findByPhone(args.phone_number);
  const verified  = db.verifyPin(customer.id, args.pin);
  if (verified) {
    return res.json({
      // The model receives this
      response: `Thanks for verifying, ${customer.first_name}. What can I help you with?`,
      // The model never sees this — set_meta_data stores it for later functions
      action: [
        {
          set_meta_data: {
            verified:        true,
            customer_id:     customer.id,
            account_number:  customer.account_number,
            auth_token:      generateToken(customer.id),
            customer_tier:   customer.tier
          }
        }
      ]
    });
  }
  return res.json({
    response: "That PIN doesn't match our records. Please try again."
  });
}

// Verification with metadata: the AI learns the outcome, not the data
if (funcName === 'verify_customer') {
  const customer = db.findByPhone(args.phone_number);
  const verified  = db.verifyPin(customer.id, args.pin);
  if (verified) {
    return res.json({
      // The model receives this
      response: `Thanks for verifying, ${customer.first_name}. What can I help you with?`,
      // The model never sees this — set_meta_data stores it for later functions
      action: [
        {
          set_meta_data: {
            verified:        true,
            customer_id:     customer.id,
            account_number:  customer.account_number,
            auth_token:      generateToken(customer.id),
            customer_tier:   customer.tier
          }
        }
      ]
    });
  }
  return res.json({
    response: "That PIN doesn't match our records. Please try again."
  });
}

The model learns the customer's first name and that verification succeeded. The account number, auth token, customer ID, and tier are stored in the call's data layer, accessible to subsequent functions, invisible to the model.

A downstream function reads from meta_data directly, using the stored values without returning them to the AI:

// Account details: uses metadata, keeps raw values out of the response
if (funcName === 'get_account_details') {
  const { meta_data } = req.body;
  if (!meta_data.verified) {
    return res.json({
      response: "Identity verification is required before accessing account details."
    });
  }
  const details = db.getAccountDetails(meta_data.account_number);
  return res.json({
    // The account number was used, not returned
    response: `This account has been active since ${details.created_date}.
               Current plan: ${meta_data.customer_tier}.`
  });
}

// Account details: uses metadata, keeps raw values out of the response
if (funcName === 'get_account_details') {
  const { meta_data } = req.body;
  if (!meta_data.verified) {
    return res.json({
      response: "Identity verification is required before accessing account details."
    });
  }
  const details = db.getAccountDetails(meta_data.account_number);
  return res.json({
    // The account number was used, not returned
    response: `This account has been active since ${details.created_date}.
               Current plan: ${meta_data.customer_tier}.`
  });
}

// Account details: uses metadata, keeps raw values out of the response
if (funcName === 'get_account_details') {
  const { meta_data } = req.body;
  if (!meta_data.verified) {
    return res.json({
      response: "Identity verification is required before accessing account details."
    });
  }
  const details = db.getAccountDetails(meta_data.account_number);
  return res.json({
    // The account number was used, not returned
    response: `This account has been active since ${details.created_date}.
               Current plan: ${meta_data.customer_tier}.`
  });
}

// Account details: uses metadata, keeps raw values out of the response
if (funcName === 'get_account_details') {
  const { meta_data } = req.body;
  if (!meta_data.verified) {
    return res.json({
      response: "Identity verification is required before accessing account details."
    });
  }
  const details = db.getAccountDetails(meta_data.account_number);
  return res.json({
    // The account number was used, not returned
    response: `This account has been active since ${details.created_date}.
               Current plan: ${meta_data.customer_tier}.`
  });
}

The account number was used to fetch the record. It was never sent back to the model. The conversation log reads: "This account has been active since March 2021. Current plan: Professional." The account number, customer ID, and auth token appear nowhere in what the AI said.

Metadata accumulation across conversation

The security payoff is obvious from a single function pair. The architectural payoff shows up once the conversation runs long.

Take a caller who verifies their identity, asks about an order, then wants to change it. Each function reads the metadata it needs and writes its own results back on top, so the stored state grows as the call goes on.

After verify_customer:

{
  "verified":       true,
  "customer_id":    "cust_8821",
  "account_number": "ACC-004419",
  "auth_token":     "tok_9f2a...",
  "customer_tier":  "professional"
}
lookup_order reads what's already there, does its work, and carries the existing state forward alongside its own results:
js
if (funcName === 'lookup_order') {
  const { meta_data } = req.body;
  const order = db.findOrderByCustomer(meta_data.customer_id);
  return res.json({
    response: `Found it. Your order is currently ${order.status}.`,
    action: [
      {
        // carry the existing metadata forward, then add this function's results
        set_meta_data: {
          ...meta_data,
          order_id:     order.id,
          order_status: order.status,
          order_items:  order.items
        }
      }
    ]
  });
}

{
  "verified":       true,
  "customer_id":    "cust_8821",
  "account_number": "ACC-004419",
  "auth_token":     "tok_9f2a...",
  "customer_tier":  "professional"
}
lookup_order reads what's already there, does its work, and carries the existing state forward alongside its own results:
js
if (funcName === 'lookup_order') {
  const { meta_data } = req.body;
  const order = db.findOrderByCustomer(meta_data.customer_id);
  return res.json({
    response: `Found it. Your order is currently ${order.status}.`,
    action: [
      {
        // carry the existing metadata forward, then add this function's results
        set_meta_data: {
          ...meta_data,
          order_id:     order.id,
          order_status: order.status,
          order_items:  order.items
        }
      }
    ]
  });
}

{
  "verified":       true,
  "customer_id":    "cust_8821",
  "account_number": "ACC-004419",
  "auth_token":     "tok_9f2a...",
  "customer_tier":  "professional"
}
lookup_order reads what's already there, does its work, and carries the existing state forward alongside its own results:
js
if (funcName === 'lookup_order') {
  const { meta_data } = req.body;
  const order = db.findOrderByCustomer(meta_data.customer_id);
  return res.json({
    response: `Found it. Your order is currently ${order.status}.`,
    action: [
      {
        // carry the existing metadata forward, then add this function's results
        set_meta_data: {
          ...meta_data,
          order_id:     order.id,
          order_status: order.status,
          order_items:  order.items
        }
      }
    ]
  });
}

{
  "verified":       true,
  "customer_id":    "cust_8821",
  "account_number": "ACC-004419",
  "auth_token":     "tok_9f2a...",
  "customer_tier":  "professional"
}
lookup_order reads what's already there, does its work, and carries the existing state forward alongside its own results:
js
if (funcName === 'lookup_order') {
  const { meta_data } = req.body;
  const order = db.findOrderByCustomer(meta_data.customer_id);
  return res.json({
    response: `Found it. Your order is currently ${order.status}.`,
    action: [
      {
        // carry the existing metadata forward, then add this function's results
        set_meta_data: {
          ...meta_data,
          order_id:     order.id,
          order_status: order.status,
          order_items:  order.items
        }
      }
    ]
  });
}

Now the stored state holds both sets of data:

{
  "verified":       true,
  "customer_id":    "cust_8821",
  "account_number": "ACC-004419",
  "auth_token":     "tok_9f2a...",
  "customer_tier":  "professional",
  "order_id":       "ORD-55102",
  "order_status":   "processing",
  "order_items":    ["SKU-881", "SKU-334"]
}

{
  "verified":       true,
  "customer_id":    "cust_8821",
  "account_number": "ACC-004419",
  "auth_token":     "tok_9f2a...",
  "customer_tier":  "professional",
  "order_id":       "ORD-55102",
  "order_status":   "processing",
  "order_items":    ["SKU-881", "SKU-334"]
}

{
  "verified":       true,
  "customer_id":    "cust_8821",
  "account_number": "ACC-004419",
  "auth_token":     "tok_9f2a...",
  "customer_tier":  "professional",
  "order_id":       "ORD-55102",
  "order_status":   "processing",
  "order_items":    ["SKU-881", "SKU-334"]
}

{
  "verified":       true,
  "customer_id":    "cust_8821",
  "account_number": "ACC-004419",
  "auth_token":     "tok_9f2a...",
  "customer_tier":  "professional",
  "order_id":       "ORD-55102",
  "order_status":   "processing",
  "order_items":    ["SKU-881", "SKU-334"]
}

By the time modify_order runs, everything it needs is already sitting in metadata, the customer's identity, their authorization, the order details, and not one piece of it came from the model:

if (funcName === 'modify_order') {
  const { meta_data, argument: args } = req.body;
  if (!meta_data.verified) {
    return res.json({ response: "Identity verification is required." });
  }
  if (meta_data.order_status !== 'processing') {
    return res.json({
      response: `This order has already ${meta_data.order_status}.
                 A return can be initiated instead.`
    });
  }
  db.modifyOrder({
    customerId: meta_data.customer_id,
    orderId:    meta_data.order_id,
    changes:    args.modifications,
    token:      meta_data.auth_token
  });
  return res.json({
    response: "The order has been updated. Changes will show up within a few minutes."
  });
}

if (funcName === 'modify_order') {
  const { meta_data, argument: args } = req.body;
  if (!meta_data.verified) {
    return res.json({ response: "Identity verification is required." });
  }
  if (meta_data.order_status !== 'processing') {
    return res.json({
      response: `This order has already ${meta_data.order_status}.
                 A return can be initiated instead.`
    });
  }
  db.modifyOrder({
    customerId: meta_data.customer_id,
    orderId:    meta_data.order_id,
    changes:    args.modifications,
    token:      meta_data.auth_token
  });
  return res.json({
    response: "The order has been updated. Changes will show up within a few minutes."
  });
}

if (funcName === 'modify_order') {
  const { meta_data, argument: args } = req.body;
  if (!meta_data.verified) {
    return res.json({ response: "Identity verification is required." });
  }
  if (meta_data.order_status !== 'processing') {
    return res.json({
      response: `This order has already ${meta_data.order_status}.
                 A return can be initiated instead.`
    });
  }
  db.modifyOrder({
    customerId: meta_data.customer_id,
    orderId:    meta_data.order_id,
    changes:    args.modifications,
    token:      meta_data.auth_token
  });
  return res.json({
    response: "The order has been updated. Changes will show up within a few minutes."
  });
}

if (funcName === 'modify_order') {
  const { meta_data, argument: args } = req.body;
  if (!meta_data.verified) {
    return res.json({ response: "Identity verification is required." });
  }
  if (meta_data.order_status !== 'processing') {
    return res.json({
      response: `This order has already ${meta_data.order_status}.
                 A return can be initiated instead.`
    });
  }
  db.modifyOrder({
    customerId: meta_data.customer_id,
    orderId:    meta_data.order_id,
    changes:    args.modifications,
    token:      meta_data.auth_token
  });
  return res.json({
    response: "The order has been updated. Changes will show up within a few minutes."
  });
}

Across the whole exchange, the AI's only job was to read the caller's intent and route to the right function. Every authorization check, every database query, every API call ran on data the model never held. Even the conversation logs read just like an ordinary support call. The sensitive part simply isn't in it.

Performance case

Security is the obvious reason to reach for this pattern. There's a second payoff that tends to show up only after a voice AI has been in production for a while: redundant database queries.

Without metadata, every function that needs the customer's record goes and fetches it. verify_customer fetches it. get_account_details fetches it. lookup_order fetches it to confirm the order belongs to the right person. modify_order fetches it again to check authorization. Four functions, four round trips, all pulling largely the same record.

With metadata, the identity and account details are fetched once, during verification, and carried forward. Every function after that reads the stable fields it needs from the metadata layer instead of going back to the database. On this call, that's three round trips saved. One call, no big deal. But at real volume, the cumulative effect on latency and database load is easy to measure. The code gets simpler too: each function can trust that meta_data.customer_id is there and valid instead of re-fetching and re-validating from scratch.

So the pattern earns its keep twice, once in the audit log and once on the infrastructure bill.

Response vs. metadata

The decision rule is simple. If a value would cause a problem in a transcript that a compliance team, a support engineer, and the customer could all read at the same time, it belongs in metadata.

Account IDs, auth tokens, verification flags, permission levels, internal database keys, session identifiers: all metadata. The customer's name is fine in the conversation, they're the one who said it. What they're calling about is fine, that's the topic. Public information like pricing, hours, or policy details is fine too, there's nothing there to protect.

The tricky ones are fields like order status or account tier. They don't feel sensitive because they aren't financial data. But a status of "suspended" or a tier of "at risk of churn" isn't something you want a backend system handing to the model without controlling how it gets framed. When you're not sure which side of the line a value sits on, put it in metadata. A function can always pull a value out and build a careful response around it. A value already sitting in the model's context can't be taken back.

Scope as a design decision 

An AI agent that doesn't know a customer's account number isn't crippled. It's correctly scoped. It can run a complete, genuinely helpful conversation without ever holding data that belongs in a backend function rather than a language model.

The backend knows what it needs to execute. The model knows what it needs to converse. Those are two different sets of knowledge, and metadata is what keeps them apart, with no custom middleware, no separate caching layer, and no fighting the platform to enforce a boundary it was built to support.

This blog is part 1 of a series. Part 2 covers the highest-stakes version of keeping sensitive data out of context: collecting payment over the phone with the SignalWire Markup Language (SWML) pay method, where the card number and CVV are captured by the platform and sent straight to your payment gateway, never reaching your functions or the model's context. The AI only ever sees the result.

Start building with data that stays where it belongs

Have questions about implementing the pattern in your own agent? Bring them to the SignalWire Discord, where the developer community and the SignalWire team can help you work through it.