Is Your LLM API Serving the Model You Requested? Fingerprinting AI with Random Numbers

If an API accepts a model name containing claude or gpt, does that prove the requested model actually handled the request?

Asking the model, “Are you really Claude?” does not help. A model can report whatever identity its system prompt or training tells it to report.

A much more ordinary question may reveal more: “Name one random integer from 1 to 100.” Ask it repeatedly, and the answers can begin to outline the behavior of the model behind the API.

A preprint submitted on July 11, 2026, titled “One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions,” turns those biases into a behavioral LLM fingerprint.

In Figure 1, the researchers asked the same question 30 times. GPT-4o spread its answers across values including 42, 37, and 57. Claude Sonnet 5 concentrated on 47, while Llama 3.3 favored 53. Qwen3-Max returned 42 on all 30 runs.

One potentially misleading phrase needs to be cleared up immediately.

“One token is enough” does not mean one query can identify a model. It means each response can be approximately one token long. Identification comes from collecting many such responses and comparing their distribution.

A model name is only a string in an API request

From an application’s point of view, an LLM API accepts a model name and a prompt, then returns text. Clients usually cannot inspect the model weights, raw logits, or inference server.

Even a first-party API can update models or route traffic internally in ways clients cannot fully observe. The serving chain becomes more opaque when an aggregator, reseller, or several inference providers sit between the application and the model.

The paper considers threats such as substitution with a cheaper model, aggressive quantization, or rollback to an older checkpoint. It does not claim that every service is behaving dishonestly. The underlying problem is that clients have few technical ways to verify the string they put in the model field.

This matters to AI coding systems. A coding agent may suddenly become worse at repairing tests. A fallback provider may behave differently despite exposing the same model name. A previously reliable prompt may stop reproducing its earlier results after a silent update. The model is not the only possible cause, but without observable signals it is difficult to isolate the cause at all.

The paper in 30 seconds

The study measured 165 models across 19 documented model families and 53 serving providers. It used ten tasks in English, Russian, Chinese, and Arabic, producing 40 task-language probe cells.

The researchers collected 326,047 responses: 23.3 million input tokens and 1.16 million output tokens. Total collection cost was $34.44, or an average of $0.21 per model.

The prompts asked for numbers, letters, words, colors, animals, cities, and coin flips—answers that do not require long generations. After normalizing the responses, the researchers built histograms and treated the resulting distributions as model fingerprints. The dataset and collection and analysis software were released alongside the paper.

Why an AI cannot choose a truly random number

An LLM does not interpret the word “random,” construct a fair 100-sided die, and roll it internally. It estimates a probability distribution for the next token from the preceding context, then selects a candidate according to its decoding configuration.

That probability distribution carries many layers of history:

• how often numbers and words appeared in the training corpus;
• numbers humans tend to perceive as random;
• how the tokenizer divides candidate answers into tokens;
• instruction tuning and preference optimization;
• probability changes introduced by distillation or quantization;
• decoding settings such as temperature;
• differences in serving implementations; and
• linguistic or cultural selection biases.

Ask 100 people to choose a favorite number and their answers will not be perfectly uniform either. Schools, countries, and generations may produce slightly different favorites. In much the same way, an LLM’s training and post-training history leaves behind selection habits.

Raising the temperature can increase output diversity, but it does not make the underlying distribution uniform. If 42 starts with a higher probability than other values, it will still appear more often over repeated samples. That bias is the evidence used in this investigation.

How the researchers collected an AI fingerprint

Ten simple probes

The battery contained the following tasks:

1. a random number from 1 to 100;
2. a random number from 1 to 10;
3. a favorite number;
4. a random letter;
5. a random word;
6. a random color;
7. a favorite color;
8. a random animal;
9. a random city; and
10. a coin flip.

It combines closed answer spaces, such as numbers and coins, with open spaces, such as words and cities. Running the tasks in four languages probes different slices of a model’s learned prior.

Repeat the same questions

For normally priced models, the researchers ran each cell 30 times at temperature 1.0 and three times at temperature 0. The temperature-1.0 count was reduced to 15 for expensive models.

One response cannot reveal whether an answer was an accident or a habit. Thirty occurrences of 42 only become useful after they are converted into a frequency: how often did 42 appear out of 30 attempts?

Normalize the responses

An API can express the same answer in several forms:

Blue
blue
blue.
“blue”

The study normalized case, punctuation, Unicode, numeral systems, and language-specific variants. Invalid, empty, refused, and reasoning-contaminated responses were recorded and separated rather than silently discarded or mixed into the fingerprint.

Result 1: Models have favorite answers

The answer distributions were far more concentrated than a uniform random process would suggest. Across 6,572 valid cells, the median share of the most common answer was 71%, and median entropy was 1.00 bit.

A uniform choice among 100 integers has an entropy of roughly 6.64 bits. A median near one bit indicates that answers in many cells collapsed onto a small set of candidates.

When samples from the same model were split into two halves, the median cell-level Jensen–Shannon distance was 0.075. The corresponding median for different models was 0.489. Two fingerprints from the same model therefore tended to be close, while fingerprints from different models tended to be much farther apart.

The important signal is not one specific number. “It answered 42, so it must be Qwen” is not a valid test. The fingerprint comes from a distribution spanning repeated answers, multiple prompts, and multiple languages.

Result 2: The habits reveal model lineage

When the researchers clustered nearby fingerprints, models from documented families such as GPT, Qwen, Mistral, Claude, and Llama tended to appear near their relatives.

In a leave-one-out nearest-neighbor test, the family of 163 models was classified with 59.5% accuracy. The frequency-weighted chance baseline was 18.4%, making the observed rate about 3.2 times higher than chance.

That is not enough to identify an unknown model with certainty. Heavy post-training can also overwrite much of a base model’s original behavior. Still, recovering the documented family more than half the time from choices of numbers, colors, and cities is a striking amount of lineage information from such small outputs.

Result 3: Roughly 100 short queries can support identity checks

The verification experiment compared a fingerprint collected from a trusted reference deployment with one collected from an endpoint under audit.

flowchart LR
    accTitle: Verifying an LLM with a behavioral fingerprint
    accDescr: The same short questions are repeatedly sent to a trusted reference model and an API under audit. Their answer distributions are compared with Jensen–Shannon distance, and the distance is evaluated against a threshold.
    A[Trusted reference model] --> C[Repeat the same short probes]
    B[API under audit] --> C
    C --> D[Reference answer distribution]
    C --> E[Audit answer distribution]
    D --> F[Jensen–Shannon distance]
    E --> F
    F --> G[Compare with threshold]
    G --> H[Match or investigate]

Verifying an LLM with a behavioral fingerprint

Using all 40 cells, the study reported an AUC of 0.971 and an equal error rate, or EER, of 7.3%. Eight cells produced an EER of 10.6%, corresponding to 120 queries when each cell was sampled 15 times. A single cell produced a much weaker EER of 23.3%.

EER is the operating point where the rate of rejecting a genuine identity equals the rate of accepting an impostor identity. Lower is better, but 7.3% is not comparable to a password, signature, or cryptographic attestation. It is a useful operational signal, not proof of model identity.

The phrase “roughly 100 queries” is a rounded description of the eight-cell, 15-repetition setting. A small demo using four cells 30 times also makes 120 requests, but that does not mean it inherits the paper’s measured accuracy.

Suspicious differences behind the API boundary

The study reports that an endpoint marketed as a proprietary flagship model had an answer distribution statistically indistinguishable from an open-weight Qwen model.

It also examined 34 provider pairs that were expected to serve the same model. Ten pairs showed distributional differences comparable to differences between distinct models. Across the full cross-provider verification experiment, however, AUC remained 0.880, suggesting that much of the fingerprint survived a change in serving provider.

None of these observations proves intentional model substitution. Updated weights, authorized quantization, different decoding implementations, caching, or other serving configuration changes can also shift a distribution. The paper explicitly frames these findings as statistical deviations in served behavior, not accusations of fraud.

Jensen–Shannon divergence without the intimidating math

Consider three distributions:

Model A: 42=60%, 47=20%, 53=20%
Model B: 42=55%, 47=25%, 53=20%
Model C:  7=80%, 13=20%

A and B look similar. C uses a largely different set of answers.

Jensen–Shannon divergence, or JSD, measures the difference between two probability distributions. With base-2 logarithms, it falls between zero and one. Zero means the distributions are identical; values closer to one indicate greater separation. It is symmetric, so comparing A with B produces the same result as comparing B with A. It also works when an answer appears in only one distribution.

The key is to compare distributions, not individual responses. Counts are first converted into probabilities, then the two probability vectors are measured against each other.

Trying AI fingerprint collection with Node.js

The following small experiment targets an OpenAI-compatible Chat Completions API. It is not a complete reproduction of the paper. Its purpose is to make answer bias and between-model distance visible in a local experiment.

It uses three files:

• collect.js calls four probes 30 times each;
• normalize.js canonicalizes answers and builds histograms; and
• compare.js computes mean JSD between two runs.

Collect responses

import { writeFile } from "node:fs/promises";

const baseUrl = process.env.LLM_BASE_URL;
const apiKey = process.env.LLM_API_KEY;
const model = process.env.LLM_MODEL;
const output = process.argv[2] ?? "results.json";

if (!baseUrl || !apiKey || !model) {
  throw new Error("Set LLM_BASE_URL, LLM_API_KEY, and LLM_MODEL");
}

const prompts = [
  { id: "number-100", text: "Name one random integer from 1 to 100. Answer with the value only." },
  { id: "color", text: "Name one random color. Answer with one word only." },
  { id: "animal", text: "Name one random animal. Answer with one word only." },
  { id: "coin", text: "Flip a coin. Answer with Heads or Tails only." }
];

const repetitions = 30;

async function ask(prompt) {
  const response = await fetch(`${baseUrl.replace(/\/$/, "")}/chat/completions`, {
    method: "POST",
    headers: {
      "authorization": `Bearer ${apiKey}`,
      "content-type": "application/json"
    },
    body: JSON.stringify({
      model,
      temperature: 1,
      max_tokens: 16,
      messages: [{ role: "user", content: prompt }]
    })
  });

  if (!response.ok) {
    throw new Error(`API ${response.status}: ${await response.text()}`);
  }

  const data = await response.json();
  return {
    answer: data.choices?.[0]?.message?.content ?? "",
    usage: data.usage ?? null,
    responseModel: data.model ?? null,
    collectedAt: new Date().toISOString()
  };
}

const results = [];
for (const prompt of prompts) {
  for (let run = 0; run < repetitions; run += 1) {
    try {
      results.push({ promptId: prompt.id, run, ...(await ask(prompt.text)) });
    } catch (error) {
      results.push({ promptId: prompt.id, run, error: String(error) });
    }
  }
}

await writeFile(output, JSON.stringify({ model, prompts, repetitions, results }, null, 2));
console.log(`Saved ${results.length} attempts to ${output}`);

The cap is 16 rather than one because numbers and quotation marks may tokenize differently across providers. A one-token cap can truncate an otherwise valid answer. Check the actual output-token count in usage. Some compatible APIs also expect max_completion_tokens instead of max_tokens.

Normalize answers

export function normalizeAnswer(value) {
  return value
    .normalize("NFKC")
    .trim()
    .toLowerCase()
    .replace(/^["'“”‘’「『]+|["'“”‘’」』]+$/g, "")
    .replace(/[.!?,;:。！？、，；：]+$/g, "")
    .trim();
}

export function histograms(document) {
  const output = {};

  for (const row of document.results) {
    if (row.error) continue;
    const answer = normalizeAnswer(row.answer);
    if (!answer) continue;

    output[row.promptId] ??= {};
    output[row.promptId][answer] = (output[row.promptId][answer] ?? 0) + 1;
  }

  return output;
}

A production audit needs stricter handling for numeral systems, color synonyms, language-specific variants, refusals, and malformed responses. Over-normalizing different meanings into one category can destroy the fingerprint rather than improve it.

Compare two fingerprints

import { readFile } from "node:fs/promises";
import { histograms } from "./normalize.js";

const [leftPath, rightPath] = process.argv.slice(2);
if (!leftPath || !rightPath) {
  throw new Error("Usage: node compare.js reference.json audit.json");
}

const readJson = async (path) => JSON.parse(await readFile(path, "utf8"));

function probabilities(counts, keys) {
  const total = Object.values(counts).reduce((sum, value) => sum + value, 0);
  return keys.map((key) => (counts[key] ?? 0) / total);
}

function kl(p, q) {
  return p.reduce((sum, value, index) => {
    if (value === 0) return sum;
    return sum + value * Math.log2(value / q[index]);
  }, 0);
}

function jsd(left, right) {
  const keys = [...new Set([...Object.keys(left), ...Object.keys(right)])];
  const p = probabilities(left, keys);
  const q = probabilities(right, keys);
  const midpoint = p.map((value, index) => (value + q[index]) / 2);
  return (kl(p, midpoint) + kl(q, midpoint)) / 2;
}

const left = histograms(await readJson(leftPath));
const right = histograms(await readJson(rightPath));
const promptIds = Object.keys(left).filter((id) => right[id]);
const distances = promptIds.map((id) => ({ id, jsd: jsd(left[id], right[id]) }));
const meanJsd = distances.reduce((sum, row) => sum + row.jsd, 0) / distances.length;

console.table(distances);
console.log({ meanJsd });

Collect a reference run and an audit run, then compare them:

node collect.js reference.json
node collect.js audit.json
node compare.js reference.json audit.json

Also compare two independently collected runs from the same model. Their distance estimates normal variation in your environment. Even if another model produces a larger distance, do not import the paper’s threshold and label an endpoint fake. Your prompt set, sample count, model population, and serving configuration are different.

Uses in AI coding and independent applications

Detecting model drift

Store a baseline distribution from a trusted environment and repeat the probes each week. A sudden JSD increase beyond the historically normal range can trigger an investigation into model updates or serving changes.

Comparing fallback providers

If an application normally uses a first-party API but falls back to another provider during an outage, compare fingerprints from both routes. They do not need to be identical, but the distance becomes useful metadata when explaining changes in application quality.

Checking AI coding-agent reproducibility

Record fingerprint distance alongside code-generation success rates, test pass rates, and latency. If performance and the fingerprint change at the same time, the model route becomes an investigation target alongside the prompt and repository.

Comparing local LLM deployments

Try the same base model with Q4 and Q8 quantization, different GGUF files, Ollama versus LM Studio, or different inference engines. This makes it possible to observe how much quantization and decoding implementations move the answer distribution.

Running a periodic CI check

A scheduled job can collect a fingerprint and notify Slack or email only when its distance crosses a locally calibrated warning range. Do not begin with one permanent threshold. First collect repeated baselines across time and providers so that normal variation is represented.

What this research cannot tell us

1. It is not cryptographic proof. It cannot prove that two endpoints use bit-identical weights.
2. It needs a trusted reference. Verification requires a previously enrolled fingerprint from a deployment such as a first-party API.
3. Fingerprints can drift after updates. The paper measured short-horizon stability, not drift over many months.
4. The census used one aggregator. The protocol is not tied to that aggregator, but the ecosystem findings may not generalize to every API market.
5. Mandatory hidden-reasoning models were excluded. A post-reasoning answer comes from a different generation process than the direct single-token completion being measured.
6. An anomaly does not imply misconduct. Updates, quantization, distillation, caching, and decoding settings can all change the distribution.
7. Resistance to prompt adaptation is not settled. Everyday tasks can be paraphrased, but the paper leaves a dedicated paraphrase-invariance experiment to future work.
8. Short outputs still incur input and request costs. Rate limits, repeated requests, and input tokens must be included in operational planning.

The technique is therefore closer to a smoke detector than a digital signature. An alarm should start an investigation involving serving logs, quality evaluations, and version data—not produce an immediate accusation.

Conclusion: AI models cannot completely hide how they choose

Even a tiny LLM response reflects tokenizer design, training data, post-training, quantization, distillation, and decoding behavior.

One answer of 42 tells us almost nothing. Collect roughly 100 answers across numbers, colors, animals, and cities, however, and small biases become distributions. Those distributions can provide evidence about model identity and lineage.

Behavioral fingerprinting is not a magical solution to LLM API model verification. It does offer a cheap, black-box signal for model drift and serving-path changes that clients have historically struggled to observe.

A model can be instructed to report almost any name. Completely hiding its selection habits appears to be harder.

Frequently asked questions

Can a model really be identified from one token?

Not from one request. The output of each request can be approximately one token long, but verification requires distributions collected over many requests and probe conditions.

Why not ask the model for its name?

Self-reported identity can be changed by the system prompt or training data, so it is not technical proof.

Would temperature 0 make comparison easier?

Temperature 0 often fixes an answer, but the study observed cases where different providers produced different fixed answers for the same model. The main fingerprint uses sampled distributions at temperature 1.0.

Can this method prove that an API is fake?

No. It demonstrates a statistical difference from a reference distribution. Model updates, quantization, and serving configuration are among several possible causes.

Can I add this to my own application?

Yes, as an auxiliary signal for drift monitoring or provider comparison. Establish repeated baselines in your own environment and combine the signal with quality tests and operational logs.