How to verify an AI tool announcement before you trust it

An AI announcement tells you what its publisher wants the market to notice. It rarely contains everything a buyer, builder or operator needs to make a decision.

A launch post has limited space and a commercial purpose. It may describe a research result, a model, a product and an eventual rollout in the same page. Headlines and summaries then compress those distinctions further.

The useful question is: what, precisely, can I rely on today?

The following method can be completed quickly for a news brief or used as the first stage of a procurement review. NIST’s Measure guidance informs its emphasis on context, defined metrics and documented limitations. The OECD AI principles inform the checks on transparency, robustness and data handling. The result is a small evidence record that can support a decision.

1. Find the primary source

Start with material closest to the work: vendor documentation, a model or system card, a public repository, a standards document or the research paper. Use social posts and launch videos to find the claim, then move to the source that defines it.

Look for more than one first-party page. The announcement explains the intended story; the documentation usually explains how the product behaves; the pricing, limits and security pages explain the operating conditions. Release notes can show whether an apparently new capability has been available in preview for months.

Record the URL, publication date and the date you checked it. Save a copy or note the relevant version when the decision matters. Product pages can change after launch, leaving a later reader unable to reconstruct what was known at the time.

For research claims, inspect the comparison method. Was the result peer reviewed? Is the paper a preprint? Are the code, data and evaluation prompts available? A vendor’s own benchmark can still be informative, but the article should attribute it as a vendor result rather than an independent finding.

2. Confirm actual access

“Available” can mean generally available, public preview, private preview, waitlisted, limited to one country, restricted to an enterprise plan or accessible only through an API. Those differences determine whether a reader can reproduce the result.

Check:

eligible countries and languages;
account and payment requirements;
plan, licence or cloud-region restrictions;
whether access is immediate or waitlisted;
API and application availability;
quotas and staged rollout dates; and
whether the documented version is the one the product currently serves.

Do not report access from the presence of a button alone. Sign-in gates, region checks and account entitlements often appear later. If you cannot test access, say what the publisher states and make the limitation visible.

Access also has a date. “Unavailable” on launch morning can become inaccurate the next day, while a preview may close. Time-stamp the finding so the article remains honest after the rollout changes.

3. Separate the model from the product

A model is one component of a service. The application around it supplies system instructions, memory, retrieval, file parsing, safety controls and tools. An API may expose settings that the consumer product hides. Enterprise and individual plans may use different data controls even when the model name is identical.

When a company announces a capability, identify the layer:

a research technique demonstrated in a paper;
a model that accepts a particular input or produces a particular output;
an API feature available to developers;
an application feature exposed to end users; or
an agent product combining a model with tools and permissions.

Do not assume the capability travels between layers. A large context window in an API does not establish that a chat application accepts a file of the same size. A benchmark result for a base model does not describe a product with retrieval and safety filters. A tool shown in a controlled agent demo may not be available to outside developers.

This is also why model selection should happen in the intended product surface. The model chooser explains how to compare the complete configurations a team could actually deploy.

4. Read the limits before the highlights

Limitations determine the practical meaning of the headline. Search the documentation for rate limits, file sizes, supported formats, context limits, retention, excluded use cases, preview conditions, known failures and deprecation dates.

Pricing deserves the same care. A quoted input-token price may exclude output, reasoning, search, storage or tool charges. A free preview does not reveal production cost. Estimate the whole task, including retries and human review, rather than multiplying the lowest visible price by an optimistic token count.

Pay attention to the comparison baseline. “Twice as fast” needs the earlier system, hardware, workload and quality target. “State of the art” needs the benchmark version, evaluation settings and competing systems. If the baseline is missing, preserve the claim as a claim and do not complete it with an assumption.

Known limitations are not footnotes to hide at the end of an article. A browser tool that cannot handle authenticated sessions may still be useful for public research, but it is a different product from one that can complete a staff workflow behind a login.

5. Reproduce a narrow claim

Choose a task with an observable result and a clear boundary. “It feels intelligent” is not reproducible. “It extracted the five required fields from this public sample and returned valid JSON” is.

Before testing, write down:

the claim being checked;
the exact input and expected evidence;
product, plan and version;
date, region and interface;
settings and tools available; and
pass, fail and partial-success conditions.

Keep the input and raw output. Run the task more than once when the system is probabilistic. Include one awkward case rather than selecting only the demonstration that suits the feature.

A single success shows possibility, not reliability. A small review can establish that a feature exists and appears to work for a stated example. Production-readiness claims require a representative evaluation, including failures. The twenty-case evaluation method is a practical next step.

Be careful with hands-on language. A playground test is hands-on evidence for that surface only. A repeated vendor benchmark remains a vendor benchmark. Testing five URLs supports a conclusion about those requests, so the scope of the prose should match them. The Kitesurf review is a worked example of these boundaries.

6. Check data handling and authority

Do not put client records, unpublished work or personal information into a new service merely to test it. Use public or synthetic material until the plan-specific terms and configuration have been reviewed.

Establish:

whether prompts, files and outputs are retained;
whether they may be used to train or improve models;
retention periods and deletion controls;
processing regions and subprocessors where relevant;
access logging and administrative controls;
differences between consumer, business and API plans; and
what permissions an agent or connector receives.

An agent deserves extra scrutiny because it can act. Identify the tools it may call, the data each tool exposes and the actions that require human approval. Retrieved web pages and documents are untrusted input; instructions inside them should not acquire authority over the host system.

Security badges and enterprise labels are leads for investigation, not substitutes for the applicable contract and configuration. If the required information is not public, mark it as unresolved.

7. Publish what remains uncertain

Good coverage separates three things: what the source says, what the writer observed and what remains unknown.

Use direct labels:

Documented: the current documentation states the feature or limit.
Observed: a dated test produced a particular result.
Inferred: the evidence suggests a consequence that has not been tested directly.
Unknown: pricing, access, behaviour or policy could not be confirmed.

This language makes an article easier to update. It also helps readers decide whether the evidence is sufficient for their own risk level.

An evidence record can be compact. The entries below are illustrative:

Field	Example entry
Claim	Returns rendered HTML after client-side JavaScript
Primary source	Documentation URL and version
Access	Public playground, checked 30 August 2026
Test	Known client-rendered page, expected control present
Result	Passed in two runs
Limits	Preview service; no authenticated-session test
Confidence	Feature exists; production reliability not established

Before publication, check every number, date, availability statement and comparative adjective against the source. Link to the precise page rather than a company homepage. Remove claims that are only repeated across secondary articles without a traceable origin.

Decide what the evidence permits

The conclusion may be “test this now,” “watch the rollout,” “use only for low-risk work” or “there is not enough information yet.” Most announcements do not require an immediate purchase or migration.

For readers, the best next action is specific and proportionate: open the documentation, test one representative case, review a data policy, compare the complete cost or wait for general availability. For publishers, the obligation is equally practical: update the article when a material fact changes and retain the date of the original observation.

Sources

Last reviewed: 30 Aug 2026. See our editorial policy for how we test, source and correct our work.