Automation & AIModule 1: AI without the mysteryLesson 3 of 15
Course progress13%
18 min lesson · Updated August 2026
Why can AI give confident wrong answers?
Generative AI can produce confident false or internally inconsistent material because it generates plausible patterns rather than guaranteeing truth; verification must match the consequence of error and use independent authoritative evidence.
A fluent AI draft separates into factual claims; each claim passes through source, date, scope and consequence checks before low-risk approval or human expert escalation.
Hallucination is often called confabulation
NIST uses “confabulation” for confidently presented erroneous or false generated content. It can include invented facts, citations, calculations, quotes, tool results, reasoning or contradictions.
The output may sound certain because style and factual reliability are separate. Asking the model whether it is sure is not independent verification.
Why false output happens
Cause
Effect
Probabilistic generation
A plausible continuation can be wrong.
Missing/ambiguous context
The model fills gaps or chooses an unsupported interpretation.
Stale or incomplete knowledge
Recent or niche facts may be absent or distorted.
Retrieval failure
Wrong, partial or conflicting source passages enter the answer.
Tool/data error
A correct explanation may be attached to a failed or malformed action.
Adversarial input
Prompt injection or manipulated sources steer output.
Verify claims, not prose tone
01
Extract each material claim
02
Identify source type needed
03
Open original source
04
Check date, jurisdiction and definition
05
Confirm calculation/data
06
Seek independent expert review if high impact
07
Correct or remove unsupported claim
08
Record evidence and approver
Scale review with consequence
Risk
Example
Low
Brainstorming internal workshop titles; obvious human selection may be enough.
Moderate
Public product explanation; check every fact against approved documentation.
High
Medical, legal, financial, safety or employment decision; qualified human review and stricter controls are required.
Action-taking
Refund, deletion, payment or account change; validate inputs, permissions, limits and confirmation before execution.
Citations can also be invented
Open the link or source, confirm it exists, and verify that it actually supports the sentence. A real-looking journal title, DOI or quotation can still be fabricated or misapplied.
When retrieval provides citations, test whether the cited passage is relevant and complete. Citation presence is not proof of correctness.
Human review can fail too
People may accept fluent output too quickly—automation bias—or only check claims they already doubt. Give reviewers a checklist, enough time, domain access and authority to reject output.
Measure reviewer agreement and escaped errors. If staff cannot reliably verify the work, the task may be unsuitable for generation.
Design for uncertainty
Source-bound answers
“Not enough information” allowed
Confidence not presented as probability unless calibrated
Explicit dates and jurisdiction
Calculations performed by reliable tool
Quoted evidence for critical fields
Escalation path
No automatic high-impact action
Logs and sampled audits
Stop/rollback control
Real-world example
Example: a fabricated regulation citation
Example
A model confidently cites a regulation that does not exist. The reviewer searches the regulator’s official database, finds no match and removes the advice. The workflow is changed so future legal claims require a supplied official source and qualified approval.
Try this
Run a claim audit
Generate or choose one AI answer with five factual claims. Build a table of claim, source needed, actual source, date, limitation and decision. Count how many survived unchanged.
Common questions
Questions beginners ask.
What is an AI hallucination?
A generated statement or output that is false, unsupported or inconsistent while often sounding confident; NIST uses the term confabulation.
Why does AI invent citations?
It can generate plausible-looking text patterns without retrieving or validating a real source.
Can asking “are you sure?” verify an answer?
No. Verification requires independent evidence or reliable tools, not another ungrounded response.
Does RAG eliminate hallucinations?
No. Retrieval can improve grounding but can retrieve wrong, incomplete, stale or malicious material.
What is automation bias?
A human tendency to accept automated output too readily, especially when it appears confident or convenient.
How much review is enough?
Review depth should scale with consequence, uncertainty and ability to detect errors.
Can an AI confidence score be trusted?
Only when its meaning and calibration are validated for the use; fluent self-confidence is not a measured probability.
When should AI output be rejected entirely?
When material claims cannot be verified, the use is prohibited, risks exceed controls or no accountable reviewer can assess it.
