Why human-like AI fails without quantitative grounding

This article compares the two classes in terms of training data, learning tasks, and measurable error. Then it presents published research on how each class behaves when asked to estimate a physical quantity. The final sections cover the circumstances under which physics AI itself degrades.

This article is for CAE, CFD, and FEA engineers assessing whether an AI tool fits a verification workflow; engineering program managers under cost and delivery pressure; and Executives setting R&D direction.

The consequence of quantitative grounding is measurable in engineering hours. Committing to tooling requires a stated error bound on the prediction. Without such a bound, verification costs about as much analysis effort as producing the result with conventional simulation, and the net saving approaches zero.

In theory, two classes of AI systems could compete for engineering work.

• Conversational AI. Trained on text written by humans about physical systems. The class includes large language models and vision-language models.
• Physics AI. Trained on field values computed by engineering simulation. The class includes models built on computational fluid dynamics and finite element datasets.

We show that the second wins.

Two companion articles frame what follows. Neural Concept sets out the ceiling of text-trained systems by analyzing the limits of generative AI and separating the two model families by what each is trained to produce, comparing predicting physics with describing it.

What does quantitative grounding mean for an engineering AI model?

Quantitative grounding describes the source of the training data. The format of the model’s answers is a separate matter. A pressure value reported to four decimal places does not tell a reader how it was obtained.

Three conditions separate a grounded artificial intelligence model from an ungrounded one, and each has an established name.

1. Ground truth. Training examples contain numerical values produced by a computer simulation or physical measurement, not by a person describing a result.
2. Held-out test set. Set aside and never shown during training. The model is then asked to predict them. The model error is the difference between the prediction and the simulated or measured values.
3. Applicability domain. The training data covers a range of designs and operating conditions. The documented range then determines whether a new design falls inside it.

Remarks:

• All three conditions are independent, so each must be satisfied on its own.
• Model size is irrelevant here because parameter count and training budget govern capacity, while the three conditions govern the source and the traceability of the training values. Current LLMs satisfy none of them at any scale.

For decades, engineering has required verification and validation. Before finalizing a design, engineers compared a calculated result with an exact solution, a result from a finer grid, or experimental data. Applying the same standard to a trained model extends this established practice.

Neural Concept explains how these practices come together to produce a working capability in its section on engineering intelligence for product development.

Artificial Intelligence in engineering: two classes of AI systems

Artificial Intelligence systems are usually presented as belonging to a single category, even though their training data differ almost entirely and span major fields such as machine learning, deep learning, natural language processing, computer vision, and robotics.

Conversational AI is trained on what humans have written about physical systems, while Physics AI is trained on “hard” data from numerical simulations or laboratory tests of those systems.

Which data set trains each class?

• Conversational AI. The training signal consists of human-written sentences about physical systems. Conversational models are trained to predict the next word in a sequence of human-written text. In contrast, vision-language models apply the same objective to images and videos with human-written captions. In each case, the training signal comes from what humans have written about the world. The term “generative AI” refers to this type of system, since widely used generative models produce text and images by learning patterns from human-authored material.
• Physics AI. Physics AI is trained on a different signal altogether. Each training example pairs a 3D geometry and a set of operating conditions with the field values a simulation computes for that geometry. In the vehicle aerodynamics pressure example, the model obtained pressure at every point on the vehicle surface by solving the equations of fluid motion on a computational grid.

Which engineering problems does each class solve?

Each class performs its own task and fails at the others’. A conversational model processes text about pressure fields; a physics AI model computes them. Only the second can return one for a new geometry.

• Conversational AI handles language and perception tasks, such as natural language processing and speech recognition. Current models also function as few-shot learners on tasks not present in their training data. Reinforcement learning from human feedback refines their outputs. An image classifier trained on labeled photographs reaches recognition accuracy that handwritten rule sets have never matched. Tech companies build generative AI models mainly to optimize for breadth of demand, which suits a consumer product and fits a numerical tolerance poorly. Sectors carrying formal certification duties, such as aerospace and national security programs, apply acceptance criteria that these models were never built to meet.
• Physics AI handles field prediction for designs similar to its training data. Supervised learning on simulation output produces a model that returns pressure, stress, or temperature at every point of a new geometry. The error of that prediction is measurable in physical units.

Can conversational models learn structure?

Conversational models can learn structure beyond statistical association. Gurnee and Tegmark concluded that the neural networks in the Llama-2 family, a widely used LLM, contain the basic components of a world model.

A world model is an internal representation of how the external world is arranged, in this case, where places are located and when events occur. The authors reached this conclusion by probing the network, which encodes space and time along straight lines, with individual neurons responding to specific coordinates.

• Reference: Gurnee, W. and Tegmark, M. (2024). Language Models Represent Space and Time. International Conference on Learning Representations. arXiv:2310.02207.

The question has a long history. Two decades before the current era of Machine Learning and LLMs, research on embodied cognition argued that thought is constrained, and at the same time enabled, by the body’s morphology.

• Reference: Pfeifer, R. and Bongard, J. (2006). How the Body Shapes the Way We Think: A New View of Intelligence. MIT Press.

In engineering, the question is narrower than the debate over human intelligence, and the training data settles it.

A world model based on human descriptions records what humans say about physical systems. It does not include the pressure field over a particular vehicle at a particular speed because the training text contains no such field. The model has learned the distribution of drag statements and can reproduce them convincingly. The training text does not mention drag.

Table 1. Comparison of conversational AI and physics AI across 2 solver families. Neural Concept examines the same separation at the design-task level in its analysis of generative AI’s limitations in design work.

The same comparison distinguishes the two model classes by what each is trained to produce in engineering contexts.

How can Conversational and Physics AI work together?

The two AI classes don’t need to compete for the same task. Conversational models handle natural-language intent, geometry description, and design dialogue. Physics AI supplies the quantitative field predictions that sit behind those requests.

Design Copilots

Neural Concept’s AI Design Copilot illustrates the division of labor and exemplifies top applications of AI in engineering, such as design optimization, digital twins, and workflow automation. An engineer issues a natural-language instruction; the system maps it to geometry and operating conditions, then returns CAD-ready models and multiphysics predictions whose errors have already been measured on held-out simulation data. The language layer expresses the goal; the physics layer enforces conservation laws and reports a calculable residual.

In this arrangement, each component stays within its competence. The conversational front-end never invents a pressure value. The physics model never interprets free-form English. The combination therefore inherits the grounded model’s measurable error bounds while retaining the usability of ordinary engineering conversation.

Why the difference is a regression problem

The two classes also perform different mathematical tasks, which makes the output checkable.

• Conversational AI performs classification. At each step, the model selects one word from a fixed vocabulary, making its error discrete: the selected word either matches the reference text or it doesn’t. A word is either right or wrong, so the size of the error is undefined.
• Physics AI performs regression to predict continuous values at every point in a 3D geometry. Its error is a signed quantity in physical units. A predicted surface pressure that differs from the simulated value by a stated number of pascals is off by exactly that amount. Mapping the error over the vehicle surface then shows where the model performs worst.

The two tasks are written as follows.

Conversational AI: ŵ = argmax P(w | c), w ∈ V

Physics AI: p̂ = f(G, B), p̂ ∈ ℝᴹ

DrivAerNet++ experience

One vehicle design yields hundreds of thousands of supervised data points. DrivAerNet++, a public automotive dataset, provides the figures.

DrivAerNet++ contains 8,000 car designs, each modeled using high-fidelity computational fluid dynamics simulations. The published dataset exceeds 39 TB and has enabled state-of-the-art aerodynamic prediction with Neural Concept’s Geometric Regressor on DrivAerNet++.

• Reference: Elrefaie, M., Morar, F., Dai, A. and Ahmed, F. (2024). DrivAerNet++: A Large-Scale Multimodal Car Dataset with Computational Fluid Dynamics Simulations and Deep Learning Benchmarks. Advances in Neural Information Processing Systems 37, Datasets and Benchmarks Track. arXiv:2406.09624.

Each DrivAerNet++ design includes a surface pressure field defined at more than 500,000 points, along with velocity and wall shear stress fields. The field error is reported in pascals as the root-mean-square over those points.

RMSE = √[ (1/M) · Σᵢ₌₁ᴹ (p̂ᵢ − pᵢ)² ]

Here, pᵢ is the simulated pressure at point i and p̂ᵢ is the predicted pressure at the same point.

Quick facts:

• One vehicle design supplies hundreds of thousands of supervised values
• Every one of those values has a computed reference, leaving a wrong prediction at any point detectable
• The values satisfy the conservation laws enforced by the simulation, whereas a corpus of text is not subject to any comparable constraint.

The density of supervision explains why model performance in engineering can be reported as a distribution. Neural Concept describes how to assemble such a training pipeline in its overview of AI simulation for engineering and expands on AI in engineering, its applications, key benefits, and future trends.

What does conversational AI do when asked for a physical quantity?

Conversational AI returns a typical value for the object in question. In the output, a recalled value and a computed value are indistinguishable.

Researchers at Stanford measured how well vision-language models in the same class estimate physical quantities from video. Their benchmark, QuantiPhy, contains over 3,300 video-and-text items, each with a reference value computed from the scene. The model receives one object property, such as its length, and is asked for another, such as its speed or width.

One benchmark test isolates measurement by eliminating visual input.

• Reference: Li, P., Xiang, T., Mao, E., Wei, S., Chen, X., Masood, A., Li, F.-F. and Adeli, E. (2025). QuantiPhy: A Quantitative Benchmark Evaluating Physical Reasoning Abilities of Vision-Language Models. arXiv:2512.19526.

The researchers provided a textual prompt stating that the yellow car in the road scene was 5.67 meters long and omitted the video.

The speed question functions as a control. Without video, the model returned a speed far from the reference value. The result confirms that the ablation removed the motion information as intended. The width question was therefore answered under the same conditions.

The width question produces the following finding. Under identical conditions and without available pixel data, the model returned a width score of 0.7 on the benchmark’s mean relative accuracy metric. Mean relative accuracy averages, across a set of error thresholds, are an indicator of whether the relative error falls below each threshold.

MRA = (1 / |C|) · Σ 𝟙( |ŷ − y| / |y| < 1 − θ ), θ ∈ C

Here,

• y is the reference dimension,
• ŷ is the predicted dimension,
• C is the set of thresholds. The scale runs from 0 to 1, so 0.7 denotes a prediction close to the reference dimension.

The authors attribute the width score to the model’s learned prior over typical car dimensions, the only remaining source of information. They further propose that the same prior may account for a substantial portion of the model’s size estimates, even when video is provided. The paper does not quantify that proportion. In an interview about that work, Adeli outlined the general pattern, noting that on physics questions, the AI gives answers that at first seem plausible but, upon closer examination, turn out to be little more than guesses.

• Reference: Myers, A. (2026). AI Can’t Do Physics Well, and That’s a Roadblock to Autonomy. Stanford Institute for Human-Centered Artificial Intelligence, 26 January 2026.

The same study reports two further findings.

• Complex scenes raised the scores. Current models performed better on visually complex scenes, where accurate object detection is harder, and guesses are more likely to pass.
• Contradictory premises were overridden. The prompt stated that a car in the scene measured 6,000 meters. The models left their width estimates unchanged because the learned prior took precedence over the stated premise.

The QuantiPhy results have a narrow scope. The benchmark evaluates vision-language models that estimate kinematic quantities from video without computing a reference at any stage. A model trained on a simulation dataset has a computed reference for each example. Therefore, the same error is measurable on held-out designs and would be detected before deployment.

Why AI hallucination is a scoring problem before it is a physics problem

People often attribute AI hallucination to a lack of connection with reality. The published account is more precise. Researchers at OpenAI and Georgia Tech identify two causes. Training and evaluation procedures reward guessing over acknowledging uncertainty. Hallucinations begin as binary-classification errors produced by ordinary statistical pressure during pretraining.

• Reference: Kalai, A. T., Nachum, O., Vempala, S. S. and Zhang, E. (2025). Why Language Models Hallucinate. arXiv:2509.04664.

The mechanism operates through the scoring rule. Under a scheme that awards one point for a correct answer and zero for a declined one, a model that always guesses outscores one that sometimes declines. Most benchmarks score accuracy or pass rate. Optimization toward those benchmarks, therefore, selects for confident answers to unanswerable questions. The same analysis notes that minimizing the training loss during pretraining yields well-calibrated probabilities, meaning the model’s stated confidence matches its actual accuracy. The calibration breaks down afterward, under evaluation rules that penalize any expression of uncertainty.

Retrieval-augmented generation (RAG) is frequently proposed as a remedy, inserting authoritative documents into the model’s input so that answers are grounded in retrieved text. For questions answerable from a document, it works. For a pressure field over a new vehicle geometry, the document exists only after a simulation has produced the field.

A model can be built to decline cases outside its applicability domain, and you can score the decline. Read this way, AI safety in an engineering setting becomes a measurable property of the deployed system. Neural Concept addresses how to set such limits within an existing simulation workflow in its guidance on AI for computer-aided engineering.

Which failure mode does quantitative grounding leave untouched?

Representativeness and measurement are separate properties of a data set, and quantitative grounding addresses only the second. Representativeness governs whether a skewed sample produces biased algorithms, a question studied mainly in the social sciences and in human rights law. Measurement governs whether a reported number was computed at all. A model may reproduce its training distribution exactly and still report a pressure value it never computed. Engineers who fully understand this distinction recognize that the two checks answer different questions, so that more data corrects one while leaving the other untouched. AI initiatives that merge them audit the wrong property.

Why does transfer to new environments break down?

Transfer breaks down when a model encounters a new setting without any examples from that setting. The abstract rule learned elsewhere does not survive a change in context.

Current models generally have to relearn a new setting before prior structural knowledge helps them, whereas human learners apply that knowledge on their first attempt.

Engineering work rarely repeats a solved case. A new program brings a new geometry, revised material specifications, or operating conditions the previous program did not cover. The useful measure is therefore model performance under shifted conditions vs. performance on familiar cases.

Xiang and colleagues measured the effect at the 2026 Cognitive Science conference, using a task in which participants must first deduce a hidden rule and then apply it in a modified setting. Even successful models required an initial mapping specific to the new environment before they gained efficiency. The authors call this mapping environmental grounding: the model relearns the new setting’s particulars before its prior knowledge becomes useful.

• Reference: Xiang, L., Ma, Y., Cao, Z., Zhu, Y., and Zhu, S.-C. (2026). Grounding Before Generalizing: How AI Differs from Humans in Causal Transfer. Cognitive Science Society. arXiv:2604.24062.

Human learners apply the rule immediately, without that relearning step. An engineering program cannot always provide the examples the model would need. When a company enters an unfamiliar operating range, it must choose a first-of-its-kind design, so such examples do not yet exist. Under real-world conditions, the decision cannot wait for the second design.

One result from the Cognitive Science study runs counter to the simple narrative. In the text-only condition, the models matched or exceeded human discovery efficiency. Adding visual information in both the image-only and the text-and-image conditions degraded overall performance. The authors read this as evidence that models rely on symbolic processing.

Current models perform well within a symbolic channel but struggle to apply an abstract rule to new environments without first relearning the setting. Because structural change across programs is the norm in engineering, any deployment estimate must include the cost of that relearning.

Read how to manage costs across successive design cycles in the AI-driven product development optimization approach.

Table 2. Four studies on whether models measure or recall

How is quantitative grounding verified?

Quantitative grounding is verified by calculating model error in physical units on designs excluded from training, then reporting how often the stated error range contains the simulated value.

Model error is evaluated directly on data withheld from training; the dataset is partitioned (split) into two subsets:

• The training set contains examples the model learns from. Typically, this is the largest subset.
• The test set is withheld entirely and used only once to report the quoted error. It constitutes the held-out data.

DrivAerNet++ was split into 5,819 designs for training and 1,154 for testing, with a further subset reserved for validation, yielding a training fraction of approximately 73%. A validation subset is used to select hyperparameters and to determine when training stops. Its use is optional. In its satellite thermal study, Neural Concept used a default configuration, carried over hyperparameters from earlier datasets rather than tuning them for this case, and reported L1 relative error separately on the training and test sets.

Average accuracy is insufficient on its own, since a model that predicts drag within 2% across a test set may still fail badly on the one vehicle geometry a program depends on. Two further measures address this.

Brier and ECE scores

Brier score. Evaluates a probabilistic prediction by the mean squared difference between the stated probability and the outcome, penalizing confident errors more heavily than hedged ones

BS = (1/N) · Σᵢ₌₁ᴺ (cᵢ − oᵢ)²

In the Brier score,

• cᵢ is the stated confidence for prediction i
• oᵢ is the outcome, taking the value 1 when the prediction is correct and 0 otherwise.

Expected calibration error. Compares stated confidence with observed accuracy, revealing whether a model reporting 90% confidence turns out correct approximately 90% of the time

ECE = Σₘ₌₁ᴹ (|Bₘ| / N) · | acc(Bₘ) − conf(Bₘ) |

In the calibration error,

• predictions are grouped into M confidence bins,
• Bₘ is the m-th bin
• acc and conf are the accuracy and the mean stated confidence, respectively.

A perfectly calibrated model returns ECE = 0.

• References:
  • Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review 78(1), 1–3.
  • Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. International Conference on Machine Learning.
  • The binned estimator follows Naeini, Cooper, and Hauskrecht (2015).

Field prediction and use case

For field prediction, the same logic requires that the predicted range include the simulated value at the specified frequency. The measured quantity is the empirical coverage.

Ĉ = (1/N) · Σᵢ₌₁ᴺ 𝟙( yᵢ ∈ [ ŷᵢ − δᵢ , ŷᵢ + δᵢ ] )

Here, δᵢ is the half-width of the range stated for design i.

A model whose stated 95% confidence intervals contain the simulated value in only 70% of test designs yields Ĉ = 0.70 for a nominal 1 − α = 0.95. The gap between the two figures quantifies the overconfidence.

Improving accuracy, therefore, becomes a measured activity. The trade-off is explicit. A wider predicted range contains the reference value more often and supports a weaker decision.

Reporting the error as a distribution over held-out designs, rather than as a single average, shows how often the large deviations occur. The figure plots the distribution of the L1 relative error for a thermal prediction case in a study of design under thermal constraints.

Comparison with simulation is one link in the chain. Comparison with physical measurement is the other. This survey compares how available systems report their accuracy in AI tools for mechanical engineers.

Fluid dynamics problems illustrate the pattern most directly, since a single converged simulation provides a complete surface pressure field for each training geometry. Neural Concept explains how it generates those results and what they contain in its introduction to computational fluid analysis and its applications.

Across an R&D portfolio, the effect compounds

Each completed simulation campaign enlarges the training dataset available to the next program. New data from one program improves the next, creating a self-improvement cycle for the training corpus across programs, provided the organization retains access to both the data and the model weights.

Read how organizations accumulate that asset in their assessment of AI in research and development. The principle of AI Ownership then determines whether the organization retains control of the resulting model.

Where human-in-the-loop oversight still applies

Even with grounding in place, design engineers retain several responsibilities that measurement cannot discharge.

• Judging the domain. Deciding whether a new design falls within the applicability domain, a judgment requiring knowledge of the physics
• Reading the margin and interpreting a predicted margin in context. A 5% error on a thermal margin matters more in a battery enclosure than in a bracket.
• Diagnosing disagreement. Deciding whether a gap between prediction and expectation indicates model failure or a mistaken expectation

Human insight applies at the interpretation stage. Human oversight is justified only when you can identify, in advance, the cases that require scrutiny. The reported confidence value provides that signal.

Oversight of a grounded model differs from oversight of an ungrounded one. A model that cannot report its own error forces the reviewer to reconstruct the answer independently. When examining the output of a well-calibrated model, two checks are required: first, that the design lies within the model’s applicability domain, and second, that the stated error is appropriate for the decision in question.

Extending the applicability domain requires control of the model. An organization that cannot examine the training data or retrain the model on its own designs cannot extend it. Neural Concept sets out the consequences of this dependency in its position on building enterprise AI without borrowed access.

Limits of physics AI

A model trained on simulation data can also return a field that looks right but is wrong. The conditions are worth stating plainly.

1. Prediction outside the applicability domain. A model trained on vehicles at highway speeds contains no information about flow behavior near the speed of sound. Predictions at those speeds extrapolate beyond the physics. The output arrives in the same format and with the same apparent confidence as a prediction from within the domain.
2. Novel geometry. A model trained on variations of a single vehicle architecture may perform poorly on a design that introduces a feature absent from all training examples, such as an additional cooling duct or a new structural load path.
3. Propagated simulation error. When training simulations model turbulence in a way that doesn’t suit the flow regime, the trained model faithfully reproduces that error. Agreement with a simulation does not prove agreement with physical reality.
4. Coarse training grids. A model inherits artifacts from the computational grids used to produce its training data, and those artifacts remain invisible in the reported error because the test designs were computed on the same grids.

The consequence scales with the application. During early screening, a prediction outside the applicability domain costs a wasted design iteration. In a certification-relevant load path, the same prediction can lead to catastrophic outcomes. Therefore, document the domain before the model informs any irreversible decision.

The four limits above define the technology’s role. A well-calibrated model that reports its confidence and declines designs outside its applicability domain accelerates the design cycle’s exploration phase, after which conventional simulation and physical testing verify the selected candidates. Neural Concept examines how that division of work is changing as it discusses the future of engineering decision-making.

Advanced topics: uncertainty estimation and data fusion

Two capabilities extend a grounded model beyond its fixed applicability domain.

Uncertainty estimation lets the predictor enter an unseen zone of the shape space and report how far it can be trusted there. The engineer receives a confidence value alongside the field and returns the doubtful designs to the solver. Neural Concept presented a low-cost method at ICML that uses convergence rate as a proxy for uncertainty and sets out the industrial case for uncertainty quantification in CAE.

Data fusion trains a single model on multiple sources simultaneously. An organization with 40 years of wind tunnel data and 20 years of CFD results can combine them into a single training dataset, allowing measured and computed references to correct each other.

• Reference: Durasov, N., Oner, D., Donier, J., Le, H. and Fua, P. (2024). Enabling Uncertainty Estimation in Iterative Neural Networks. Proceedings of the 41st International Conference on Machine Learning, PMLR 235:12172-12189. arXiv:2403.16732. Computer Vision Laboratory, EPFL, and Neural Concept SA.

The training data governs AI development for engineering. A model trained on human descriptions returns a value whose accuracy has never been measured. A model trained on simulation results returns a field with a measured error on a held-out test set and a recorded applicability domain. The implications for procurement are direct: you can rigorously check only the second value before a program commits to it.

Responsible deployment requires three answers, enough to turn a plausible value into decision-making evidence, as outlined in the table.

Table 3. Three questions before deployment