The Importance of Uncertainty Quantification for Deep Learning Models

When issuing a prediction, one can question how much human or artificial intelligence is sure about the prediction. Without going into detail on the dramatic consequences of overconfident artificial agents in such situations as Autonomous Driving, this article will focus on the Neural Concept’s core competence: building predictive models that are surrogates of more resource-intensive CAE models.

CAE simulations are physics-based and solve numerical equations on computational domains, mostly of industrial interest (CAD shapes). We will assume that CAE yields a reasonable estimation of a real life behaviour and can therefore be assumed as “Ground Truth” (otherwise, it can be supplemented by experimental data). Similarly to the human brain, Deep Learning data-driven models learn by seeing examples and extracting information from these results. In our case, learnt examples are from CAD or various inputs (materials, boundary conditions etc.) and learnt results are CAE (or experimental). Hence, when facing a new unknown example, the model will predict the result in real-time, based on the data used for its training.

In general, when issuing a prediction, one can question human or artificial intelligence on how much she/it is sure about the prediction. The same is true for our Neural Network models.

The Deep Learning implementation of Neural Concept Shape is a tool to build and support surrogate models that produce outputs. The surrogate model (trained neural network) undergoes extensive training and testing phases, having measurables such as I1 (the mean average error) and R2 (the coefficient of determination). However, an R2 is a sort of “backwards-looking” measurable on the training/testing phase. During the neural network deployment phase, engineers would like to have a “dynamic” way to assess if the prediction is reliable. Suppose the neural network issues a signal of uncertainty above a certain predefined level. In that case, the human or artificial agent could decide to activate remediations such as submitting the neural network to new training (for efficiency, starting from the previous neural network and using Transfer Learning technology).

Therefore, Neural Concept has also worked extensively on uncertainty estimation, to help engineers during deployment phase facing epistemic uncertainty (degree of variation due to lack of knowledge on the model we are trying to predict). As we can see from the example given with figure 1, after uploading a given CAD geometry (here called “geometry_0001.stl”), the engineer receives from the neural network real-time predictions on the values of interest together with a confidence index from the model.

Another typical application is generative design. In this case, Neural Concept Shape is an agent that may create geometric shapes far out of the initial design envelope. We wish to know if the predictions associated with the new generated shapes are still reliable, or more input/output samples that one needs for neural network re-training.

Figure 1: Application example of Neural Concept Shape for non-AI specialists (NCS /Lite), with a trained neural networkembedded in a vertical application. The user can upload geometries directly from its CAD software and instantly predict thevalues of interest for the analysis and the degree of confidence (in this case 97%).
Figure 2: Using the uncertainty feature from Neural Concept Shape, we can sort the test samples by the confidence indexgiven by the model, from lowest to highest. We show then the cumulative coefficient of determination R2 based on the sortedsamples.

Masksembles: A New Methodology to Compute Uncertainty in Prediction

Neural Concept’s staff collaborates in research topics on top of available software capabilities, mainly with EPFL (Lausanne – Switzerland). This final section will report some activity carried out on a novel promising methodology called Masksembles.

Masksembles contains in itself the explanation of its technology, as follows more detailed:

An “ensemble”, in general, is a set of virtual copies of anything (in physics, a vessel under pressure; in AI, a neural network; in society, a collection of individuals) where the extension over multiple copies allows for several different physical states or network configurations. In particular, we are interested in epistemic uncertainty.

The “Deep Ensembles” technique consists of training an ensemble of deep neural networks on the same data with random initialization of each of the neural networks in the ensemble. By running all neural networks aggregating their prediction, one obtains the best in class uncertainty estimation at the cost of computational investment.

A “mask” is, in deep learning technology, a way to drop/hide artificial neurons, thus generating several slightly different model architectures, allowing a single model to mimic ensemble behaviour.

Masksembles can generate a range of models within which MC-Dropout and Deep Ensembles are extreme cases. It joins MC-Dropout’s light computational overhead and Deep Ensembles’ performance. Using many masks approximates MC-Dropout while using a set of non-overlapping, completely disjoint masks, yields an Ensemble-like behaviour.

Figure 3: The prediction task here is to classify the red vs blue data points drawn in the [−5, 5] range from two sinusoidalfunctions with added Gaussian noise. The background colour depicts the entropy assigned by different models to individualpoints on the grid, low entropy represented by blue and high entropy represented in yellow. The figure ranges from (a)Single model to (b) – (e) Masksembles models with different parameters and finally (f) – Ensembles model. For high mask-overlap values, Masksembles behaves almost like a Single Model, compare (a) and (b) but starts acting more and more likeEnsembles as the mask-overlap value decreases, compare (e) and (f).

Conclusion

Neural Concept is already providing the possibility, even for non-specialists, to have confidence levels for predictions. The type of advanced research exposed in the last section will bring further benefits to engineers in terms of high performance and low computational costs.

Thus, we will continue to sustain engineers facing an always important question – “am I sure about my predictions”?

Bibliography

Nikita Durasov, Timur Bagautdinov, Pierre Baqué, Pascal Fua (Computer Vision Laboratory, EPFL and Neural Concept): «Masksembles for Uncertainty Estimation», arXiv:2012.08334v1 [cs.LG], 15 Dec 2020

About the author
Thomas von Tschammer
Thomas joined the team in 2018 as Director of Operations, aiming to empower engineers with next a next generation Deep-Learning tool dedicated to CAD and CAE.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
Luca Zampieri
Luca joined the Application team in 2018, aiming to build the next generation Deep-Learning tool dedicated to CAD and CAE.
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
Anthony Massobrio
Anthony is a CFD expert since 1990, working initially as a senior researcher, then moved to Engineering, acting also as technical director in a challenging Automotive Tier 1 supplier environment. Since 2001, Anthony has worked in Software & Engineering Consultancy as a Sales Engineer and manager. In 2020, Anthony fell in love with AI and has worked since then in the field of “AI for CAE” at Neural Concept and as an independent contributor.
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
About the author
Thomas von Tschammer
Thomas joined the team in 2018 as Director of Operations, aiming to empower engineers with next a next generation Deep-Learning tool dedicated to CAD and CAE.
About the author
About the author
About the author
About the author
About the author
About the author
About the author