Bayesian Learning

In this episode Gudrun speaks with Nadja Klein and Moussa Kassem
Sbeyti who work at the Scientific Computing Center (SCC) at KIT
in Karlsruhe.


Since August 2024, Nadja has been professor at KIT leading the
research group Methods for Big Data (MBD) there. She is an Emmy
Noether Research Group Leader, and a member of AcademiaNet, and
Die Junge Akademie, among others. In 2025, Nadja was awarded the
Committee of Presidents of Statistical Societies (COPSS) Emerging
Leader Award (ELA). The COPSS ELA recognizes early career
statistical scientists who show evidence of and potential for
leadership and who will help shape and strengthen the field. She
finished her doctoral studies in Mathematics at the Universität
Göttingen before conducting a postdoc at the University of
Melbourne as a Feodor-Lynen fellow by the Alexander von Humboldt
Foundation. Afterwards she was a Professor for Statistics and
Data Science at the Humboldt-Universität zu Berlin before joining
KIT.


Moussa joined Nadja's lab as an associated member in 2023 and
later as a postdoctoral researcher in 2024. He pursued a PhD at
the TU Berlin while working as an AI Research Scientist at the
Continental AI Lab in Berlin. His research primarily focuses on
deep learning, developing uncertainty-based automated labeling
methods for 2D object detection in autonomous driving. Prior to
this, Moussa earned his M.Sc. in Mechatronics Engineering from
the TU Darmstadt in 2021.


The research of Nadja and Moussa is at the intersection of
statistics and machine learning. In Nadja's MBD Lab the research
spans theoretical analysis, method development and real-world
applications. One of their key focuses is Bayesian methods, which
allow to incorporate prior knowledge, quantify uncertainties, and
bring insights to the “black boxes” of machine learning. By
fusing the precision and reliability of Bayesian statistics with
the adaptability of machine and deep learning, these methods aim
to leverage the best of both worlds. The KIT offers a strong
research environment, making it an ideal place to continue their
work. They bring new expertise that can be leveraged in various
applications and on the other hand Helmholtz offers a great
platform in that respect to explore new application areas. For
example Moussa decided to join the group at KIT as part of the
Helmholtz Pilot Program Core-Informatics at KIT (KiKIT), which is
an initiative focused on advancing fundamental research in
informatics within the Helmholtz Association. Vision models
typically depend on large volumes of labeled data, but collecting
and labeling this data is both expensive and prone to errors.
During his PhD, his research centered on data-efficient learning
using uncertainty-based automated labeling techniques. That means
estimating and using the uncertainty of models to select the
helpful data samples to train the models to label the rest
themselves. Now, within KiKIT, his work has evolved to include
knowledge-based approaches in multi-task models, eg. detection
and depth estimation — with the broader goal of enabling the
development and deployment of reliable, accurate vision systems
in real-world applications.


Statistics and data science are fascinating fields, offering a
wide variety of methods and applications that constantly lead to
new insights. Within this domain, Bayesian methods are especially
compelling, as they enable the quantification of uncertainty and
the incorporation of prior knowledge. These capabilities
contribute to making machine learning models more data-efficient,
interpretable, and robust, which are essential qualities in
safety-critical domains such as autonomous driving and
personalized medicine. Nadja is also enthusiastic about the
interdisciplinarity of the subject — repeatedly changing the
focus from mathematics to economics to statistics to computer
science. The combination of theoretical fundamentals and
practical applications makes statistics an agile and important
field of research in data science.


From a deep learning perspective, the focus is on making models
both more efficient and more reliable when dealing with
large-scale data and complex dependencies. One way to do this is
by reducing the need for extensive labeled data. They also work
on developing self-aware models that can recognize when they're
unsure and even reject their own predictions when necessary.
Additionally, they explore model pruning techniques to improve
computational efficiency, and specialize in Bayesian deep
learning, allowing machine learning models to better handle
uncertainty and complex dependencies. Beyond the methods
themselves, they also contribute by publishing datasets that help
push the development of next-generation, state-of-the-art models.
The learning methods are applied across different domains such as
object detection, depth estimation, semantic segmentation, and
trajectory prediction — especially in the context of autonomous
driving and agricultural applications. As deep learning
technologies continue to evolve, they’re also expanding into new
application areas such as medical imaging.


Unlike traditional deep learning, Bayesian deep learning provides
uncertainty estimates alongside predictions, allowing for more
principled decision-making and reducing catastrophic failures in
safety-critical application. It has had a growing impact in
several real-world domains where uncertainty really matters.
Bayesian learning incorporates prior knowledge and updates
beliefs as new data comes in, rather than relying purely on
data-driven optimization. In healthcare, for example, Bayesian
models help quantify uncertainty in medical diagnoses, which
supports more risk-aware treatment decisions and can ultimately
lead to better patient outcomes. In autonomous vehicles, Bayesian
models play a key role in improving safety. By recognizing when
the system is uncertain, they help capture edge cases more
effectively, reduce false positives and negatives in object
detection, and navigate complex, dynamic environments — like bad
weather or unexpected road conditions — more reliably. In
finance, Bayesian deep learning enhances both risk assessment and
fraud detection by allowing the system to assess how confident it
is in its predictions. That added layer of information supports
more informed decision-making and helps reduce costly errors.
Across all these areas, the key advantage is the ability to move
beyond just accuracy and incorporate trust and reliability into
AI systems.


Bayesian methods are traditionally more expensive, but modern
approximations (e.g., variational inference or last layer
inference) make them feasible. Computational costs depend on the
problem — sometimes Bayesian models require fewer data points to
achieve better performance. The trade-off is between
interpretability and computational efficiency, but hardware
improvements are helping bridge this gap.


Their research on uncertainty-based automated labeling is
designed to make models not just safer and more reliable, but
also more efficient. By reducing the need for extensive manual
labeling, one improves the overall quality of the dataset while
cutting down on human effort and potential labeling errors.
Importantly, by selecting informative samples, the model learns
from better data — which means it can reach higher performance
with fewer training examples. This leads to faster training and
better generalization without sacrificing accuracy. They also
focus on developing lightweight uncertainty estimation techniques
that are computationally efficient, so these benefits don’t come
with heavy resource demands. In short, this approach helps build
models that are more robust, more adaptive to new data, and
significantly more efficient to train and deploy — which is
critical for real-world systems where both accuracy and speed
matter.


Statisticians and deep learning researchers often use distinct
methodologies, vocabulary and frameworks, making communication
and collaboration challenging. Unfortunately, there is a lack of
Interdisciplinary education: Traditional academic programs rarely
integrate both fields. It is necessary to foster joint programs,
workshops, and cross-disciplinary training can help bridge this
gap.


From Moussa's experience coming through an industrial PhD, he has
seen how many industry settings tend to prioritize short-term
gains — favoring quick wins in deep learning over deeper, more
fundamental improvements.
To overcome this, we need to build long-term research
partnerships between academia and industry — ones that allow for
foundational work to evolve alongside practical applications.
That kind of collaboration can drive more sustainable, impactful
innovation in the long run, something we do at methods for big
data.


Looking ahead, one of the major directions for deep learning in
the next five to ten years is the shift toward trustworthy AI.
We’re already seeing growing attention on making models more
explainable, fair, and robust — especially as AI systems are
being deployed in critical areas like healthcare, mobility, and
finance. The group also expect to see more hybrid models —
combining deep learning with Bayesian methods, physics-based
models, or symbolic reasoning. These approaches can help bridge
the gap between raw performance and interpretability, and often
lead to more data-efficient solutions. Another big trend is the
rise of uncertainty-aware AI. As AI moves into more high-risk,
real-world applications, it becomes essential that systems
understand and communicate their own confidence. This is where
uncertainty modeling will play a key role — helping to make AI
not just more powerful, but also more safe and reliable.


The lecture "Advanced Bayesian Data Analysis" covers fundamental
concepts in Bayesian statistics, including parametric and
non-parametric regression, computational techniques such as MCMC
and variational inference, and Bayesian priors for handling
high-dimensional data. Additionally, the lecturers offer a
Research Seminar on Selected Topics in Statistical Learning and
Data Science.


The workgroup offers a variety of Master's thesis topics at the
intersection of statistics and deep learning, focusing on
Bayesian modeling, uncertainty quantification, and
high-dimensional methods. Current topics include predictive
information criteria for Bayesian models and uncertainty
quantification in deep learning. Topics span theoretical,
methodological, computational and applied projects. Students
interested in rigorous theoretical and applied research are
encouraged to explore our available projects and contact us for
further details.


The general advice of Nadja and Moussa for everybody interested
to enter the field is: "Develop a strong foundation in
statistical and mathematical principles, rather than focusing
solely on the latest trends.
Gain expertise in both theory and practical applications, as
real-world impact requires a balance of both. Be open to
interdisciplinary collaboration. Some of the most exciting and
meaningful innovations happen at the intersection of fields —
whether that’s statistics and deep learning, or AI and
domain-specific areas like medicine or mobility. So don’t be
afraid to step outside your comfort zone, ask questions across
disciplines, and look for ways to connect different perspectives.
That’s often where real breakthroughs happen. With every new
challenge comes an opportunity to innovate, and that’s what keeps
this work exciting. We’re always pushing for more robust,
efficient, and trustworthy AI. And we’re also growing — so if
you’re a motivated researcher interested in this space, we’d love
to hear from you."

Literature and further information

Webpage of the group

G. Nuti, Lluis A.J. Rugama, A.-I. Cross: Efficient Bayesian
Decision Tree Algorithm, arxiv Jan 2019

Wikipedia: Expected value of sample information

C. Howson & P. Urbach: Scientific Reasoning: The Bayesian
Approach (3rd ed.). Open Court Publishing Company. ISBN
978-0-8126-9578-6, 2005.

A.Gelman e.a.: Bayesian Data Analysis Third Edition. Chapman
and Hall/CRC. ISBN 978-1-4398-4095-5, 2013.

Yu, Angela: Introduction to Bayesian Decision Theory
cogsci.ucsd.edu, 2013.

Devin Soni: Introduction to Bayesian Networks, 2015.

G. Nuti, L. Rugama, A.-I. Cross: Efficient Bayesian Decision
Tree Algorithm, arXiv:1901.03214 stat.ML, 2019.

M. Carlan, T. Kneib and N. Klein: Bayesian conditional
transformation models, Journal of the American Statistical
Association, 119(546):1360-1373, 2024.

N. Klein: Distributional regression for data analysis ,
Annual Review of Statistics and Its Application, 11:321-346, 2024

C.Hoffmann and N.Klein: Marginally calibrated response
distributions for end-to-end learning in autonomous driving,
Annals of Applied Statistics, 17(2):1740-1763, 2023

Kassem Sbeyti, M., Karg, M., Wirth, C., Klein, N., &
Albayrak, S. (2024, September). Cost-Sensitive Uncertainty-Based
Failure Recognition for Object Detection. In Uncertainty in
Artificial Intelligence (pp. 1890-1900). PMLR.

M. K. Sbeyti, N. Klein, A. Nowzad, F. Sivrikaya and S.
Albayrak: Building Blocks for Robust and Effective
Semi-Supervised Real-World Object Detection pdf. To appear in
Transactions on Machine Learning Research, 2025

Podcasts

Learning, Teaching, and Building in the Age of AI Ep 42 of
Vanishing Gradient, Jan 2025.

O. Beige, G. Thäter: Risikoentscheidungsprozesse, Gespräch im
Modellansatz Podcast, Folge 193, Fakultät für Mathematik,
Karlsruher Institut für Technologie (KIT), 2019.