Agents and multi-agent systems (MAS) have evolved from
autonomous problem-solving entities to integrating generative
AI and LLMs, ultimately leading to cooperative AI frameworks
that enhance adaptability, scalability, and collaboration. 
**Main Takeaways**
* Multi-agent systems have evolved from rule-based autonomy to cooperative AI,
emphasizing collaboration, negotiation, and ethical alignment.
* The rise of Agentic AI, driven by LLMs, introduces new opportunities for flexible
decision-making but raises challenges in efficiency and complexity.
* Integrating cooperative AI with generative models requires balancing
adaptability, transparency, and computational feasibility in multi-agent
environments 

**Context & History** 
The field of multi-agent systems emerged in the late 1980s/early 1990s,
with its main influences coming from two disparate areas [1,2]. One was the
field of AI robotics, which had begun to seriously address the issue of integrated agent architectures: how do we assemble several cognitive components of intelligence (planning, reasoning, learning, vision,…) into an integrated computational agent? The second was the nascent area of distributed AI, which studied how multiple AI systems could be made to solve problems cooperatively, by dynamically sharing information and tasks. By the mid1990s, these ideas had given rise to the new field, driven by the vision of having (semi)autonomous AI systems – agents – working on behalf of individual users in pursuit of their users’ goals, possibly interacting with other such agents in order to do so. A key insight
was that, since delegated goals might not necessarily be in harmony, it would be necessary to equip such agents with the ability to reason socially. Thus, while AI historically emphasised components of intelligence such as reasoning and problem-solving, the new field of multi-agent systems emphasised social skills such as cooperation, coordination, argumentation and negotiation. In order to underpin those skills, the development of models of Theory of Mind became central to the field. 
By the late 1990s, the field had its own conferences and journal, and was
firmly established as a key sub-field of AI. The area flourished from the late
1990s, with enormous energy devoted to (e.g.) communication languages
for autonomous agents, protocols for cooperation, coordination, and
negotiation, and the underpinning theory of these social skills. With
respect to the latter, while in the early years the practical reasoning paradigm of AI planning had been the dominant influence on the theory of multiagent systems, by the early part of this century, game theory had become the dominant theoretical foundation. Game theory, which emerged from the field of economics, is the theory of interaction between self-interested agents. Although originally devised as a tool for studying interactions between humans and human organisations, it nevertheless seemed a natural framework for studying interactions between artificial agents. A huge body of work emerged, studying (for example), how auctions might be used to allocate scarce resources, the theory
of negotiation between self-interested artificial agents, and how agents might optimally form teams to solve problems and share the associated benefits of cooperation. Interestingly, although learning in multi-agent systems was a key component of the field from the outset, it was not the centre of attention within the field in the first decade or so. 
This initial boom period for multiagent systems lasted roughly from
the mid 1990s to around 2010-15. By the end of that time, though,
some uncomfortable questions were beginning to be asked. While the field
had generated impressive quantities of scientific results, applications seemed to be thin on the ground. For sure there were some high-profile applications. The field of security games, which emerged from multi-agent systems, used ideas from game theory to allocate scarce security resources to defend high-profile targets such as airports. This work led to deployed
applications at US airports and ports. Automated high-frequence trading
systems, which plan and execute the bulk of trades on the world’s markets,
are multi-agent systems on a global scale. And agent-based modelling,
which models socio-technical systems at the level of individual decisionmakers, received a huge boost after the 2008 financial crisis, and again after the 2020 COVID-19 pandemic, where it was demonstrated to be an important tool for modelling the spread of contagion: financial in the first case; epidemiological and social policy in the second. But for all these successes, the core vision of multi-agent systems - where agents function in the context of other agents is an active area of research, exploring social concepts such as norms, organisations, practices and also values, is ongoing work, but for a large part outside the AAMAS community, but within social simulation research, and with results informing and
shaping policy-making in several areas from public health to transportation,
and urban transformation. 
While applications of multi-agent systems research (AI agents interacting
with other AI agents) has not, as yet, lived up to early expectations,
individual dialogue agents such as Alexa, Siri, Cortana are now an everyday
reality, and trace their historical roots both to work on intelligent agents
in the 1990s and work from the NLP community on dialog systems. Many
other applications of this thread of work have achieved success over the past three decades: automated call center assistants, customer service assistants, smartphone virtual assistants, smart speaker assistants, home robot assistants, that can converse with human users and accomplish tasks like ticket booking, restaurant reservations, online shopping, medical and health assistance, and sales assistance, empowered by AI modules like
speech recognition, natural language understanding, dialog management (i.e. state tracking and dialog policy), and natural language generation and speech synthesis. 
As ML surged in the early part of this century, activity in this area increased
within the multi-agent systems field. Multi-agent reinforcement
learning (MARL) grew to become the single biggest area within the field,
possibly driven in part by the fact that developing MARL experiments can be done relatively quickly and without recourse to expensive hardware. At the time of writing, while MARL represents a significant sub-field of ML as a whole, it seems to lack any clear unifying vision or direction – or application. 
**Current State & Trends** 
The emergence of LLMs from 2020 onwards has also led to increased
interest in agents [3]. LLMs can be used as part of a workflow to
automate routine tasks, and the general capabilities of such “agents” for
planning and problem solving is widely discussed. In this context, the concept of Agentic AI refers to the integration of generative AI and LLMs into autonomous agent frameworks aiming to leverage the generative capabilities of such models to enhance interaction, creativity, and real-time decision making in dynamic environments. As we write this (late 2024) there has been an explosion of startup companies hoping to commercialise such agents. Despite this renewed enthusiasm, the original aims of AAMAS from 30 years ago, such as building robust, autonomous multiagent systems capable of complex coordination and long-term reasoning, have not been fully realized. The extent to which this new wave of agent activity is informed by what went before is unclear. 
The challenge now is to understand what multi-agent systems mean in
the era of LLMs. The current direction of agentifying LLMs may lead to
overly complex and unnecessary architectures and heavy computational
costs, whereas adopting a multi-agent paradigm to the development and use of LLMs may offer a sustainable way to compose, diversify, and integrate approaches effectively. Even though distribution was one of the original drivers for the MAS field, this is still a largely unexplored direction under the current paradigm. Another trend nowadays is to recover ideas from classical cognitive architectures to add common sense skills to autonomous agents. 
An emerging trend is multi-agent architectures, which structure
AI components into modular systems that improve transparency,
adaptability, and ethical alignment. The focus on cooperative agents
highlights a shift toward AI that prioritizes collaboration, negotiation, and shared decision-making. By applying modularity, encapsulation,
and separation of concerns, these architectures enable scalable teamwork
between autonomous agents and humans, making them ideal for
hybrid AI applications requiring trust, explainability, and domain-specific
expertise. 
**Research Challenges** 
* Identify challenges and benefits of embedding GenAI-driven agents
into MAS, focusing on enhancing collaboration without disrupting
existing dynamics.
* Investigate how LLM-powered agents can improve negotiation and
decision-making in dynamic multiagent environments while ensuring
ethical alignment and safety.
* Develop architectures that integrate LLM-driven agents while maintaining
scalability, transparency, and computational efficiency in multiagent settings. 
**Community Opinion** 
The survey responses indicate a
majority of respondents finding this
theme relevant to their research, with
a growing interest in integrating Large
Language Models (LLMs) into multiagent systems. Many participants
already use AI agents, with LLMs being
the most common technique (29.34%),
highlighting their expanding role in AI driven applications. 
The potential of multi-agent systems
leveraging LLMs is seen in areas such
as collaborative problem-solving
(68.86%), distributed decision-making
(54.49%), and social simulations
(41.32%). However, challenges persist,
including misalignment between LLMs’
general knowledge and specific system
needs (59.88%), lack of interpretability
(59.28%), and security risks (50.90%).
These concerns suggest a need for
improved explainability, alignment
strategies, and robust security measures
to ensure effective deployment. 
There is also a debate on the necessity
of agentifying LLMs—while 51.5%
believe multi-agent LLM paradigms
are essential for sustainable AI,
42.33% disagree that they introduce
unnecessary complexity. The
computational cost-benefit balance
remains uncertain, with responses
divided on whether LLMs outweigh
their costs. 
The textual responses highlight a broad
spectrum of perspectives on integrating
Large Language Models (LLMs) into
multi-agent systems (MAS), with some
advocating for hybrid approaches
rather than relying solely on LLMs. Many
respondents stressed the need for
diverse AI architectures, emphasizing
modular, multi-technology systems
where LLMs play a role but do not
dominate. Governance, coordination,
and adaptability emerge as key
advantages of MAS, while concerns
include increased complexity, lack
of theoretical guarantees, and high
computational costs. Several responses
criticize the overemphasis on LLMs,
questioning whether they are truly
essential or merely a current trend.
Others highlight practical challenges
such as grounding, alignment, and
robust communication protocols,
pointing out the need for new
frameworks that integrate symbolic
reasoning, structured governance,
and scalable architectures. Overall,
the discussion reflects a critical but
open stance toward agentifying LLMs,
suggesting that context, application
domain, and technological diversity will
shape their effectiveness in multi-agent
environments. 
In summary, the survey reflects
optimism about LLM-driven multiagent systems, but also underscores
the need for addressing key challenges
before widespread adoption.
 
1. Yoav Shoham. Agent-oriented programming. Artificial Intelligence. Artificial Intelligence. 60 (1): 51–92.
2. Michael Wooldridge. An Introduction to Multi-agent Systems 2e. Wiley, 2009.
3. Julia Wiesinger, Patrick Marlow and Vladimir Vuskovic. Agents. Google whitepaper. https://archive.org/details/google-ai-agents-whitepaper

AI agents

The ability to reason has been a salient characteristic of human
intelligence, and there is a critical need for verifiable reasoning
in AI systems. 
**Main Takeaways** 
* Reasoning has always been seen as a core characteristic of human intelligence.
Reasoning is used to derive new information from given base knowledge; this
new information is guaranteed correct when sound formal reasoning is used,
otherwise it is merely plausible.
* AI research has led to a range of automated reasoning techniques. These
reasoning techniques have given rise to AI algorithms and systems, including
SAT, SMT, and constraints solvers as well as probabilistic graphical models, all of
which play a key role in critical real-world applications.
* While large pre-trained systems (such as LLMs) have made impressive
advancements in their reasoning capabilities, more research is needed to
guarantee correctness and depth of the reasoning performed by them; such
guarantees are particularly important for autonomously operating AI agents 

**Context & History** 
Reasoning is a core component of
human intelligence. From the dawn
of humanity, abductive reasoning
has been used to predict danger and
inductive reasoning made it possible
to learn regularities governing the
world. Beginning in Ancient Greece,
deductive reasoning techniques were
developed to draw valid conclusions
that follow logically from premises
known to be true. The development
of reasoning methods with such a
priori guarantees was a key factor in
the advancement of modern science,
mathematics, and engineering; notably,
according to philosophers such as
Charles Sanders Peirce, the interplay
between abduction, deduction, and
induction forms the basis of the
scientific method and hence all modern
science. Attempts to mechanize
logical reasoning can be traced back to
13th-century philosopher Ramon Lull
and lie at the heart of the concept of
computation. Probabilistic reasoning
and inference have also profoundly
impacted reasoning, often relying on
the celebrated theorem by Thomas
Bayes on inverse probability that also
forms the basis for many machine
learning and statistics approaches.
Finally, the evaluation of correct (sound)
reasoning lies at the heart of most
quantitative assessments of human
cognition. 
Not surprisingly, reasoning has been
central to the AI enterprise. Indeed,
the earliest research in AI – from Logic
Theorist onwards [1] – had a strong
focus on reasoning [2]. Since the 1960s,
AI has also embraced probabilistic
reasoning and models, initially for
medical diagnosis [3]. Since then,
the reasoning tasks addressed in AI 
research have covered the gamut from
planning and temporal reasoning to
diagnosis and explanation. While early
AI has paid attention to both plausible
reasoning (case-based, analogical,
qualitative) and sound formal reasoning
with guarantees (logical, probabilistic,
constraint-based), over the years,
the focus has shifted more towards
reasoning with formal guarantees.
There are good reasons for this when
designing AI systems and techniques
that compensate for human limitations
and weaknesses since reasoning
with guarantees is challenging for
humans. This has led to practically
impactful applications of AI systems
such as SAT, SMT, and constraints
solvers, including the verification of
correctness properties of computer
hardware and software, the safety of
communications protocols, the design
of new proteins, and, more recently, the
robustness of neural networks against
adversarial attacks. It has also resulted
in probabilistic graphical models [4,
5], which are powerful modeling and
inference tools that have found their
way into numerous applications of
reasoning in medicine, robotics, and
beyond. 
**Current State & Trends** 
The emergence of the Internet and the
associated technology that made it
possible to capture the human digital
footprint at scale, as well as the leaps in
computing power, have made possible
novel approaches to learning bottomup from data. Of particular interest are
large pre-trained models, such as LLMs,
that have shown surprising abilities in
plausible reasoning. Unlike the earlier
research on reasoning in AI, LLMs
have focused on plausible reasoning patterns as they emerge automatically
after large-scale training on petabyte
corpora. While the results have been
quite remarkable so far, the reasoning in
this context has been of the “plausible”
variety with no guarantees. 
Meanwhile, sound formal reasoning
techniques remain key to important
and impactful applications of cuttingedge AI technology for the verification
of computer hardware and software,
as well as for real-world planning and
resource allocation problems. They
are also increasingly recognized as a
crucial basis for the formal verification
of machine learning techniques such
as neural networks, e.g., in the context
of local robustness against adversarial
attacks [6]. Significant research activity
takes place in these areas, focusing on
improving various types of reasoning
algorithms (notably with respect to their
computational complexity), leveraging
learning within sound formal reasoning,
and combining reasoning and learning
techniques [7, 8]. 
**Research Challenges** 
Bringing some of the rigorous a priori or
post hoc guarantees back into plausible
reasoning patterns turbocharged by
the pre-trained models has become an
active and promising area of research
– especially where AI systems need to
work autonomously in safety-critical
domains. Research on so-called “large
reasoning models” as well as on neurosymbolic approaches is addressing
these challenges. 
Furthermore, even though formal
reasoning with correctness guarantees
is currently considerably less in vogue
than the use of generative AI techniques
for plausible reasoning, formidable and essential challenges also remain in that
area. In this context, the combination
of machine learning techniques with
formal reasoning techniques holds
considerable promise for economically
and socially valuable breakthroughs,
notably in the area of AI safety and
transparency.
The questions and challenges we face
range from the philosophical:
* What exactly is “reasoning”?

to the practical:
* Can LLM ‘reasoning’ be trusted?

and include: 
What does the future hold for the
advancement and role of symbolic
reasoning?
* To what extent can LLMs or other
generative models reproduce or
replace symbolic reasoning?
* To what degree will symbolic
reasoning be necessary or sufficient
to overcome the current limitations
of LLMs?
* How well can AI reasoning,
especially LLM ‘reasoning,’ be
explained and understood?
* How can computers better
understand and simulate human
reasoning?
* What is the role of collaborative
reasoning between humans and
computers?
* How best can LLMs and symbolic
reasoning be integrated into “neurosymbolic reasoning”?
* Are further breakthroughs, beyond
both LLMs and traditional symbolic
reasoning, required to achieve AGIlevel reasoning?
* What forms of reasoning can best
support humans when dealing with
various challenges, e.g., in medical,
scientific, engineering, and legal
domains? 

**Community Opinion** 
The AAAI community appears to
strongly agree on the importance
of reasoning in AI systems. In our
community survey, slightly over 55%
of the respondents chose to answer
specific questions related to the topic
of reasoning. Of these, 79% indicated
that the topic of reasoning is relevant
to their research (with 44.7% marking
it as “very relevant”). Of the properties
required for referring to a process
as reasoning, 77.5% of the survey
participants marked “Knowledge can
be incorporated”, 72.5% “Explanations
can be provided,” and 56.9% “Involves
multiple steps to arrive at a conclusion”.
Interestingly, merely 37.4% indicated
“Guaranteed correctness of inference
results/outcomes”, and only 23.7% that
“A formal system and solver is used,”
which reflects the recent focus on
informal, plausible reasoning, likely in
the context of generative AI methods.
This suggests that an effort may be
warranted to better communicate the
importance and success of formal,
sound reasoning techniques. Finally,
44.7% of respondents agreed that
“Reasoning involves a search process.” 
There was broad agreement among
survey participants that focusing
reasoning research in AI on humanlevel reasoning is valuable (41.6%) or
even essential (47%); similarly, a focus
on domain-specific reasoning abilities
was seen by 49.6% of respondents as
valuable, and by 42.8% as essential.
This clearly reflects the importance
attributed to a research focus on
reasoning. 
The community also sees an exciting
potential of synergy offered by logical
and probabilistic models of reasoning
that were developed in AI prior to large
pre-trained models. This is clearly
reflected in the fact that 76.9% of survey
participants marked the integration
of learning and reasoning approaches
as very important (6 or 7 on a scale
of 7); interestingly, the percentage
of respondents that considered
Explainability and verifiability as very
important was similarly high (at 71.7%). 
Finally, 61.8% of survey participants
estimated the minimal percentage of
symbolic AI techniques required for
reaching human-level reasoning to be
at least 50% (with 24.8% estimating it
at 75% or more, compared to 38.2%
estimating it at 25% or below). What
remains unclear is the degree to
which AI researchers and practitioners
realize that decidedly superhuman
levels of reasoning are required for
and displayed in the prominent and
successful applications of formal AI
reasoning techniques for scientific
and mathematical discovery and
engineering applications, as well as in AI
safety. 
1. Newell, A. & Simon, H. (1956). The logic theory machine: A complex information processing system. IRE Transactions on Information Theory 2: 61-79. 
2. Brachman, R. and Levesque, H. (2004) Knowledge Representation and Reasoning (1st Ed). Morgan Kaufman. 
3. Russell, S. J., & Norvig, P. (2016). Artificial intelligence: a modern approach (4th Ed). Pearson. 
4. Pearl, J. (1988). Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufman. 
5. Koller, D. and Friedmann, N. (2009) Probabilistic Graphical Models. The MIT Press. 
6. König, M. et al. (2024) Critically Assessing the State of the Art in Neural Network Verification. Journal of Machine Learning Research 25(12): 1-53 
7. Guo, D. et. al. (2025) DeepSeek-R: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. https://arxiv.org/abs/2501.12948 
8. Kambhampati, S. (2024). Can Large Language Models Reason and Plan? Annals of New York Academy of Sciences. March 2024.

AI Reasoning

Improving factually and trustworthiness of AI systems is the
single largest topic of AI research today, and while significant
progress has been made, most scientists are pessimistic that
the problems will be solved in the near future. 
**Main Takeaways** 
* An AI system is factual if it refrains from outputting false statements. Improving
factually of AI systems based on neural-network large language models is
arguably the biggest area of AI research today.
* Trustworthiness extends trustworthiness to include criteria such as human
understandability, robustness, and the incorporation of human values. Lack of
trustworthiness has always been an obstacle for deploying AI systems in critical
applications.
* Approaches to improving the factuality and trustworthiness of AI systems
include fine-tuning, retrieval-augmented generation, verification of machine
outputs, and replacing complex models with simple understandable models. 

**Context & History** 
A factual AI system does not output
erroneous information or hallucinate
answers. Before the era of generative
AI, problems with factuality arose when
systems were trained on bad data,
as captured by the slogan, “garbage
in, garbage out”. Work on methods
for improving data quality has a long
history in AI [1]. 
Generative AI, and in particular
large language models, employ
reconstructive memory - that is, they
rebuild memories as needed on the
basis of distributed bits of information
rather than retrieve memories from
a fixed store. The earliest generative
LLMs made an impact with their ability
to generate coherent but entirely
imaginary stories [2]. Factuality of LLMs
on a given domain was improved by
fine-tuning the model on domain data
[3]. 
Trustworthiness is a broader concept
than factuality because it includes
criteria such as understandability,
robustness, and respect of human
values. A traditional approach for
improving understandability of AI
systems is to replace complex blackbox models with simple humanunderstandable models - such as
naive Bayes [4] or generalized linear
regression [5]. Research on robustness
of machine learning studies how the
outputs of a model vary with small
changes in its training data. For
example, contrastive learning is a
method to train deep neural nets with
increased robustness [6]. Further
discussion of robustness in generative
AI appears in
this report’s section on Reasoning.
Discussion of respect of human values
by AI systems appears in many other
sections of this report and so will not be
discussed here. 

**Current State & Trends** 
As noted, fine-tuning remains the
main approach used by scientists
and engineers to improve factuality
of generative AI systems. In addition
to fine-tuning on domain-specific
documents, modern fine-tuning
includes reinforcement learning with
human feedback from thousands of
people. The cost of employing such
large numbers of human evaluators
is a major bottleneck for scaling AI
systems, and so there is much interest
in discovering methods to reduce the
amount of human feedback needed [7]. 
The second main technique for
improving factuality of generative AI
is retrieval-augmented generation
(RAG) [8]. In response to a question,
the system gathers a set of relevant
documents using traditional
information retrieval algorithms. The
AI system is then prompted to generate
an answer by combing through and
summarizing the retrieved documents.
While RAG can improve factuality, it
is dependent on the quality of data
retrieved. For example, if the target
document set is the entire web, it
can end up incorporating incorrect
information and even satirical stories in
its answer. 
Related to RAG is enabling the
generative AI system to use tools for
fact checking. Tools used by generative
AI systems include calculators, factual
databases such as citation indexes, and
formal planning and reasoning systems
[9]. A recent approach to improving
factuality is to provide the system with
a set of rules that state constraints on
space of answers. The output from the
model is verified against these rules and
inconsistent responses are culled [10].
Amazon Web Services already supports
this approach with “automated
reasoning checks” [11]. 
A third technique for improving
factuality of generative AI is chainof-thought (CoT), where a series of
prompts breaks down a question into
smaller units [12]. CoT often includes
steps where the model is asked to
reflect back on its tentative conclusions
and see if any are hallucinations. CoT
is discussed in more detail in the
Reasoning section of this report. 
The impact of data quality on factuality
was mentioned above. In addition
to fine-tuning on human curated
data, there is recent work on creating
synthetic data that is guaranteed to be
high quality for fine-tuning [13]. 
Trustworthiness, we noted, generalizes
factually and includes understandability
and robustness. One approach to
making neural network models more
understandable is to factor them into a
set of recognizers for high level features
and then combine the features using an
understandable model such as additive
regression [15]. Another approach is to
tease out how concepts and rules are
actually represented in a trained [16].
Understandability can also be
improved by employing CoT techniques
to ask a generative AI system to explain
the steps in its reasoning [17] or tell
the user when the system is uncertain
about a conclusion [18]. Finally, a
generative AI system can be asked not
to output a single answer, but instead
to distill a complex set of information into a simple human understandable
representation such as a decision tree
[19]. 
**Research Challenges** 
Factuality is far from solved. There
are a growing number of benchmark
dataset designed to test the factuality
of LLMs. One of the latest, SimpleQA
from Google, is a collection of simple,
unambiguous, timeless, and challenging
factual questions and answers
[14]. As of December 2024, the best
models from OpenAI and Anthropic
correctly answered less than half of the
questions. 
Robustness in generative AI can
be improved, as noted above, by
employing robust loss functions such
as contrastive learning. Adversarial
training, which applies perturbations in
the embedding space during training,
can improve both robustness and
generalization [20]. In addition, the
techniques for factuality generally
improve robustness as well. 
**Community Opinion** 
Over 75% of the AAAI community
strongly agreed that factuality and
trustworthiness were relevant or very
relevant to their own research. 
All six approaches mentioned in
the survey for improving factuality - external fact-checking tools,
reinforcement, improving data quality,
data curation, synthetic training, and
new neural net architectures - found
support. The greatest demand was
for more research on new neural net
architectures (73% marked important
or very important), followed closely by
external fact-checking tools (70%). 
For trustworthiness, new neural
net architectures were also viewed
as most important (77% important
or very important), followed by
enabling models to describe their
reasoning processes (70%) and use
of understandable models instead of
neural networks (61%). Notably, few
viewed research on giving AI systems
human-like personalities as important
to improving trustworthiness (24%).
Finally, the community mostly agreed
(59%) that trustworthiness as currently
formulated is ill-defined. Most also
disagreed (around 60%) that either
factuality or trustworthiness would
soon be solved. 
The AAAI community suggested
a number of additional aspects of
factuality and trustworthiness that were
not covered above. These included:
* The ability to understand and
present different sides of the same
issues, including pros and cons for
each one.
* Understanding that trustworthiness
depends upon the context of the
domain, organizational objectives,
and user objectives. It is wrong
to think of an AI system as simply
being trustworthy or untrustworthy
without regard to context.
* Transparency needs to go beyond
the models used to the actual
sources of training data. This should
include multi-source verification of
facts ingested by the models.
* The focus of work should be
on risk and mitigation rather
than on “solving” factuality and
trustworthiness.
* Attention needs to paid to giving
AI agents the ability to update
their knowledge while maintaining
reliability.
 

1. Budach, Lukas, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Sina Noack, Hendrik Patzlaff, Hazar Harmouch and Felix Naumann (2022). The Effects of Data Quality on Machine
Learning Performance. https://arxiv.org/pdf/2207.14529
2. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI. Retrieved from https://cdn.openai.com/betterlanguage-models/language_models_are_unsupervised_multitask_learners.pdf
3. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. https://doi.org/10.48550/
arXiv.1810.04805
4. Pedro Domingos & Michael Pazzani (1997). On the Optimality of the Simple Bayesian Classifier under Zero-One Loss. Machine Learning, 29(2-3), 103–130. https://doi.
org/10.1023/A:1007413511361
5. Caruana, Rich, Yin Lou, Johannes Gehrke, Paul Koch, M. Sturm and Noémie Elhadad (2015). Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day
Readmission. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015). https://people.dbmi.columbia.edu/noemie/
papers/15kdd.pdf
6. Hadsell, R., Chopra, S., & LeCun, Y. (2006). Dimensionality reduction by learning an invariant mapping. Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision
and Pattern Recognition (CVPR 2006), 2, 1735–1742. https://doi.org/10.1109/CVPR.2006.100
7. Hu, C., Hu, Y., Cao, H., Xiao, T., & Zhu, J. (2024). Teaching language models to self-improve by learning from language feedback. Findings of the Association for Computational Linguistics
(ACL 2024). Retrieved from https://aclanthology.org/2024.findings-acl.364/
8. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for
knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems (NeurIPS 2020), 33, 9459–9474. Retrieved from https://arxiv.org/abs/2005.11401
9. Guan, L., Valmeekam, K., Sreedharan, S., & Kambhampati, S. (2023). Leveraging pre-trained large language models to construct and utilize world models for model-based task
planning. Proceedings of the 33rd International Conference on Automated Planning and Scheduling (ICAPS 2023). Retrieved from https://arxiv.org/abs/2305.14909
10. Backes, J., Bolignano, P., Cook, B., Dodge, C., Gacek, A., Luckow, K., Rungta, N., Tkachuk, O., & Varming, C. (2018). Semantic-based automated reasoning for AWS access policies using
SMT. In 2018 Formal Methods in Computer-Aided Design (FMCAD) (pp. 1–9). IEEE. https://doi.org/10.23919/FMCAD.2018.8602994
11. Barth, Antje (2024). Prevent factual errors from LLM hallucinations with mathematically sound Automated Reasoning checks (preview). Posted 3 Dec 2024, retrieved 8 Feb 2025. AWS
News Blog, permalink https://aws.amazon.com/blogs/aws/prevent-factual-errors-from-llm-hallucinations-with-mathematically-sound-automated-reasoning-checks-preview
12. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. In Advances in
Neural Information Processing Systems (Vol. 35, pp. 24824–24837). https://proceedings.neurips.cc/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
13. Ding, Bosheng, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu and Shafiq R. Joty (2024). Data Augmentation using LLMs: Data
Perspectives, Learning Paradigms and Challenges. Annual Meeting of the Association for Computational Linguistics (2024). https://aclanthology.org/2024.findings-acl.97.pdf
14. Wei, Jason, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, William Fedus (2024). Measuring short-form factuality in large language
models. https://doi.org/10.48550/arXiv.2411.04368
15. Agarwal, R., Melnick, L., Frosst, N., Zhang, X., Lengerich, B., Caruana, R., & Hinton, G. E. (2021). Neural additive models: Interpretable machine learning with neural nets. Advances in
Neural Information Processing Systems, 34, 2021. https://proceedings.neurips.cc/paper/2021/hash/251bd0442dfcc53b5a761e050f8022b8-Abstract.html?utm_source=chatgpt.com
16. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., Pearce, A., Citro, C., Ameisen, E., Jones, A., Cunningham, H., Turner, N. L., McDougall, C., MacDiarmid, M., Tamkin,
A., Durmus, E., Hume, T., Mosconi, F., Freeman, C. D., Sumers, T. R., Rees, E., Batson, J., Jermyn, A., Carter, S., Olah, C., & Henighan, T. (2024). Scaling Monosemanticity: Extracting
Interpretable Features from Claude 3 Sonnet. Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html
17. Yeo, W. J., Ng, X. X., Le, T. K. C., & Lu, X. (2024). How interpretable are reasoning explanations from prompting large language models? Findings of the Association for Computational
Linguistics: NAACL 2024. Retrieved from https://aclanthology.org/2024.findings-naacl.138
18. Lin, S., Hilton, J., & Evans, O. (2022). Teaching Models to Express Their Uncertainty in Words. Transactions on Machine Learning Research. https://openreview.net/pdf?id=8s8K2UZGTZ
19. Chen, Y., Zhang, L., Wang, H., & Li, J. (2025). Zero-Shot Decision Tree Construction via Large Language Models. arXiv preprint arXiv:2501.16247. Retrieved from https://arxiv.org/
abs/2501.16247
20.Liu, X., Cheng, H., He, P., Chen, W., Wang, Y., Poon, H., & Gao, J. (2020). Adversarial training for large neural language models. arXiv preprint arXiv:2004.08994. Retrieved from https://
arxiv.org/abs/2004.08994

AI Factuality & Trustworthiness

AI & Sustainability

Artificial General Intelligence (AGI)

How should we challenge exaggerated claims about AI’s
capabilities and set realistic expectations? 
**Main Takeaways** 
* Over the last 70 years, against a background of constant delivery of new and important technologies, many AI innovations have generated excessive hype.
* Like other technologies these hype trends have followed the general Gartner Hype Cycle characterization.
* The current Generative AI Hype Cycle is the first introduction to AI for perhaps the majority of people in the world and they do not have the tools to gauge the validity of many claims 

**Context & History** 
Artificial intelligence, or AI, is the field
that studies the synthesis and analysis
of computational agents that act
intelligently [6]. AI has gone through
hype cycles multiple times since the
1956 workshop that established the
name AI and set the course for early
computer science departments to
include AI as a major component of
their research and teaching. All hype
bubbles eventually burst, as the
essence of hype is that it is beyond
reality. Over the decades this has led to
AI winters where funding has dried up
for all of AI or for specific aspects of AI
such as neural networks or robotics. 
A study published 2017 on trends with
the public perception of AI over a 30-
year period found that discussion about
AI had sharply increased since 2009
and that discussions in the public press
had been consistently more optimistic
than pessimistic [4]. The study also
found that hopes about AI applications
of AI in healthcare and education
were increasing over time. Another
finding was that concerns were growing
about loss of control of AI, ethical
implications, and the negative impact of
AI on work. 
Perhaps the difference in recent years
with prior periods is that hype has
gone beyond the pages of academic
conferences, conference papers, and
scientific magazines, out into both the
mainstream media, and social media.
AI and Artificial Intelligence have
become common words that nontechnical people have heard about
and a common subject for leaders
of almost all countries to talk about.
Governments, for the first time, have AI
policies. 
One of the problems is that AI is
actually a wide-reaching term that can
be used in many different ways. But
now in common parlance it is used
as if it refers to a single thing. In their
2024 book [5] Narayanan and Kapoor
likened it to the language of transport
having only one noun, ‘vehicle’, say, to
refer to bicycles, skate boards, nuclear
submarines, rockets, automobiles, 18
wheeled trucks, container ships, etc.
It is impossible to say almost anything
about ‘vehicles’ and their capabilities
in those circumstances, as anything
one says will be true for only a small
fraction of all ‘vehicles’. This lack of
distinction compounds the problem
of hype, as particular statements get
overgeneralized. 
The hype also sets expectations for
ordinary people. Many are fearful that
they will lose their jobs to AI in the
short term. Social scientists then work
to solve labor disruptions, e.g., for
displaced truck drivers [6], based on
predictions about AI (and in this case,
self-driving trucks) and its adoption that
turn out to be wildly optimistic. There
are no deployed self-driving trucks in
the predicted time frame. 
Hype in response to a technology
trigger is not restricted to AI. Indeed
the business intelligence company
Gartner, has deliberately made
a practice of using a graphical
representation of hype levels through
five stages that are common for many
technologies: (1) technology trigger,
(2) peak of inflated expectations, (3)
trough of disillusionment, (4) slope
of enlightenment, and (5) plateau
of productivity. They have used this
framework to track many technologies,
including quantum computers, block
chain, autonomous vehicles, nanotechnology, etc. In November 2024 they
[1] estimated that hype for Generative
AI had just passed its peak and was on
the downswing. 
The question for AI professionals is
how to respond to this hype, how to
question it, and how to help others
understand what is hubris, while
maintaining their own intellectual
modesty and probity. This is hard to do
in the middle of outsized claims about
one’s own field, and often it is up to
future historians to carefully dissect
past scientific arguments. 
Historian Thomas Haigh has tried to
do such a dissection, almost in real
time, in a recent series of articles in the
Communications of the ACM. In [2] he
gives a post-mortem on the impact of
over-hype in AI that resulted in what is
known as the AI-winter in the 1980s. His
one line summary is: “Fallout from an
exploding bubble of hype triggered the
real AI Winter in the late 1980s.” In [3] he
makes a comparison between the hype
of today and of those earlier times.
He summarizes this particular opinion
piece with the line: “From engines of
logic to engines of bullshit?” 
**Research Challenges** 
Many of us who have worked in AI for
decades face the challenge of trying to
remain honest brokers when we see
that many of the public statements of
people quite new to the field are out of
line with reality. 
The big question is whether, given
the dynamics of social media and the
search for clicks, professional opinions
and peer reviewed research papers
have any impact on dampening the
overclaims and the ways they distort
common understanding of where AI is,
and what is its potential in one year, five
years, ten years, etc. 
If we are currently left out of the
conversations how can we change that? 
**Community Opinion** 
The Community Survey gives
perspectives on the reactions to the AI
Perception vs Reality theme. First, the
results of the survey are summarized
here. 36% of the survey respondents
chose to answer the questions for this
theme. This is the summary breakdown
of the responses to each question: 
**1. How relevant is this Theme for your
own research?** 72% of respondents
said it was somewhat relevant (24%),
relevant (29%) or very relevant (19%). 
**2. The current perception of AI
capabilities matches the reality of AI
research and development.** 79% of
respondents disagreed (47%) or strongly
disagreed (32%). 
**3. In what way is the mismatch
hindering AI research?** 90% of
respondents agreed that it is hindering
research: 74% agreeing that the
directions of AI research are driven by
the hype, 12% saying that theoretical AI
research is suffering as a result, and 4%
saying that less students are interested
in academic research. 
**4. Should there be a communitydriven initiative to counter the hype
by fact-checking claims about AI?** 78%
yes; 51% agree and 27% strongly agree. 
**5. Should there be a communitydriven initiative to organize public
debates on AI perception vs reality,
with video recordings to be made
available to all?** 74% yes; 46% agree
and 28% strongly agree. 
**6. Should there be a communitydriven initiative to build and maintain
a repository of predictions about
future AI’s capabilities, to be checked
regularly for validating their accuracy?**
59% yes; 40% agree and 29% strongly
agree. 
**7. Should there be a communitydriven initiative to educate the public
(including the press and the VCs) about
the diversity of AI techniques and
research areas?** 87% yes; 45% agree
and 42% strongly agree. 
**8. Should there be a communitydriven initiative to develop a method
to produce an annual rating of the
maturity of the AI technology for
several tasks?** 61% yes; 42% agree and
19% strongly agree. 
Since the respondents to this theme
are self-selected (about a third of all
respondents), that bias must be kept
in mind. Of those who responded,
a strong and consistent (though not
completely monolithic) portion felt that
the current perception of AI capabilities
was overblown, that it had a real impact
on the field, and that the field should
find a way to educate people about the
realities.

1. Chandrasekaran, A. [2024]. What’s Driving the Hype Cycle for Generative AI, 2024. https://www.gartner.com/en/articles/hype-cycle-for-genai
2. Haigh, T. [2024]. How the AI Boom Went Bust. Vol. 67 No. 2, pp 22-26.
3. Haigh, T. [2025]. Artificial Intelligence Then and Now. Vol. 68 No. 2, pp 24-29.
4. Fast, E. and Horvitz, E. [2017]. Long-term trends in the public perception of artificial intelligence, AAAI’17: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence,
February 4, 2017, pp 963-969. https://ojs.aaai.org/index.php/AAAI/article/view/10635
5. Narayanan, A. and Kapoor, S. [2024]. AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference. Princeton University Press.
6. Poole, D. L. and Mackworth, A. K. [2023]. Artificial Intelligence: Foundations of Computational Agents, 3rd edition. Cambridge University Press.
7. Wang, S., Mack, E. A., Van Fossen, J. A., Medwid, M., Cotten, S. R., Chang, C. H., Mann, J., Miller, S. R., Savolainen, P.T. and Baker, N. [2023]. Assessing alternative occupations for truck
drivers in an emerging era of autonomous vehicles. Transportation Research Interdisciplinary Perspectives, Vol 19. May. https://doi.org/10.1016/j.trip.2023.100793

AI Perception vs. Reality

As AI becomes increasingly powerful and ubiquitous, it is disrupting skills and displacing workers. NSF’s National AI Institute for Adult Learning and Online Education (AI-ALOE) posits that AI can be part of the solution to the growing problem if we can use AI for reskilling, upskilling, and workforce development at scale. The long-term vision of AI-ALOE is to develop and use AI technologies to enhance the proficiency of online education for all adult learners, using in-person education as a benchmark. The day-to-day mission of AI-ALOE is to conduct responsible research into AI that is grounded in theories of human cognition and learning and derived from the scientific process of learning engineering. I will describe ongoing research at AI-ALOE.

AI for Reskilling, Upskilling, and Workforce Development

Small Data: A New Paradigm for the Next Generation of AI

An agent acts in its world to achieve its objectives. Intelligence allows the agent to make decisions and act. In natural domains, sensing is limited, so acting is gambling. It’s a myth that passive learning and more data are all we need. An agent cannot learn from observations alone. It needs a real body to carry out experiments in its world, testing hypotheses, to determine causation, refining its model of the world’s dynamics. The agent is acting as a scientist: refining its model through experiments and acting appropriately to achieve its objectives. Its objectives depend on its preferences and values, and those of other agents its actions impact. Determining which values to use, and how preferences can be acquired fairly, is a major non-technical challenge. We address three primary questions: What should an agent believe? What should an agent do, given its beliefs, preferences, and abilities? What should the preferences of an agent be? Integrating these issues motivates the design of our latest AI textbook, Artificial Intelligence: Foundations of Computational Agents, (3rd Ed. 2023).

Alan Mackworth is a Professor Emeritus of Computer Science at the University of British Columbia. He works on artificial intelligence with applications in constraint satisfaction, cognitive robotics, assistive technology, hybrid systems and constraint-based agents. He invented the world’s first soccer-playing robots. He has co-authored two books: Computational Intelligence: A Logical Approach (1998) and Artificial Intelligence: Foundations of Computational Agents (2023, 3rd Ed.). Alan co-founded the UBC Cognitive Systems Program, the Centre for AI, Decision-making and Action (CAIDA) and the AI network of BC (AInBC). He has served as President of AAAI, IJCAI and CAIAC. He is a Fellow of AAAI, CAIAC, AGE-WELL, CIFAR and the Royal Society of Canada.

David Poole is a Professor Emeritus of Computer Science at the University of British Columbia. He is known for his work on combining logic and probability, probabilistic inference, relational probabilistic models, statistical relational AI and semantic science. He is a co-author of two AI textbooks (Cambridge University Press, 3rd edition 2023, and Oxford University Press, 1998), and co-author of ” Statistical Relational Artificial Intelligence: Logic, Probability, and Computation”. He is a former chair of the Association for Uncertainty in Artificial Intelligence, the winner of the Canadian AI Association (CAIAC) 2013 Lifetime Achievement Award, and is a Fellow of the Association for the Advancement Artificial Intelligence (AAAI) and CAIAC.

The Essence of Intelligence is Appropriate Action (not thinking, reasoning, learning or language) and other things every student of AI should know

Professor Edward Feigenbaum: a Tribute to and Lecture by a Pioneer of AI on his 90th Birthday

We stand at the dawn of the AI era, a technological revolution poised to be the most consequential of our generation, presenting both unprecedented opportunities and profound challenges. But this promise is shadowed by significant challenges. To build a future we want, we must move beyond the hype and the headlines to confront the most pressing open problems—technical, sociotechnical, and multidisciplinary. This talk will review the rapid progress, dissect challenges ahead, and argue that our greatest task isn’t simply building smarter machines, but fostering the human wisdom to guide them towards a future that is not only intelligent but also equitable, safe, and profoundly human.

Navigating the AI Horizon: Promises, Perils, and the Power of Collaboration

Large language models have revolutionized textual reasoning, yet their ability to act meaningfully in multimodal, real-world environments remains limited. They struggle to ground their decisions in visual context, adapt to changing goals, and plan actions over time—shortcomings that stem from a lack of structured, goal-driven reasoning and insufficient representations of the physical world.

In this talk, I present a unified framework for building embodied agents that can see, simulate, and reason. I begin by introducing methods for learning world simulators from data, arguing that visual reasoning—like textual reasoning—benefits from step-by-step processing. Inverting a physics simulator becomes key: agents must infer structured 3D neural representations of objects, parts, motions, and scenes directly from raw video. I describe methods for extracting such representations using generative priors, injecting them into vision-language models (VLMs), and scaling up their supervision via generative 3D simulation and fast, modular physics engines. These simulators enable agents to anchor their predictions in grounded physical reality, reducing hallucinations and improving control.

Complementing this simulation capability, I explore techniques that enable agents to reason over time and adapt their behavior. By integrating structured memory systems, agents learn to retain and retrieve relevant experiences to inform long-horizon plans. Language-based reflective feedback allows them to refine their strategies beyond what sparse rewards offer, forming abstractions that generalize across tasks. When trained to ground their reasoning directly in the visual environment, agents gain the ability to set subgoals, explore effectively, and verify their own hypotheses.

Together, these advances point toward autonomous systems that simulate before they act, reflect after they fail, and maintain an ongoing awareness of goals, constraints, and context. I will illustrate these capabilities across web automation, robotics, and interactive assistance, showing how agents that see, simulate, and reason offer a promising path toward general-purpose embodied intelligence.

Towards Embodied Agents that See, Simulate, and Reason

From Workflows to Water Coolers: AI That Can Navigate Human Nature

Physicists aim to explain the Universe in terms of a compact, interpretable set of principles. Deducing those principles from experiments poses many challenging and problems which are ripe for application of AI and present opportunities to develop new AI techniques. I will describe how AI has changed the way particle physicists work and speculate about the role of AI in the future of fundamental physics. Finally, I will describe my experience in science communication, as an author, podcaster and television producer.

Fundamental physics and science communication

The pursuit of high-efficiency Artificial General Intelligence (AGI) requires more than brute-force scaling of model size and data. While scaling remains a key driver of capability, equally important are scalable architectural and principles—designs that continue to work, improve, and remain controllable as we vary model scale, domains, and modalities. Central to our approach is the concept of the “Specialized Generalist” – a pathway that achieves deep expertise across multiple domains without sacrificing broad generalization capabilities. In this talk, we introduce the “Specialized Generalist” paradigm and our implementation of it, SAGE (Synergistic Architecture for Generalized Expertise), a three-layer architecture designed to balance specialization and generalization in a systematic way. We will describe how SAGE’s Base Model, Synergy Fusion, and Exploration-Evolution layers interact in practice, focusing on concrete mechanisms for coordinating domain-specific expertise with broad general reasoning. We will share empirical results and recent advances in large reasoning models, embodied AI, and scientific applications to further illustrate the approach. A central motivation is to support “AGI for Science” by building a stable plateau of capabilities that can reliably assist with complex scientific workflows rather than isolated demos. Finally, we will outline the safety and governance questions that arise when deploying Specialized Generalist systems in high-impact settings, and discuss what we have learned so far about monitoring, alignment, and operational safeguards.

Quest of AI towards Specializable Generalist: From Reasoning to Scientific Discovery

There has been a lot of exciting recent progress on new and powerful machine learning algorithms and architectures: how to learn. But for autonomous agents acting in the dynamic, uncertain world, it is at least as important to be able to identify which concepts and subproblems to focus on: what to learn.
This talk presents methods for identifying what to learn within the framework of reinforcement learning, focusing especially on applications in multiagent systems and robotics.

From How to learn to What to learn in Multiagent Systems and Robotics

In this work, we introduce MultiReflect, a novel multimodal self-reflective Retrieval Augmented Generation (RAG)-based automated fact-checking pipeline. MultiReflect is designed to address the challenges of rapidly outdated information, limitations in human query capabilities, and expert knowledge barriers in fact-checking. Our proposed pipeline leverages the latest advancements in Large Language Models (LLMs) and Retrieval Augmented Generation (RAG) to enhance fact verification across text and images. Specifically, by integrating multimodal data processing with RAG-based evidence reflection, our system improves the accuracy of fact-checking by utilizing internet-sourced verification. We evaluate our results on the VERITE benchmarks and using several multimodal LLMs, outperforming baselines in binary classification.

MultiReflect: Multimodal Self-Reflective RAG-based Automated Fact-Checking

Retrieval-augmented generation (RAG) is a powerful paradigm for leveraging external data to enhance the capabilities of large language models (LLMs). However, most existing RAG solutions are tailored for single-modality or limited multimodal scenarios, restricting their applicability in real-world contexts where diverse data sources—including text, tables, images, and videos—must be integrated seamlessly. In this work proposes a unified \textit{Multimodal Retrieval-augmented generation (mRAG)} system designed to unify information processing across all four modalities. Our pipeline ingests and indexes data from PDFs and videos using tools like Amazon Textract, Transcribe, Langfuse, and multimodal LLMs (e.g., Claude 3.5 Sonnet) for structured extraction and semantic enrichment. The dataset includes text queries, table lookups, image-based questions, and videos. Evaluation with the Deepeval framework shows improved retrieval accuracy and response quality, especially for structured text and tables. While performance on image and video queries is lower, the multimodal integration framework remains robust, underscoring the value of unified pipelines for diverse data.

Multimodal Retrieval-Augmented Generation: Unified Information Processing Across Text, Image, Table, and Video Modalities

Automatic metaphor detection has often been inspired by linguistic procedures for manual metaphor identification. In this work, we test how closely the steps required by the Metaphor Identification Procedure VU Amsterdam (MIPVU) can be translated into prompts for generative Large Language Models (LLMs) and how well three commonly used LLMs are able to perform these steps. We find that while the procedure itself can be modeled with only a few compromises, neither language model is able to match the performance of supervised, fine-tuned methods for metaphor detection. All models failed to sufficiently filter out literal examples, where no contrast between the contextual and a more basic or concrete meaning was present. Both versions of LLaMa however signaled interesting potentials in detecting similarities between literal and metaphoric meanings that may be exploited in further work.

Using Large Language Models to Perform MIPVU-Inspired Automatic Metaphor Detection

We present a collection of human association norms to German personal name compounds (PNCs) such as "Tore-Klose" (goal-Klose) and corresponding full names (Miroslav Klose), thus providing a novel testbed for PNC evaluation, i.e., analogical vs. contrastive positive vs. negative perception effects. The associations are obtained in an online experiment with German native speakers, analyzed regarding our novel intertwined PNC--person association setup, and accompanied by an LLM synthetic generation approach for augmentation.

Tore-Klose: Record Scorer, Goal Hunter, Machine? Human Association Norms for German Personal Name Compounds

With the development of large language models (LLMs), social biases in these LLMs have become a pressing issue.
Although there are various benchmarks for social biases across languages, the extent to which Japanese LLMs exhibit social biases has not been fully investigated.
In this study, we construct the Japanese Bias Benchmark dataset for Question Answering (JBBQ) based on the English bias benchmark BBQ, with analysis of social biases in Japanese LLMs.
The results show that while current open Japanese LLMs with more parameters show improved accuracies on JBBQ, their bias scores increase.
In addition, prompts with a warning about social biases and chain-of-thought prompting reduce the effect of biases in model outputs, but there is room for improvement in extracting the correct evidence from contexts in Japanese. 
Our dataset is available at https://github.com/ynklab/JBBQ_data.

JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language Models

An growing number of studies have examined the social bias of rapidly developed large language models (LLMs). Although most of these studies have focused on bias occurring in a single social attribute, research in social science has shown that social bias often occurs in the form of intersectionality---the constitutive and contextualized perspective on bias aroused by social attributes. In this study, we construct the Japanese benchmark inter-JBBQ, designed to evaluate the intersectional bias in LLMs on the question-answering setting. Using inter-JBBQ to analyze GPT-4o and Swallow, we find that biased output varies according to its contexts even with the equal combination of social attributes.

Intersectional Bias in Japanese Large Language Models from a Contextualized Perspective

In the current landscape of automatic language generation, there is a need to understand, evaluate, and mitigate demographic biases, as existing models are becoming increasingly multilingual. To address this, we present the initial eight languages from the Massive Multilingual Holistic Bias (MMHB) dataset and benchmark consisting of approximately 6 million sentences. The sentences are designed to induce biases towards different groups of people which can yield significant results when using them as a benchmark to test different text generation models. To further scale up in terms of both language coverage and size and to leverage limited human translation, we use systematic approach to independently translate sentence parts. This technique carefully designs a structure to dynamically generate multiple sentence variations and significantly reduces the human translation workload. The translation process has been meticulously conducted to avoid an English-centric perspective and include all necessary morphological variations for languages that require them, improving from the original English HOLISTICBIAS. Finally, we utilize MMHB to report results on gender bias and added toxicity in MT tasks.

Towards Massive Multilingual Holistic Bias

Bias in Natural Language Processing (NLP) applications has become a critical issue, with many methods developed to measure and mitigate bias in word embeddings and language models. However, most approaches focus on single categories such as gender or ethnicity, neglecting the intersectionality of biases, particularly in non-English languages. This paper addresses these gaps by studying both single-category and intersectional biases in Italian word embeddings and language models. We extend existing bias metrics to Italian, introducing GG-FISE, a novel method for detecting intersectional bias while accounting for grammatical gender. We also adapt the CrowS-Pairs dataset and bias metric to Italian. Through a series of experiments using WEAT, SEAT, and LPBS tests, we identify significant biases along gender and ethnic lines, with particular attention to biases against Romanian and South Asian populations. Our results highlight the need for culturally adapted methods to detect and address biases in multilingual and intersectional contexts.

Detecting Bias and Intersectional Bias in Italian Word Embeddings and Language Models

Large Language Models (LLMs) have been shown to be biased in prior work, as they generate text that is in line with stereotypical views of the world or that is not representative of the viewpoints and values of some demographic groups. In this work, we propose using data from parallel men's and women's events at the Olympic Games to investigate different forms of gender bias in language models. We define three different types of bias, and find that models are consistently biased against women when the gender is ambiguous in a prompt. In this case, the model frequently retrieves only results of the men's event with or without acknowledging it as such, revealing pervasive gender bias in LLMs in the context of athletics.

Sports and Women’s Sports: Gender Bias in a Generation with Olympic Data

The tendency for Natural Language Processing (NLP) technologies to reproduce stereotypical associations, such as associating Black people with criminality or women with care professions, is a site of major concern and, therefore, much study. Stereotyping is a powerful tool of oppression, but the social and linguistic mechanisms behind it are largely ignored in the NLP field. Thus, we fail to effectively challenge stereotypes and the power asymmetries they reinforce. This opinion paper problematizes several common aspects of current work addressing stereotyping in NLP, and offers practicable suggestions for potential forward directions.

Power(ful) Associations: Rethinking ``Stereotype'' for NLP

In recent years, various methods have been proposed to evaluate gender bias in large language models (LLMs). A key challenge lies in the transferability of bias measurement methods initially developed for the English language when applied to other languages. This work aims to contribute to this research strand by presenting five German datasets for gender bias evaluation in LLMs. The datasets are grounded in well-established concepts of gender bias and are accessible through multiple methodologies. Our findings, reported for eight multilingual LLM models, reveal unique challenges associated with gender bias in German, including the ambiguous interpretation of male occupational terms and the influence of seemingly neutral nouns on gender perception. This work contributes to the understanding of gender bias in LLMs across languages and underscores the necessity for tailored evaluation frameworks.

Exploring Gender Bias in Large Language Models: An In-depth Dive into the German Language

Reporting bias is the tendency for speakers to omit unnecessary or obvious information while mentioning things they consider relevant or surprising. In descriptions of people, reporting bias can manifest as a tendency to over report on attributes that deviate from the norm. While social bias in language models has garnered a lot of attention in recent years, a majority of the existing work equates "bias" with "stereotypes". We suggest reporting bias as an alternative lens through which to study how social attitudes manifest in language models. We present the MARB dataset, a diagnostic dataset for studying the interaction between social bias and reporting bias in language models. We use MARB to evaluate the off-the-shelf behavior of both masked and autoregressive language models and find signs of reporting bias with regards to marginalized identities, mirroring that which can be found in human text. This effect is particularly pronounced when taking gender into account, demonstrating the importance of considering intersectionality when studying social phenomena like biases.

Introducing MARB — A Dataset for Studying the Social Dimensions of Reporting Bias in Language Models

Gender-inclusive language is often used with the aim of ensuring that all individuals, regardless of gender, can be associated with certain concepts. While psycholinguistic studies have examined its effects in relation to human cognition, it remains unclear how Large Language Models (LLMs) process gender-inclusive language. Given that commercial LLMs are gaining an increasingly strong foothold in everyday applications, it is crucial to examine whether LLMs in fact interpret gender-inclusive language neutrally, because the language they generate has the potential to influence the language of their users. This study examines whether LLM-generated coreferent terms align with a given gender expression or reflect model biases. Adapting psycholinguistic methods from French to English and German, we find that in English, LLMs generally maintain the antecedent’s gender but exhibit underlying masculine bias. In German, this bias is much stronger, overriding all tested gender-neutralization strategies.

Adapting Psycholinguistic Research for LLMs: Gender-inclusive Language in a Coreference Context

Bias in Nepali NLP is rarely addressed, as the language is classified as low-resource, which leads to the perpetuation of biases in downstream systems. Our research focuses on gender bias in Nepali-English machine translation, an area that has seen little exploration. With the emergence of Large Language Models(LLM), there is a unique opportunity to mitigate these biases. In this study, we quantify and evaluate gender bias by constructing an occupation corpus and adapting three gender-bias challenge sets for Nepali. Our findings reveal that gender bias is prevalent in existing translation systems, with translations often reinforcing stereotypes and misrepresenting gender-specific roles. However, LLMs perform significantly better in both gender-neutral and gender-specific contexts, demonstrating less bias compared to traditional machine translation systems. Despite some quirks, LLMs offer a promising alternative for culture-rich, low-resource languages like Nepali. We also explore how LLMs can improve gender accuracy and mitigate biases in occupational terms, providing a more equitable translation experience. Our work contributes to the growing effort to reduce biases in machine translation and highlights the potential of LLMs to address bias in low-resource languages, paving the way for more inclusive and accurate translation systems.

Gender Bias in Nepali-English Machine Translation: A Comparison of LLMs and Existing MT Systems

Large language models (LLMs) often inherit and amplify social biases embedded in their training data. A prominent social bias is gender bias. In this regard, prior work has mainly focused on gender stereotyping bias – the association of specific roles or traits with a particular gender – in English and on evaluating gender bias in model embeddings or generated outputs. In contrast, gender representation bias – the unequal frequency of references to individuals of different genders – in the training corpora has received less attention. Yet such imbalances in the training data constitute an upstream source of bias that can propagate and intensify throughout the entire model lifecycle. To fill this gap, we propose a novel LLM-based method to detect and quantify gender representation bias in LLM training data in gendered languages, where grammatical gender challenges the applicability of methods developed for English. By leveraging the LLMs' contextual understanding, our approach automatically identifies and classifies person-referencing words in gendered language corpora. Applied to four Spanish-English benchmarks and five Valencian corpora, our method reveals substantial male-dominant imbalances. We show that such biases in training data affect model outputs, but can surprisingly be mitigated leveraging small-scale training on datasets that are biased towards the opposite gender. Our findings highlight the need for corpus-level gender bias analysis in multilingual NLP. We make our code and data publicly available.

Leveraging Large Language Models to Measure Gender Representation Bias in Gendered Language Corpora

Large Language Models (LLMs) offer promising alternatives to traditional occupational coding approaches in survey research. Using a German dataset, we examine the extent to which LLM-based occupational coding differs by gender. Our findings reveal systematic disparities: gendered job titles (e.g., “Autor” vs. “Autorin”, meaning “male author” vs. “female author”) frequently result in diverging occupation codes, even when semantically identical. Across all models, 54%–82% of gendered inputs obtain different Top-5 suggestions. The practical impact, however, depends on the model. GPT includes the correct code most often (62%) but demonstrates female bias (up to +18 pp). IBM is less accurate (51%) but largely balanced. Alibaba, Gemini, and MiniLM achieve about 50% correct-code inclusion, and their small (< 10 pp) and direction-flipping gaps could indicate a sampling noise rather than gender bias. We discuss these findings in the context of fairness and reproducibility in NLP applications for social data.

Mind the Gap: Gender-based Differences in Occupational Embeddings

Large Language Models (LLMs) have demonstrated impressive capabilities in reasoning tasks, yet their reliance on static prompt structures and limited adaptability to complex scenarios remains a major challenge. In this paper, we propose the **Deductive and Inductive (DID)** method, a novel framework that enhances LLM reasoning by dynamically integrating both deductive and inductive reasoning approaches. Drawing from cognitive science principles, DID implements a dual-metric complexity evaluation system that combines Littlestone dimension and information entropy to precisely assess task difficulty and guide decomposition strategies. DID enables the model to progressively adapt its reasoning pathways based on problem complexity, mirroring human cognitive processes. We evaluate DID's effectiveness across multiple benchmarks, including the AIW, MR-GSM8K, and our custom Holiday Puzzle dataset for temporal reasoning. Our results demonstrate great improvements in reasoning quality and solution accuracy - achieving 70.3% accuracy on AIW (compared to 62.2% for Tree of Thought), while maintaining lower computational costs.

The Role of Deductive and Inductive Reasoning in Large Language Models

Large Language Models (LLMs) have shown remarkable performance across diverse tasks without domain-specific training, fueling interest in their potential for time-series forecasting. While LLMs have shown potential in zero-shot forecasting through prompting alone, recent studies suggest that LLMs lack inherent effectiveness in forecasting. Given these conflicting findings, a rigorous validation is essential for drawing reliable conclusions. In this paper, we evaluate the effectiveness of LLMs as zero-shot forecasters compared to state-of-the-art domain-specific models. Our experiments show that LLM-based zero-shot forecasters often struggle to achieve high accuracy due to their sensitivity to noise, underperforming even simple domain-specific models. We have explored solutions to reduce LLMs' sensitivity to noise in the zero-shot setting, but improving their robustness remains a significant challenge. Our findings suggest that rather than emphasizing zero-shot forecasting, a more promising direction would be to focus on fine-tuning LLMs to better process numerical sequences. Our experimental code is available at https://github.com/junwoopark92/revisiting-LLMs-zeroshot-forecaster.

Revisiting LLMs as Zero-Shot Time Series Forecasters: Small Noise Can Break Large Models

Large language models (LLMs) have encoded vast amounts of knowledge in their parameters, but the acquired knowledge can sometimes be incorrect or outdated over time, necessitating rectification after pre-training. Traditional localized methods in knowledge-based model editing (KME) typically assume that knowledge is stored in particular intermediate layers. However, recent research suggests that these methods do not identify the optimal locations for parameter editing, as knowledge gradually accumulates across all layers in LLMs during the forward pass rather than being stored in specific layers. This paper, for the first time, introduces the concept of critical transmission paths into KME for parameter updating. Specifically, these paths capture the key information flows that significantly influence the model predictions for the editing process. To facilitate this process, we also design a parameter-aware contrastive rectifying algorithm that considers less important paths as contrastive examples. Experiments on two prominent datasets and three widely used LLMs demonstrate the superiority of our method in editing performance.

Parameter-Aware Contrastive Knowledge Editing: Tracing and Rectifying based on Critical Transmission Paths

Empirical risk minimization (ERM) incentivizes models to exploit shortcuts, i.e., spurious correlations between input attributes and labels that are prevalent in the majority of the training data but unrelated to the task at hand. This reliance hinders generalization on minority examples, where such correlations do not hold. Existing shortcut mitigation approaches are model-specific, difficult to tune, computationally expensive, and fail to improve learned representations. To address these issues, we propose InterpoLated Learning (InterpoLL) which interpolates the representations of majority examples to include features from intra-class minority examples with shortcut-mitigating patterns. This weakens shortcut influence, enabling models to acquire features predictive across both minority and majority examples. Experimental results on multiple natural language understanding tasks demonstrate that InterpoLL improves minority generalization over both ERM and state-of-the-art mitigation methods, without compromising accuracy on majority examples. Notably, these gains persist across encoder, encoder-decoder, and decoder-only architectures, demonstrating the method's broad applicability.

Mitigating Shortcut Learning with InterpoLated Learning

Reward models (RMs) play a crucial role in reinforcement learning from human feedback (RLHF), aligning model behavior with human preferences. However, existing benchmarks for reward models show a weak correlation with the performance of optimized policies, suggesting that they fail to accurately assess the true capabilities of RMs. To bridge this gap, we explore several evaluation designs through the lens of reward overoptimization, i.e., a phenomenon that captures both how well the reward model aligns with human preferences and the dynamics of the learning signal it provides to the policy. The results highlight three key findings on how to construct a reliable benchmark: (i) it is important to minimize differences between chosen and rejected responses beyond correctness, (ii) evaluating reward models requires multiple comparisons across a wide range of chosen and rejected responses, and (iii) given that reward models encounter responses with diverse representations, responses should be sourced from a variety of models. However, we also observe that a extremely high correlation with degree of overoptimization leads to comparatively lower correlation with certain downstream performance. Thus, when designing a benchmark, it is desirable to use the degree of overoptimization as a useful tool, rather than the end goal.

Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization

We present Sparse Interpolated Mixture-of-Experts (SIMoE) instruction-tuning, an end-to-end algorithm designed to fine-tune a dense pre-trained Large Language Model (LLM) into a MoE-style model that possesses capabilities in multiple specialized domains. During instruction-tuning, SIMoE automatically identifies multiple specialized experts under a specified sparsity constraint, with each expert representing a structurally sparse subset of the seed LLM's parameters that correspond to domain-specific knowledge within the data. SIMoE simultaneously learns an input-dependent expert merging strategy via a router network, leveraging rich cross-expert knowledge for superior downstream generalization that surpasses existing baselines. Empirically, SIMoE consistently achieves state-of-the-art performance on common instruction-tuning benchmarks while maintaining an optimal performance-compute trade-off compared to all baselines.

Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts

Knowledge distillation can be a cost-effective technique to distill knowledge in Large Language Models, if the teacher output logits can be pre-computed and cached. However, successfully applying this to pre-training remains largely unexplored. In this work, we prove that naive approaches for sparse knowledge distillation such as caching Top-K probabilities, while intuitive, provide biased estimates of teacher probability distribution to the student, resulting in suboptimal performance and calibration. We propose an importance-sampling-based method 'Random Sampling Knowledge Distillation', which provides unbiased estimates, preserves the gradient in expectation, and requires storing significantly sparser logits. Our method enables faster training of student models with marginal overhead (<10%) compared to cross-entropy based training, while maintaining competitive performance compared to full distillation, across a range of model sizes from 300M to 3B.

A Sample Offline Saves Time: Knowledge Distillation in the LLM Era

A range of recent works addresses the problem of compression of sequence of tokens into a shorter sequence of real-valued vectors to be used as inputs instead of token embeddings or key-value cache. These approaches are focused on reduction of the amount of compute in existing language models rather than minimization of number of bits needed to store text. Despite relying on powerful models as encoders, the maximum attainable lossless compression ratio is typically not higher than x10. This fact is highly intriguing because, in theory, the maximum information capacity of large real-valued vectors is far beyond the presented rates even for 16-bit precision and a modest vector size. In this work, we explore the limits of compression by replacing the encoder with a per-sample optimization procedure. We show that vectors with compression ratios up to x1500 exist, which highlights two orders of magnitude gap between existing and practically attainable solutions. Furthermore, we empirically show that the compression limits are determined not by the length of the input but by the amount of uncertainty to be reduced, namely, the cross-entropy loss on this sequence without any conditioning. The obtained limits highlight the substantial gap between the theoretical capacity of input embeddings and their practical utilization, suggesting significant room for optimization in model design.

Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity

We introduce TESS 2, a general instruction-following diffusion language model that outperforms contemporary instruction-tuned diffusion models, as well as matches and sometimes exceeds strong autoregressive (AR) models. We train TESS 2 by first adapting a strong AR model via continued pretraining with a diffusion loss and then performing further instruction tuning. We find that adaptation training as well as the choice of the base model is crucial for training good instruction-following diffusion models. We further propose reward guidance, a novel and modular inference-time guidance procedure to align model outputs without needing to train the underlying model. Finally, we show that TESS 2 further improves with increased inference-time compute, highlighting the utility of diffusion LMs in having fine-grained controllability over the amount of compute used at inference time.

SAD-LM: A Large-Scale Generalist Diffusion Language Model

Large language models (LLMs) struggle with maintaining accurate knowledge due to conflicting/outdated parametric memories. While locate-and-edit methods address this, their reliance on models' internal representations leads to robustness failures in long-context reasoning and paraphrased queries. We identify a fundamental limitation of locate-and-edit methods: existing semantic keys (for memory localization) cannot simultaneously satisfy robustness (context-invariant activation) and specificity (precise knowledge discrimination). Through theoretical error-bound analysis, we establish formal criteria for effective editing.
Our solution introduces \textit{Robust Edit Pathway (REP)}, a plug-and-play module that: (1) disentangles editing keys from native model representations; (2) dynamically adjusts keys via contrastive learning to achieve robustness-specificity balance. 
Extensive experiments across various editing methods (ROME/MEMIT/R-ROME/EMMET), existing LLMs (LLaMA2, QWen, Mistral), and datasets (CounterFact, ZsRE) show that REP improves success rate over robustness tests by up-to 66.4\% while maintaining the success rate unaffected.

Keys to Robust Edits: From Theoretical Insights to Practical Advances

In this study, we provide constructive proof that Transformers can recognize and generate hierarchical language efficiently with respect to model size, even without the need for a specific positional encoding.
Specifically, we show that causal masking and a starting token enable Transformers to compute positional information and depth within hierarchical structures.
We demonstrate that Transformers without positional encoding can generate hierarchical languages. 
Furthermore, we suggest that explicit positional encoding might have a detrimental effect on generalization with respect to sequence length.

Theoretical Analysis of Hierarchical Language Recognition and Generation by Transformers without Positional Encoding

Understanding human preferences is crucial for improving foundation models and building personalized AI systems. However, preferences are inherently diverse and complex, making it difficult for traditional reward models to capture their full range. While fine-grained preference data can help, collecting it is expensive and hard to scale. In this paper, we introduce Decomposed Reward Models (DRMs), a novel approach that extracts diverse human preferences from binary comparisons without requiring fine-grained annotations. Our key insight is to represent human preferences as vectors and analyze them using Principal Component Analysis (PCA). By constructing a dataset of embedding differences between preferred and rejected responses, DRMs identify orthogonal basis vectors that capture distinct aspects of preference. These decomposed rewards can be flexibly combined to align with different user needs, offering an interpretable and scalable alternative to traditional reward models. We demonstrate that DRMs effectively extract meaningful preference dimensions (e.g., helpfulness, safety, humor) and adapt to new users without additional training. Our results highlight DRMs as a powerful framework for personalized and interpretable LLM alignment.

Rethinking Diverse Human Preference Learning through Principal Component Analysis

As artificial intelligence surpasses human capabilities in text generation, the necessity to authenticate the origins of AI-generated content has become paramount. Unbiased watermarks offer a powerful solution by embedding statistical signals into language model-generated text without distorting the quality. In this paper, we introduce MCmark, a family of unbiased, Multi-Channel-based watermarks. MCmark works by partitioning the model's vocabulary into segments and promoting token probabilities within a selected segment based on a watermark key. We demonstrate that MCmark not only preserves the original distribution of the language model but also offers significant improvements in detectability and robustness over existing unbiased watermarks. Our experiments with widely-used language models demonstrate an improvement in detectability of over 10% using MCmark, compared to existing state-of-the-art unbiased watermarks. This advancement underscores MCmark's potential in enhancing the practical application of watermarking in AI-generated texts.

Improved Unbiased Watermark for Large Language Models

State space models (SSMs) achieve efficient sub-quadratic compute complexity but often exhibit significant performance drops as context length increases. Recent work attributes this deterioration to an exponential decay in hidden-state memory. While token filtering has emerged as a promising remedy, its underlying rationale and limitations remain largely non-understood. In this paper, we first investigate the attention patterns of Mamba to shed light on why token filtering alleviates long-context degradation. Motivated by these findings, we propose LAMB, a training-free, attention-guided token filtering strategy designed to preserve critical tokens during inference. LAMB can boost long-context performance for both pure SSMs and hybrid models, achieving up to an average improvement of 30.35% over state-of-the-art techniques on standard long-context understanding benchmarks. Our analysis and experiments reveal new insights into the interplay between attention, token selection, and memory retention, and are thus expected to inspire broader applications of token filtering in long-sequence modeling.

LAMB: A Training-Free Method to Enhance the Long-Context Understanding of SSMs via Attention-Guided Token Filtering

Automated Theorem Proving is an important and challenging task. Although large language models (LLMs) have demonstrated remarkable potential in mathematical reasoning, their performance in formal theorem proving remains constrained by the scarcity of high-quality supervised fine-tuning (SFT) data. To address this limitation, we propose a **Q**uality-**D**riven **T**heorem **S**ynthesis method (QDTSynth) in Lean4. During the statement synthesis, we enhance Monte Carlo Tree Search (MCTS) with an adaptive adjustment mechanism that dynamically optimizes the search strategy based on the synthesis of statements. In addition, we propose diversity screening and the self-assessment method to select theorems that exhibit both diversity and high quality from the initially synthetic statements, enabling the synthesis of a high-quality Lean4 theorem dataset. After fine-tuning three open-source large language models on our synthetic dataset, experiments on the miniF2F benchmark demonstrate that QDTSynth significantly improves the performance of various open-source LLMs in theorem proving tasks. Our work offers a promising new direction for the future synthesis of high-quality formal mathematical theorems.

QDTSynth: Quality-Driven Formal Theorem Synthesis for Enhancing Proving Performance of LLMs

Tasks are central in machine learning, as they are the most natural objects to assess the capabilities of current models. The trend is to build general models able to address any task. Even though transfer learning and multitask learning try to leverage the underlying task space, no well-founded tools are available to study its structure. This study proposes a theoretically grounded setup to define the notion of task and to compute the inclusion between two tasks from a statistical deficiency point of view. We propose a tractable proxy as information sufficiency to estimate the degree of inclusion between tasks, show its soundness on synthetic data, and use it to reconstruct empirically the classic NLP pipeline.

Underline digital video library

Browse The Future of AI Research

Browse AAAI 2026 Invited Talks

Browse Underline Originals

No results!

Browse Natural Language Processing

Browse Machine Learning

Stay up to date with the latest Underline news!

PRODUCT

COMPANY

RESOURCES