Research Project

LAMR: Layer Roles Matter

Mitigating Modality Imbalance in Multimodal Emotion Reasoning via Adaptive Layer Profiling

Pengyu HuangPeipei Song^*Beier Zhu^*Zhiyuan HanJinpeng HuXun Yang^*

University of Science and Technology of China · Hefei University of Technology

^* Corresponding authors

Overview

TL;DR: We identify two complementary modality-imbalance failures in multimodal emotion reasoning, profile the functional roles of Transformer layers, and apply role-conditioned attention rebalancing at inference time—without additional training.

Multimodal emotion reasoning requires models to combine linguistic, acoustic, and visual evidence. Existing multimodal large language models often over-rely on a dominant modality while under-utilizing interactions across modalities, producing incomplete or misleading emotional interpretations.

We introduce Layer-Aware Modality Rebalancing (LAMR). Modality Entropy and Cross-modal Intensity adaptively partition the model into stabilization and integration layers. LAMR then balances modality participation in stabilization layers and promotes relevant cross-modal propagation in integration layers. Experiments on MER-UniBench and MME-Emotion show consistent gains across PandaGPT, Emotion-LLaMA, and AffectGPT, with only 3.5% inference overhead.

1 · Motivation — A Two-Sided Attention Dilemma

Before designing a fix, we look inside three popular MER models and find that the failure is structural, not accidental. Models collapse onto a single dominant modality and, simultaneously, leave most cross-modal interactions under-attended. The two failures co-exist at every layer and are anti-correlated across depth—layers with small modality gaps tend to show weak cross-modal attention, and vice versa.

Diagnostic findings: attention statistics, an erroneous reasoning example, two balance dilemma, and layer role heterogeneity — **Figure 1.** (a) Attention statistics reveal that the model collapses onto a dominant modality while under-using cross-modal interactions. (b) This produces flawed emotional readings—e.g. confusing *urgency* with neutral advice. (c) The same two-sided dilemma holds at every layer. (d) Cross-modal attention and modality entropy are anti-correlated across depth, exposing the stabilization / integration layer split.

2 · Method — Layer-Aware Modality Rebalancing

Inspired by the layer-wise heterogeneity above, we split the model into two functional groups: stabilization layers that clean up single-modality features, and integration layers that fuse cross-modal semantics. We profile each layer with two metrics, Modality Entropy (ME) and Cross-modal Intensity (CMI), then apply a different intervention to each group: Modality Anchoring Equilibrium on stabilization layers, and Intermodal Diffusive Propagation on integration layers. No retraining, no extra parameters.

Overview of adaptive layer role profiling and layer-wise attention rebalancing in LAMR — **Figure 2.** Left (Sec. 3.3): ME and CMI are combined into a unified layer score that partitions the MLLM into stabilization and integration layers. Right (Sec. 3.4): stabilization layers receive modality anchoring equilibrium, integration layers receive intermodal diffusive propagation.

3 · Qualitative Results

Does the rebalance actually change the output? On two representative scenes, AffectGPT reads the speaker's tone as enthusiastic endorsement or casual teasing, while AffectGPT + LAMR grounds the same scenes in cross-modal evidence and recovers the intended dissatisfaction / bitterness.

Qualitative comparison of AffectGPT vs AffectGPT+Ours on two emotion reasoning cases — **Figure 3.** Top: AffectGPT reads the audio cue as “enthusiastic endorsement” / “casual teasing”, missing the actual *dissatisfaction / bitterness* carried by the speaker's tone. Bottom: AffectGPT + LAMR correctly grounds the same scenes in cross-modal evidence and recovers the intended emotion.

4 · Cross-Backbone Results

LAMR is backbone-agnostic. The same recipe applied to AffectGPT, PandaGPT, and EmotionLLaMA consistently enlarges under-used cross-modal axes (top row, 9 pairwise attention types) and pulls the modality triangle (bottom row, Text / Audio / Video balance) toward a more balanced shape—with only 3.5% inference overhead.

Radar charts comparing nine pairwise attention types and three modality balances across AffectGPT, PandaGPT and EmotionLLaMA, with and without LAMR — **Figure 4.** Top row: pairwise T–T, T–A, T–V, A–A, A–T, A–V, V–V, V–T, V–A attention. Bottom row: Text / Audio / Video balance. Solid = baseline MLLM, dashed = **+ LAMR**. Across all three backbones, LAMR consistently enlarges the under-used cross-modal axes and pulls the modality triangle toward a more balanced shape.

BibTeX

@inproceedings{huang2026lamr,
  title     = {Layer Roles Matter: Mitigating Modality Imbalance in
               Multimodal Emotion Reasoning via Adaptive Layer Profiling},
  author    = {Huang, Pengyu and Song, Peipei and Zhu, Beier and
               Han, Zhiyuan and Hu, Jinpeng and Yang, Xun},
  booktitle = {Proceedings of the ACM Multimedia Conference},
  year      = {2026},
  publisher = {ACM}
}