Lip reading systems infer linguistic content by analyzing a speaker’s lip movements，relying entirely on visual modality compared with traditional speech recognition，thus facing greater uncertainty due to missing modalities and temporal variations. Although recent advances in lip motion modeling and semantic alignment have driven technological progress，existing methods predominantly focus on generic modeling while neglecting the critical impact of individual differences. In practical applications，considerable performance variations across speakers remain a major bottleneck for cross-scenario generalization，making speaker adaptation a core challenge in lip reading research. Current speaker adaptation approaches fall into two categories： 1） generalization strategies without target speaker data，such as constructing speaker-invariant representations or employing visual i-vectors for cross-speaker modeling，and 2） personalized strategies with limited target data，including fine-tuning，model pruning，parameter-efficient prompt learning，and contrastive learning. Recent studies have also explored vision-language collaborative mechanisms to enhance individual semantic and visual style perception. However，existing methods still inadequately address the explicit decoupling and utilization of speaker features under zero-resource conditions，particularly failing to distinguish the differential impacts of static appearance features and dynamic behavioral patterns on lip motion modeling. This study aims to tackle this zero-resource adaptation problem by proposing a novel speaker-adaptive lip reading method based on memory-guided prompting. The proposed method seeks to enhance the model’s ability to generalize across unseen speakers without accessing any speaker-specific adaptation data. To evaluate systematically the performance of lip reading models in speaker generalization scenarios，this study constructs a new benchmark dataset，LRS2-ID，on the basis of the public LRS2 dataset. LRS2-ID not only preserves the diversity of the original corpus but also introduces speaker identity-controlled annotations to support separate modeling and evaluation of “seen/unseen speakers”，providing a practical benchmark for assessing zero-shot adaptation capabilities.

In this study，we introduce an explicit memory prompting framework that leverages structured visual speaker prototypes as external guidance during training and inference. Specifically，the framework constructs two types of speaker-specific memory banks： a static visual pattern memory and a dynamic visual speech pattern memory. Static visual patterns capture facial appearance and are obtained by averaging face features over multiple representative frames. Dynamic speech patterns are derived from frame-level features in the frequency domain，emphasizing interframe variations while suppressing static components. Negative samples are introduced for contrastive learning，where shuffled and reversed sequences generate temporally incorrect samples，training the model to identify correct speaker-specific dynamic patterns. Additionally，a gradient reversal layer is incorporated at the frame level to suppress the model’s reliance on static appearance features during backpropagation，further enhancing dynamic pattern recognition. Static and dynamic patterns are clustered into prototype memory banks using K-means clustering，ensuring compactness and scalability. During inference，when encountering an unseen speaker，the model retrieves relevant prototypes from these memory banks based on similarity with the input sequence. The retrieved prototypes are then injected into the attention mechanism of a Conformer-based encoder as key and value components. A hierarchical injection strategy is adopted： Static prototypes are fused into shallow encoder layers to enhance appearance modeling，while dynamic prototypes are injected into deeper layers to guide temporal behavior modeling. This cross-layer prompting scheme allows the model to align and adapt to the unseen speaker’s patterns dynamically without modifying model parameters or requiring supervised adaptation.

Comprehensive experiments on the LRS2-ID dataset demonstrate the effectiveness of the proposed method. Results show that the method reduces the word error rate by over 4.3% absolutely，without requiring target speaker adaptation data. Ablation studies confirm the contributions of each component： Static prompting alone aids stable appearance modeling，particularly for speakers with consistent visual traits，while dynamic prompting captures personalized speech behavior and benefits speakers with diverse visual conditions. The fusion of both prompt types，combined with structured memory organization，consistently enhances model robustness across varying speaker profiles. Per-speaker performance analysis reveals consistent improvements across most speakers，including those with considerable intervideo appearance changes or high articulation variability. Visualization of representative samples highlights the complementary strengths of static and dynamic memory prompts，particularly in challenging scenarios involving illumination shifts，pose variations，or expressive speech.

This study presents a novel memory-based prompting framework for speaker-adaptive lip reading in zero-resource conditions. By explicitly modeling static and dynamic visual speaker patterns and integrating them hierarchically into the attention mechanism of a Conformer-based encoder，the proposed method achieves robust generalization to unseen speakers without requiring adaptation data. Experimental results on the LRS2-ID benchmark confirm the method’s effectiveness，stability，and practical value. This study advances structured prompting and memory-enhanced adaptation，paving the way for scalable and user-friendly lip reading systems capable of generalizing across diverse real-world users. Future work may explore finer-grained adaptive mechanisms and additional modalities to further enhance its performance.