Diffusion models have rapidly become the dominant paradigm of generative visual models. However,the memorization-forgetting mechanisms inherent in these models make them unintentionally memorize sensitive information from training datasets,which further aggravates the privacy and copyright issues in the image domain. Although the memorization and forgetting mechanisms have been deeply studied in the field of language models,a systematic review on diffusion models,which are the core technology of visual generation tasks,is still lacking. Our survey aims to bridge this gap,and the existing works are reviewed from five aspects critically. First,we briefly introduce the theoretical modeling and architecture of diffusion models,including the formulations of denoising diffusion probabilistic models,score-based generative models,and flow matching,as well as the architectures of U-Net,DiT,and MMDiT. Such theoretical and architectural foundations not only support the strong generative capability of diffusion models but also provide the necessary preliminaries for understanding why and how memorization arises during the iterative denoising process. On this basis,the definitions of memorization in diffusion models are introduced from non-temporal and temporal perspectives. For non-temporal diffusion models,the memorization can be divided into global memorization and local memorization,where the former focuses on the global similarity between generated and training images,and the latter is concerned with the partial replication of sensitive components such as faces,signatures,and watermarks. For temporal diffusion models,the memorization can be further decomposed into content memorization and motion memorization,given that the leakage in video diffusion models is reflected not only in the static appearance of objects but also in the dynamic motion patterns across frames. Once the manifestations of memorization are formally defined,a natural follow-up question is what factors cause such memorization behaviors and how they can be theoretically interpreted. Accordingly,the understanding of memorization in diffusion models is then discussed from the perspectives of model factor,data factor,and theoretical view. The model factor is focused on the influence of over-parameterization and architectural components such as the cross-attention modules,while the data factor reveals that the long-tailed distribution and duplicated samples can significantly increase the memorization risk. The theoretical view is supported by the manifold memorization hypothesis and the geometry-adaptive harmonic representation,which explain why diffusion models tend to memorize specific samples from a geometric standpoint. Building upon these qualitative understandings,more quantitative tools are required for the practical assessment of memorization risks. To this end,the quantification methods of memorization are introduced from the perspectives of a model auditor and a malicious attacker. The auditor-based methods can be divided into proxy- and replication-based methods. The proxy-based methods are designed to approximate the memorization score by influence functions,gradient variance,and geometric proxies such as the Hessian curvature. The replication-based methods utilize pixel-,perceptual-,and semantic-level similarity metrics,such as structural similarity index measure(SSIM),learned perceptual image patch similarity(LPIPS),self-supervised copy detection (SSCD),and contrastive language-image pretraining(CLIP),to detect the duplication between generated images and training images. The attacker-based methods can be divided into membership inference and extraction attacks,which are designed to determine whether a sample is in the training dataset or to recover the original training samples directly. The threat models can be further categorized into white-,gray-,and black-box settings according to the attacker’s knowledge of the target model. Given that these quantification methods have empirically verified the severity of memorization-induced privacy leakage,how to effectively mitigate such risks becomes the next critical concern. Therefore,the memorization mitigation methods under a unified framework of machine unlearning are introduced,including 1) differential privacy,2) prompt optimization,and 3) machine unlearning. The differential privacy-based methods are typically applied in the pre-training stage and can provide formal privacy guarantees through input perturbation,output perturbation,or gradient perturbation such as DP-SGD,but at the cost of generative utility. The prompt optimization-based methods intervene at the inference stage by perturbing the text embedding or attention scores to prevent the model from generating memorized content,which is lightweight but less rigorous. The machine unlearning-based methods can be focused on fine-tuning the model parameters to redirect the memorized concepts to null concepts,locating and editing the memorization-related neurons,or applying closed-form parameter editing to remove the influence of specific samples or concepts in a post-hoc manner. The three families of methods are complementary to one another in terms of intervention stage,computational overhead,and privacy guarantee,and their integration provides a more comprehensive perspective on memorization mitigation in diffusion models. In summary,although considerable progress has been made along the abovementioned five aspects,several critical challenges still remain to be addressed in future research. 1) The privacy data processing pipeline and benchmarks of diffusion models are required to be standardized further,given that the absence of unified datasets and evaluation protocols hampers fair comparison and reproducibility. 2) The memorization definitions and machine unlearning algorithms that are more compatible with the characteristics of diffusion models are required to be developed,considering that the existing methods are mostly inherited from classifiers or language models and cannot fully exploit the iterative and cross-modal nature of diffusion processes. 3) The memorization-forgetting mechanisms in novel learning scenarios such as test-time training,federated learning,and continual learning are required to be explored,and the deployment in vertical domains such as medical imaging and financial generation is required to be accelerated for the cooperation of academia and industry. Furthermore,the data privacy policy-relevant ethical issues need to be considered for the responsible deployment of diffusion models in the future.
| 科 Family | 属数 Number of genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) | 属 Genus | 种数 Number of species | 占总种数比例 Percentage of total species (%) |
|---|---|---|---|---|---|---|
| 鹅膏菌科Amanitaceae | 2 | 11 | 5.26 | 鹅膏菌属 Amanita | 10 | 4.78 |
| 小菇科 Mycenaceae | 2 | 12 | 5.74 | 丝盖伞属 Inocybe | 5 | 2.39 |
| 多孔菌科 Polyporaceae | 8 | 14 | 6.70 | 蜡蘑属 Laccaria | 5 | 2.39 |
| 红菇科 Russulaceae | 3 | 23 | 11.00 | 小皮伞属 Marasmius | 6 | 2.87 |
| 小菇属 Mycena | 11 | 5.26 | ||||
| 光柄菇属 Pluteus | 5 | 2.39 | ||||
| 红菇属 Russula | 17 | 8.13 | ||||
| 栓菌属 Trametes | 5 | 2.39 |