Semi-Supervised Multi-Modal Clustering and Classification with Incomplete Modalities

Yang Yang, De Chuan Zhan, Yi Feng Wu, Zhi Bin Liu, Hui Xiong, Yuan Jiang

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


In this paper, we propose a novel Semi-supervised Learning with Incomplete Modality (SLIM) method considering the modal consistency and complementarity simultaneously, and Kernel SLIM (SLIM-K) based on matrix completion for further solving the modal incompleteness. As is well known, most realistic data have multi-modal representations, multi-modal learning refers to the process of learning a precise model for complete modalities. However, due to the failures of data collection, self-deficiencies, or other various reasons, multi-modal examples are usually with incomplete modalities, which generate utility obstacle using previous methods. In this paper, SLIM integrates the intrinsic consistency and extrinsic complementary information for prediction and cluster simultaneously. In detail, SLIM forms different modal classifiers and clustering learner consistently in a unified framework, while using the extrinsic complementary information from unlabeled data against the insufficiencies brought by the incomplete modal issue. Moreover, in order to deal with missing modality in essence, we propose the SLIM-K, which takes the complemented kernel matrix into the classifiers and the cluster learner respectively. Thus, SLIM-K can solve the defects of missing modality in result. Finally, we give the discussion of generalization of incomplete modalities. Experiments on 13 benchmark multi-modal datasets and two real-world incomplete multi-modal datasets validate the effectiveness of our methods.

Original languageEnglish (US)
Article number8786151
Pages (from-to)682-695
Number of pages14
JournalIEEE Transactions on Knowledge and Data Engineering
Issue number2
StatePublished - Feb 1 2021
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Science Applications
  • Computational Theory and Mathematics


  • Semi-supervised learning
  • incomplete multi-modal learning
  • matrix completion
  • modal complementarity
  • modal consistency


Dive into the research topics of 'Semi-Supervised Multi-Modal Clustering and Classification with Incomplete Modalities'. Together they form a unique fingerprint.

Cite this