Copula ordinal regression framework for joint estimation of facial action unit intensity

Robert Walecki, Ognjen Rudovic, Vladimir Pavlovic, Maja Pantic

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


—Joint modeling of the intensity of multiple facial action units (AUs) from face images is challenging due to the large number of AUs (30+) and their intensity levels (6). This is in part due to the lack of suitable models that can efficiently handle such a large number of outputs/classes simultaneously, but also due to the lack of suitable data the models on. For this reason, majority of the methods resort to independent classifiers for the AU intensity. This is suboptimal for at least two reasons: the facial appearance of some AUs changes depending on the intensity of other AUs, and some AUs co-occur more often than others. To this end, we propose the Copula regression approach for modeling multivariate ordinal variables. Our model accounts for ordinal structure in output variables and their non-linear dependencies via copula functions modeled as cliques of a conditional random fields. The copula ordinal regression model achieves the joint learning and inference of intensities of multiple AUs, while being computationally tractable. We demonstrate the effectiveness of our approach on three challenging datasets of naturalistic facial expressions and we show that the estimation of target AU intensities improves especially in the case of (a) noisy image features, (b) head-pose variations and (c) imbalanced training data. Lastly, we show that the proposed approach consistently outperforms (i) independent modeling of AU intensities and (ii) the state-of-the-art approach for the target task and (iii) deep convolutional neural networks.

Original languageEnglish (US)
Article number2728534
Pages (from-to)297-312
Number of pages16
JournalIEEE Transactions on Affective Computing
Issue number3
StatePublished - Jul 18 2019

All Science Journal Classification (ASJC) codes

  • Software
  • Human-Computer Interaction


  • Action unit intensity
  • Conditional random fields
  • Facial expression analysis
  • Ordinal regression
  • Spontaneous facial behavior

Fingerprint Dive into the research topics of 'Copula ordinal regression framework for joint estimation of facial action unit intensity'. Together they form a unique fingerprint.

Cite this