Context-sensitive conditional ordinal random fields for facial action intensity estimation

We address the problem of modeling intensity levels of facial actions in video sequences. The intensity sequences often exhibit a large variability due to the context factors, such as the person-specific facial expressiveness or changes in illumination. Existing methods usually attempt to normalize this variability in data using different feature-selection and/or data pre-processing schemes. Consequently, they ignore the context in which the target facial actions occur. We propose a novel Conditional Random Field (CRF) based ordinal model for context-sensitive modeling of the facial action unit intensity, where the W5+ (Who, When, What, Where, Why and How) definition of the context is used. In particular, we focus on three contextual questions: Who (the observed person), How (the changes in facial expressions), and When (the timing of the facial expression intensity). The contextual questions Who and How are modeled by means of the newly introduced covariate effects, while the contextual question When is modeled in terms of temporal correlation between the intensity levels. We also introduce a weighted softmax-margin learning of CRFs from the data with a skewed distribution of the intensity levels, as commonly encountered in spontaneous facial data. The proposed model is evaluated for intensity estimation of facial action units and facial expressions of pain from the UNBC Shoulder Pain dataset. Our experimental results show the effectiveness of the proposed approach.