Depression is the leading cause of disability in the world, with over 350 million people affected worldwide. To address this, many researchers have developed computer-based therapy systems to treat this condition. However, many of these systems have been shown to have significantly worse retention than face-to-face counseling, limiting their effectiveness. We believe this discrepancy is partially due to these systems’ inability to respond to a user’s emotional state in real time.

We have developed an affectively aware virtual agent for depression counseling: an animated computer character that responds to users’ affective states in real time during virtual therapy sessions. The virtual agent guides users through a manualized cognitive behavioral therapy (CBT) intervention, while sensing users’ affect from their facial expressions and changes in speech prosody. During this simulated face-to-face conversation, the agent provides real-time tailored responses in response to sensed user affect. The automatically generated responses corresponded to empathic statements that allowed participants to pause the interaction in order to calm and compose themselves and reflective statements that rephrased emotionally sensitive content elicited from participants to emulate empathic listening.

We conducted a quasi-experimental pilot study to assess acceptance of the system by individuals with mild to moderate depression. Ten individuals (5 male, 5 female, aged 18-28) participated in a single 30-minute CBT-based session with the affectively aware virtual agent. We measured depression (Patient Health Questionnaire-9, PHQ-9) and state anxiety before and after the session and self-reported attitudes towards the system on scale measures and via a semi-structured interview.

Participants scored between 5 and 15 (mean=6.6, SD=2.7) on baseline depression measures (PHQ-9). No significant changes were found in pre-post tests for depression (pre-interaction mean=8.0, SD=4.62, post-interaction mean=6.4, SD=3.13) or anxiety (pre-interaction mean=23.3, SD=6.46, post-interaction mean=21.78, SD=7.61), although both trended towards improvement. Agent ratings were generally neutral across the board, with satisfaction (mean=4.5, SD=1.35), desire to continue using (mean=4.2, SD=1.6), trust (mean=3.9, SD=1.66), and likability (mean=4.4, SD=1.7) scoring around the mid-point. Thematic analysis of the semi-structured interviews indicated that 50% of participants reported that the agent evoked emotional responses in them during the interactions and that 70% of participants felt the agent understood their emotional state. Participants stated that this was caused by the agent presenting them with “information they did not realize” (Patient 5) and that the agent “… understood my emotions because I felt that it gave me the right responses” (Patient 8).

This study suggest that it is possible for a virtual agent to evoke and respond to a user’s emotional state in real time during counseling sessions. Although we did not design this study to produce significant changes in depressive symptoms due to its size and duration, the majority of users expressed that they felt the agent understood their emotional state and responded appropriately.