This study reports on the recognition of different arousal levels, elicited by affective sounds, performed using estimates of autonomic nervous system dynamics. Specifically, as a part of the circumplex model of affect, arousal levels were recognized by properly combining information gathered from standard and nonlinear analysis of heartbeat dynamics, which was derived from the electrocardiogram (ECG). Affective sounds were gathered from the International Affective Digitized Sound System and grouped into four different levels of arousal. A group of 27 healthy volunteers underwent such elicitation while ECG signals were continuously recorded. Results showed that a quadratic discriminant classifier, as applied implementing a leave-one-subject-out procedure, achieved a recognition accuracy of 84.26%. Moreover, this study confirms the crucial role of heartbeat nonlinear dynamics for emotion recognition, hereby estimated through lagged Poincaré plots.