Inferring the motivations of others is a fundamental aspect of social interaction. However, making such inferences about infants can be challenging. This investigation examined adults' ability to infer the eliciting event of an infant's behavior and what information adults utilize to make such inferences. In Study 1, adult participants viewed recordings of 24-month-old infants responding to an actor's emotional display (joy, sadness, fear, anger, or disgust) toward a broken toy and were asked to infer which emotion the actor expressed using only the infant's behavioral responses. Importantly, videos were blurred and muted to ensure that the only information available regarding the actor's emotion was the infant's reaction. Overall, adults were poor judges of the elicitors of infants' behaviors with accuracy levels below 50%. However, adults' categorizations appeared systematic, suggesting that they may have used consistently miscategorized emotions. To explore this possibility, a second study was conducted in which a separate sample of adults viewed the original recordings and were asked to identify infants' goal-directed behaviors (i.e., security seeking, social avoidance, information seeking, prosocial behavior, exploration, relaxed play). Overall, adults perceived a variety of infant differentiated responses to discrete emotions. Furthermore, infants' goal-directed behaviors were significantly associated with adults' earlier "miscategorizations." Infants who responded with specific behaviors were consistently categorized as having responded to specific emotions, such as prosocial behavior in response to sadnesss. Taken together, these results suggest that when explicit emotion information is unavailable, adults may use heuristics of emotional responsiveness to guide their categorizations of emotion elicitors.