We placed observers' interpersonal communication-cognition as a social skill and investigated the mechanism behind it. From the result of Study 1, although ability in face-to-face approaches influenced the accuracy of an observer's judgment of interpersonal communication in highly expressive conversations that were easy to judge, this did not occur in low-expressive conversations that were difficult to judge, suggesting a relationship between the two. In addition, to examine clues for improvement in accuracy, we conducted a lens model analysis in Study 1. Interactants' judgments about conversations were positively correlated to speeches and negatively correlated to adapters, while those of observers were positively correlated to speeches, gestures, and smiles, resulting in asymmetry of interpersonal communication-cognition between interactants and observers. In Study 2, a series of observational experiments showed the possibility of improvement in accuracy by skill training. These results suggested the validity of the placement of observers' interpersonal communication-cognition as a social skill and helped to explain some part of its mechanism.