Previous studies on attribution judgments concerning crime victims have commonly used the term "responsibility" to measure the negative implications regarding victims. However, responsibility is a concept that should be placed upon offenders, not victims. Victims have frequently been judged negatively, but the use of "responsibility" potentially inhibits the accurate understanding of such negative implications. Additionally, in judicial practice, "responsibility" is basically a term attributed to offenders. We therefore observed a certain shortcoming in the current research framework attributing responsibility to victims. Through judicial decisions and interviews with victims, we derived other labels supposedly containing negative victim judgments ("carelessness" and "fault") , and, together with the label "responsibility," considered whether people evaluate the victims using such labels. Moreover, to confirm whether these labels point to qualitatively distinct concepts, we examined their relationships with causal attribution. The results revealed that respondents rated the victim significantly lower on responsibility than the other negative labels, and we also found different prognostic factors for the labels. The implications of the study were discussed.