Existing models of facial identity perception often assume that information conveyed by facial stimuli may ultimately determine perceptual judgments. Capitalizing on sequential effects, the present study investigates whether facial identity is judged relative to the context shaped by stimuli presented in previous trials. When categorizing a sequence of facial identities, the results demonstrated that the participants’ categorization of the current faces varied according to the local sequential context provided by the immediately preceding faces and this variation depended on the relative distance between these two successive faces. Notably, the identity-based sequential effects occurred when the female participants categorized only female faces and the male participants categorized faces of both genders. Moreover, the mechanisms underlying the sequential effects were qualitatively different in the male and female participants. The female participants tended to respond the current faces with the same category label as on the preceding faces, whereas the males tended to respond with a different label. The present study thus demonstrates that the relative information between the preceding and current faces may be used as evidence to inform a judgment. Furthermore, this process is not unitary, but multifaceted, depending on the gender of the faces and participants.