Humans are much better at cooperation than simple evolutionary models explain. Some, like Christopher Boehm, have suggested that cooperative punishment of defectors - rule breakers, liars, psychopaths, those who don't play together well, etc - is the key explanatory principle. Ernst Fehr and Simon Gachter, writing in Nature, claim to have experimental results demonstrating this. Their abstract:
Human cooperation is an evolutionary puzzle. Unlike other creatures, people frequently cooperate with genetically unrelated strangers, often in large groups, with people they will never meet again, and when reputation gains are small or absent. These patterns of cooperation cannot be explained by the nepotistic motives associated with the evolutionary theory of kin selection and the selfish motives associated with signalling theory or the theory of reciprocal altruism. Here we show experimentally that the altruistic punishment of defectors is a key motive for the explanation of cooperation. Altruistic punishment means that individuals punish, although the punishment is costly for them and yields no material gain. We show that cooperation flourishes if altruistic punishment is possible, and breaks down if it is ruled out. The evidence indicates that negative emotions towards defectors are the proximate mechanism behind altruistic punishment. These results suggest that future study of the evolution of human cooperation should include a strong focus on explaining altruistic punishment.
It's worth noting that this apparently fundamental human instinct, vital as it is to society, is put to all sorts of uses that are now considered inappropriate, or at least non PC: Xenophobia, racial discrimination, harassment and worse of heretics, picking on non-conformists, homophobia, etc., etc. At the least it explains why those behaviors are so hard to root out. At worst, yet more evidence that the human race is irretrievably flawed.
Unfortunately, the details are behind an evil Nature paywall, so I have no way to judge their experiment's methodology or validity.
Nevermind: An ungated copy: http://www.matchism.org/refs/Fehr_2002_AtruisticPunishment.pdf