There are genetic penalties for small population sizes: deleterious mutations are more likely to persist by genetic drift. Consequently, Eurasians have more deleterious mutations than Africans. Small endogamous populations like Ashkenazi Jews and many tribal groups are even worse off in this respect. The most severe effects are seen in populations that have undergone severe population bottlenecks, especially if they are recent, as in the cheetahs.
Of course with a current world population in the billions, and endogamy in drastic retreat, such bottlenecks are now usually restricted to very isolated groups.