Colorblindness
Most people (80%) with colorblindness have an abnormal red or green cone opsin, whereas the other 20% are completely lacking the red or green cone opsin. The abnormal opsin has amino acid substitutions which alter the spectral response by affecting the amount of energy which can be absorbed by the retinal double bond. The result is that the difference in activity between the abnormal green and the normal red cone (deutans) or the abnormal red cone and the normal green cone (protans) is too small.
This makes it difficult for colorblind people to distinguish colors that already have small differences in red/green contrast. Those with defective (not absent) red or green cones are still able to perceive colors which have large differences between the amount of red or green in them.
Likewise blues present little trouble, since they have very little red or green, and hence no contrast to miss. Yellows are also fairly well perceived, since the signal contains very little blue, which is contrasted to the sum of red and green – this sum is not affected by whether it is a green or red cone adding to it.
More detail
The genes that code the opsin pigments in the red and green cones are located on the X chromosome. Since males have only one X chromosome, they will be colorblind if that X chromosome has a defective gene. Women must have 2 defective X chromosomes in order for them to be colorblind, so it is much less common in females. The gene for the blue cone pigment is on chromosome 7, and since normal individuals have 2 chromosome 7's, blue-yellow colorblindness is also uncommon.
The red and green opsins are normally quite similar, differing only by about 20 of their 350 amino acids. When there are fewer amino acid differences in these opsin proteins, the difference in frequency response is smaller. Which amino acid substitutions are present is responsible for differences in colorblindness.
Another important factor in variability among the colorblind, which is not as much discussed, is that the mosaic of cones in the retina has a lot of variability in the percent of red cones and green cones. One might imagine that comparing activity of a green cone to the surrounding hundreds of cones would be considerably different if that surround average consisted of a lot of red cones vs a lot of green cones, especially if the spectral difference was small.