Your retina—the light-sensitive lining at the back of your eye—is packed with light-receiving cells called rods and cones. Only the cones are sensitive to color. These cells are clustered mainly in the central region of the retina.
When you see something out of the corner of your eye, its image focuses on the periphery of your retina, where there are few cones. Thus, it isn’t surprising that you can’t distinguish the color of something you see out of the corner of your eye.
The rods are more evenly spread across the retina, but they also become less densely packed toward the outer regions of the retina. Because there are fewer rods, you have a limited ability to resolve the shapes of objects at the periphery of your vision. (The diagram below shows the distribution of rods and cones across your retina. Click to enlarge.)
In the center of your field of view is a region in which the cones are packed tightly together. This region is called the fovea. This region, which is surprisingly small, gives you the sharpest view of an object. The amount of your eyeball covered by the fovea is just a couple of square millimeters—similar to the fraction of the night sky that appears to be covered by the moon.
You can demonstrate this effect more simply by focusing on one of the words on this page while at the same time trying to make out other words to the right or left. You may be able to make out a word or two, depending on how far the page is from your eyes. But the area that you can see clearly is the area imaged on the fovea of your eye.
Generally, you are not aware of the limitations of your peripheral vision. You think that you have a clear view of the world because your eyes are always in motion. Wherever you look, you see a sharp, clear image.
Interestingly, your peripheral vision is very sensitive to motion—a characteristic that probably had strong adaptive value during the earlier stages of human evolution. The photo below shows a typical distribution of data for this activity. (Individual answers may vary.)