Well, to be exact, and with regards to color theory, our skins represent many shades of colors, roughly ranging from 15° to 30°. That's from a reddish to a yellowish orange:

The two adjacent ramps of shades/tints in lightness and saturation don't show all colors, but just the ones at exactly 15° and 30°. So there are even more nuances between them.

So brown is actually a dark and saturated tint of orange.
And practically all skin colors can be derived from that.

Maybe that's why the Dutch are so keen on proudly celebrating our nationality and equality in orange... ;-)

Another funny approach: skin tones in Pantone inks:
Human - a work in progress

Oorspronkelijk geplaatst op Quora: Why do we label ourselves as white or black when we're actually all varying shades of brown?"