The Enneagram instincts are self-preservation, sexual (also called one-to-one and intimate), and social. Although rarely mentioned, the instincts are used with the Enneagram types in two quite different ways: the subtype approach and the stacking approach. Let me start with a simplistic example of the difference between subtypes or subcategories and stackings or preferences.
Under the category of "bird" we have the subcategories of "duck", "cuckoo", and "ostrich". Bird is a type of animal and duck, cuckoo, and ostrich are types or subtypes of bird. All three kinds of bird have characteristics of a bird but they also have characteristics that distinguish each from one another. This is analogous to the instinctual subtype approach where a person is identified as one of three subtypes of a type.
With three flavors of ice cream, most people will have a preference for one over the other two. This preferred flavor is the one that is sought after most. If the first preference is unavailable then the next preference will be chosen. If neither of the first two are available then the least preferred flavor will be chosen because there's no other choice or if it's disliked then it might simply be avoided. While some people may find it easy to determine their preferences, some people can find it hard determining which of the three they prefer most or which of the remaining two they prefer most. This is analogous to the stacking or preferential approach to the instincts where one instinct is primary, one is secondary, and one is tertiary in preference.
The subtype approach is how the instincts were originally used with the Enneagram personality types. This approach is used to distinguish three variations or subtypes for each Enneagram type: the self-preservation (sp) subtype, the sexual (sx) subtype, and the social (so) subtype. An instinctual subtype is simply one of three kinds of that type (much like a duck, cuckoo, and ostrich are each a kind of bird). The nine Enneagram types are subsequently expanded to twenty-seven subtypes (i.e., three subtypes for each of the nine types).
With the subtype approach, you can determine subtype in two ways. You can first determine which of the nine types is the best fit then which of the three subtypes for that type is the best fit (i.e., "This type fits me best and this subtype of this type fits me best."). You can also simply look at all twenty-seven subtypes and determine which subtype fits best (you just have to make sure that the type also applies because the subtype shares the characteristics of type as well).
The stacking approach is more recent than the subtype approach. It doesn't look at type first, but looks at instinct first and it doesn't simply try to determine which one of the instincts is dominant but looks at the preferential ordering of all three instincts. For example, instead of simply identifying with self-preservation (sp), the individual would have self-preservation as the first instinct and either the sexual instinct second (sp/sx) or the social instinct second (sp/so). The third instinct is implied by its omission (e.g., sp/sx has so third while sp/so has sx third).
The obvious difference between the two approaches is that the subtype only designates the primary instinct while the stacking approach designates the preferential order for all three instincts. However, the differences are more than simply that.
With the subtype approach, instinct is not viewed independently of type. Someone who identifies as sp 1 and someone identifying as sp 2 have nothing in common. A dominant self-preservation (sp) instinct has only to do with determining variations of type. In other words, the instinct is simply a way of creating three different subtypes and not a way of exploring similarities between different types with the same instinct.
In the stacking approach, instinct is independent of type. You can actually use it without referring to Enneagram type. A person identifying as sp/sx 1 does have something in common with someone identifying as sp/sx 2. They both have a primary preference for the self-preservation (sp) instinct and a secondary preference for the sexual (sx) instinct. You could say that two people identifying with the same stacking would have a lot in common in spite of their dominant type being different (e.g., sp/sx 1 and sp/sx 2).
People not realizing that there are two very different approaches to using the instincts with the Enneagram types tend to combine the two approaches and not even realize that they're combining two different things. This can be a problem when an author writes about the instincts using the subtype approach but the individual uses those subtype descriptions with the stacking approach. They don't really go together because the subtype approach looks at type then instinct to determine variations of type while the stacking approach looks at instinctual preference and may or may not even look at how that affects type. Another way to say it is that an instinctual subtype description may not make a good instinctual stacking description because instinctual subtype is using instinct in a different way.