what makes a d&d sub race stick?

[Psyren]

I think the key question for most subraces is "does it still make sense to mechanically represent this". Unlike races which are more or less an objective *thing* in the game world subraces tend to not draw such a hard line and are arguably a different culture as opposed to a different bloodline a lot of the time. Like I'm not a Tolkien scholar or anything but I'm not sure he considered Legolas and Lady whatsername to be distinct on a genetic level or anything like that.

So I guess since the main distinction between the halfling and dwarf subraces were their ability scores they were folded in since there wasn't a role for them to play in the mechanics anymore.

Adding to this - the starting feat we're getting in 5.5e means you have another way to represent the cultural/regional differentiators as well, such as Mountain Dwarves' armor training (Lightly Armored) or Hill Dwarves' hardiness (Tough).

While the basic principle here seems correct to me, note that the media which first draw people to D&D have shifted. I run games at a shop and see a decent number of new players. Far more people show up who were inspired by video games and actual play series (Baldur's Gate III is often cited) than by fantasy literature, so plenty of people come in wanting to play D&D-specific races. The Lord of the Rings still has enough broad recognition that people understand what an Elf or a Dwarf is, but modern fantasy literature has taken a sharp turn away from fantastical races as a setting trope.

bilbo and Gimli are not that big any more there like your dad's favourite rock band or something they are still going and deeply foundational but lots of things have since been spawned that have changed things greatly.

notice anything that seems to be desired beyond humans or old Tolkien options? it would suggest where the wind is presently blowing

Indeed; we've already seen Tieflings and Dragonborn start to eclipse Dwarves and Halflings in popularity (and we had a whole thread on the subject not too long ago too.)

The three most common themes I encounter are (in no particular order), contemporary media emulation, special snowflakes, and horny.

"Contemporary Media Emulation" just means "cultural evolution."
And increasingly - the folks who want to be gnomes and hobbits are becoming the "special snowflakes" I'd say.