The key difference between M and AO is that M is sold in stores like Best Buy and Wal-Mart, while AO is not. This is analogous to movies. While both NC-17 movies and X movies are only available to adults, X rated movies are definitely not sold in Wal-Mart. Though this isn't an official ESRB stance, most every retailer, large and small will not carry AO games if it sells games to a general audience.
I think this makes sense for video games and movies. There is a difference between sexual and pornographic, just as there is a difference between violence and snuff. Some things are adult, other things are very adult. It's reasonable to expect that many adults wouldn't want that level of material in the average store, not only preferring it be kept away from kids, but kept away from them as well. Hence why AO games are usually only available online and why X rated movies are usually placed in adults-only areas of stores or sold at specialty stores.
X and AO are different in that AO is an actual rating, while X is simply a placeholder as the pornograpic movie industry doesn't participate in the MPAA's CARA rating system. That's different with video games, because when a company has agreed to have its games rated by the ESRB, it agrees to have all of its games rated by the ESRB.
So, when Sierra submits Leisure Suit Larry, they get an AO, because the game is explicitly sexual throughout. Most adult games, however, are made by publishers that only publish adult games. Sierra, however, publishes a wide variety so it and other publishers choose to stay in good standing with the ESRB and submit their explicitly adult games.
The ESRB only lists 23 AO games on its website. I don't know if this is a complete list, but compare that tiny number to 1139 M rated games listed. Clearly AO is reserved for a tiny minority of extreme cases. So, why the complaints?