A couple of examples might make this clearer. One of the most-successful self-published books in the last decade has been a "pricing guide" to Beanie Babies. Specialized catalogs and shopping guides, particularly quickly produced books for fad markets, can be successful when self-published. The manufacturer (Ty, Inc.) actually made the success of that particular book possible by refusing to itself provide comprehensive listings of its offerings. Combined with the "collector mentality" and litigation over knock-offs, this created a specialized market of somewhat irrational consumersperfect for exploitation with a good-looking guidebook.
The most-prominent "franchise" arising from self-publishing is probably Chicken Soup for the _____. Even a cursory examination shows that this is an outlier. It had its beginnings in the "inspirational stories" subcategory that appears mainly in nonmainstream bookstores. Only afterward has it grown to the current empire, which can no longer be called "self-publishing"if it even could, because the books are collections of material provided by a large number of contributors. That the contributor's contract is one of the most rapacious possible further removes these books from "self-publishing." The gap this filled was one that is imperfectly filled, even in the nonmainstream bookstores. Most "inspirational stories" are extremely poorly written and clichéd; this is easier to take in lots of small chunks from a variety of voices and viewpoints than in novel-length indigestible lumps. That is, it's easier to swallow the pills (and keep them down) in small doses…
Last, and least, there's the wretched Celestine Prophecy. Like Chicken Soup, this book originated outside of mainstream distribution (too bad it didn't stay there). Note, too, that almost all of its financial success has come after the book was picked up for commercial publication.
Well, there is one rather ironic exception. One of the more-worthwhile self-published books out there is Dan Poynter's The Self-Publishing Manual, now in its 14th edition. A self-published book about how to self-publish. That sounds a great deal like the get-rich-quick schemes whose cash flow comes from selling get-rich-quick schemes to even greater fools, although Poynter's advice is far, far more sound than even in the commercially published books on self-publishing.
In any event, the kind of book that I see most-often proposed for "serious self-publishing" is the thriller/mystery novel, usually a first novel. Admittedly, this is something that the publishing industry encourages through its own low standards for what it publishes. And, to put one myth to rest: John Grisham's first book was neither self-published nor published through a vanity press. He purchased copies from his actual publisher and sold them from the trunk of his car, true enough; but it was a royalty-paying (albeit not on the copies he purchased) small press.