Robert Goolrick's A RELIABLE WIFE is a literary novel while Shirley Jump's MARRY-ME CHRISTMAS is not. Agatha Christie's MURDER ON THE ORIENT EXPRESS is a literary novel while Tamar Myer's BATTER OFF DEAD is not. John Steinbeck's THE RED PONY is a literary novel while Nicholas Evans' THE HORSE WHISPERER is not.
Or is it? Sometimes it's hard to tell the difference. It may not matter much unless you're submitting a novel to agents or editors. Some consider literary novels; others don't. How can you categorize yours? A few thoughts:
1. A literary novel isn't written according to a formula or set conventions. In some cases this is the same as saying it's not a genre story. For example, one formula for a genre romance goes something like this: Strong, handsome, somewhat evil man is the central male character. Innocent, basically good woman is the central female character. The story consists of slowly bringing central male and central female characters together. In the process, the female character in some way reforms the male character and the ending suggests the two will live happily ever after.
2. Characters drive a literary story. And those characters are unique, with unique blends of characteristics. They're not stock characters. At least the main ones aren't.
3. Since the plot is driven by unique characters, it isn't predictable. On the other hand, the author of a literary novel doesn't necessarily rely on unpredictability or suspense to keep readers reading.
4. Authors of literary novels don't think they have life completely figured out. They use writing as a means of exploring life.
5. Literary novels engage intellect as well as spirit and emotions. But I don't think reading of literary stories is confined to intellectuals. I've seen uneducated people of apparently average intelligence devour literary novels, preferring them to non-literary works.
6. Literary novels don't sell as well as commercial ones though there are obvious exceptions.