Okay, I know the topic question sounds rather strange but as I'm reading a book characterized as fantasy/horror (and where the horror comes in is beyond me in the book) and there is a very strong love/hate relationship going on between the hero and heroine similar to what I've read in many romance novels, I'm wondering what makes this book be characterized as a fantasy novel rather than say romance novel (maybe under paranormal?).

I've answered this in another forum and basically what I believe makes a book a romance novel is the focus of the romance. To me a romance novel needs to have the romance as the main focus and not a side story. Many romantic suspense authors write a blend of suspense and romance that sometimes its hard to determine which is the main genre. Also to me I must have the HEA. If there is no HEA then to me it qualifies more as a tragedy, such as Romeo and Juliet.

I have always found this a mystery.  I started out reading suspense/mystery novels and found quite a few of  them identified as Romance in the library.  I wasn't sure what the differentation was.  I finally came to the conclusion that the difference is sex.  You don't find explicit sex scenes in hard-core mystery/suspense novels.  Many of them have a romance going, but no explicit sex.  I may be wrong with this analysis, but that's how it strikes me. 

According to RWA: “Two basic elements comprise every romance novel: a central love story and an emotionally-satisfying and optimistic ending".  Romance can be classified into one of many subgenres, like contemporary, historical, regency, and paranormals, but what typically separates the paranormal romance from a fantasy novel with a romantic theme is the ending.  Like Sommersby or Cold Mountain could be called romance novels, except for the endings, which make them historical or romantic fiction.  Sex is not a pre-requisite, but is generally considered to be one of the elements that makes a romance novel more publishable.

Libraries have their own ways of cataloguing books and they may not be consistant. I see books by romance authors wind up in general fiction and the J.D. Robb books are all over the place.  And at our library there's no hardcover romances, they just go into general fiction.

I'm glad you brought up J. D. Robb---is the In Death series considered romance? Or is it something else? 

The book I'm currently reading is Soulless and to me it's a romance--or at least the romance is a big part of the story. Do authors have any say in what category their books are marketed as? Of course, maybe some of it has to do with the publishing company itself. I'm guessing that some publishing companies specialize in a certain genre so they automatically categorize a book that can go in any of number of categories in the category they are known for publishing?

This was discussed some a few months ago in a thread about the difference between Chick Lit and Romance.

As Kim said, by definition, a romance novel has to have BOTH of (1) an optimistic / HEA ending and (2) a central love story.    

I don't believe that a book has to have just one genre.   Take the J.D. Robb books or the Pink Carnation books as examples - or any romantic suspense novels.      They are both romance and mysteries.    (and in the case of the Pink Carnation books, we also have historic fiction and chick lit in the mix).

But a story can have romance - lots of romance - in it and not be a romance novel.    An example would be The Time Traveller's Wife.    A very romantic book - but without the HEA they aren't technically romance novels.    I call them romantic novels or love stories.

When it comes to genres or tags, however, all that's needed is for the book to have a fair amount of romance in it in order to be tagged Romance.   That doesn't mean it's a romance novel, necessarily.

I don't think you can use sex to determine if a book is a romance or something else with a romantic element. There are MANY romance novels and in fact entire categories and genres of romance novels where there is nothing more scorching than a kiss between the H&H.