I'm not picky about publish date.

I'm open to various genres, but do not read much romance. Non-fiction is fine, too.

I just went through a crumbling mansion phase too. Unfortunately, I had little success finding the perfect haunted house book.

I tried The House of Seven Gables, Turn of the Screw, The Haunting of Hill House

No luck.

I have not tried Hell House yet.

The Dwelling by Susie Moloney           The Haunting by Ruby Jean Jensen                  The House that Jack Built by Graham Masterton

The Little Stranger by Sarah Walters - haven't read it yet but there is a "crumbling mansion" in it. If you read it let me know how you like it . . . it's been sitting on my TBR for a few months.

For old southern dilapidated mansions, perhaps New Mercies by Sandra Dallas would fit what you are looking for.

Well, the mansion isn't exactly crumbling, but Rebecca by Daphne duMaurier captures that decayed Gothic feeling perfectly.  Also The Thirteenth Tale by Diane Setterfield mostly takes place in a very dilapidated mansion.

I really loved House of the Seven Gables and The Haunting of Hill House, both are classics and the prose is a little flowery in Seven Gables, but they're very atmospheric.   Although as far as Shirley Jackson goes, We Have Always Lived in the Castle is more of a decaying mansion story and less of a haunted house story than Hill House.

If you're into classics then Wuthering Heights or Jane Eyre would be definites, and of course Fall of the House of Usher (even though its only a short story).

And for non-fiction, Midnight in the Garden of Good and Evil is fantastic.

If I'm not mistaken, The Thirteenth Tale by Diane Setterfield features a rather bleak house.

Shadow of the Wind by Carlos Ruiz Zafon features some ancient locations. I wouldn't say they were a character in the plot, but it's a good read nonetheless.

The Haunting of Hill House should be a good one for you. 

The Clue in the Crumbling Wall was good if you don't mind juvenile fiction

The Big Steal, 2nd in the Sterling Glass series by Emyl Jenkins.

Nonfiction by Tracy Kidder - "House"  - It sounds like a strange topic, but is really very interesting.  It's the story of one couple building a house from design to moving in.  I really liked this book and have read several others by him.

You might try this one:

http://www.paperbackswap.com/book/details/9780091897314-Castles+In+The+Air+The+Restoration+Adventures+Of+Two+Young+Optimists+And+A+Crumbling+Old+Mansion

You might also try The Keep by Jennifer Egan although a lot of the story is told in flashbacks that have nothing to do with the castle which is the main setting.

Sheila,

I really wanted to like LIttle Stranger, but I just couldn't.  About halfway through I gave up (which I VERY rarely do).  It just didn't interest me at all.  Maybe it was just me though!

Sara

The Guynd by Belinda Rathbone.  An American woman who married a bona-fide laird in Scotland and moved to his manor house, which was literally crumbling.  Good nonfiction.

Castle Rackrent by Maria Edgeworth - the very first crumbling mansion novel ever.  Full disclosure: its on my bookshelf :).

The Woman in White by Willkie Collins, and of course Northanger Abbey which is a parody of all the gothic fiction coming out at the time.  It's sort of a different animal but Little, Big by John Crowley might be worth a shot. 

ETA - this is a wonderful ss collection, with some heavily anthologized stories (Usher, a Rose For Emily, Rappacini's Daughter etc.). but it also has many, many hard to find early gothics.  Decaying, creepy mansions abound.

The Witching Hour series by Anne Rice. GREAT series!

Buried Blossoms by Stephen Lewis (hard to find, but it's a great book)

His Dark Kiss by Eve Silver has a crumbling mansion: http://www.paperbackswap.com/book/details/9780821779675-His+Dark+Kiss

IIRC -this one is pretty much a straight gothic, although some of hers feature a paranormal element as well.

I enjoyed Castles in the Air: The Restoration Adventures of Two Young Optimists and a Crumbling Old Mansion by Judy Corbett. It's non-fiction, but a good read.

I also thought of Dodie Smith's I Capture the Castle, if you haven't read that one yet.

And I have a book on my TBR list called China Court: The Hours of a Country House by Rumer Godden. Not sure how "crumbly" the house is in that one, but it sounds like a classic country house story, anyway.

I've read some of them previously and added some new ones to my list. I'm glad people enjoyed Castles in the Air, I put it on my wish list a week or so ago and couldn't find much info / opinions about it.

I'm also in the middle of reading a really, really silly mystery by John Dickson Carr called Castle Skull.  I love his mysteries, but when I recomended them to my mom, her reaction was basically, that they are just so stupid. I was like, well, yes, but I love them.  There's a classic, moldering mansion, formerly owned by the world's greatest illusionist, filled with creepy booby traps.  It's awesome.

Also - Marjorie Allingham's The Crime at Black Dudley.

Deanna Raybourn has three mystery/romances called Silent in the Grave, Silent in the Sanctuary and Silent on the Moor. They take place in the 1880s in England, and although the first one doesn't have an old crumbling mansion in it that I can recall, the second and third books do.

I loved The Little Stranger.  Also, I just started The Physick Book of Deliverance Dane.  It doesn't have a crumbling mansion, but a crumbling small home in or close to Salem, Mass. that's somehow going to tie in with the Salem witches, I'm thinking.

Forgot to mention Carlisle Street. I've read it three times. It's such a good, creepy horror novel.

Burnt Offerings by Robert Marasco features a crumbling, old mansion as one of the pivotal characters.  It is a horror novel, kinda sorta on the lines of Shirley Jackson's Hill House.