Question is why I can post one set of books with the same title but different ISBNs but try to post a book that has a different ISBN than the one on my shelf but same title but this one can't be posted?

as long as they have a different ISBN they should both be accepted for posting.  You may have to contact TPTB to work this out.

Most likely the site decided on that rule to prevent someone from accidently posting the same book twice. Then it also makes things fairer to give everyone a chance of getting their book requested.

If you were to post the actual titles and ISBNs involved, I think we would be more likely to be able to figure out the issue.

Question is why I can post one set of books with the same title but different ISBNs

Yes, this is allowed.

but try to post a book that has a different ISBN than the one on my shelf but same title but this one can't be posted?

You must post your books by ISBN, because, although it might not matter to some book requestors ... some are ordering a very specfic ISBN of the book they want. Different editions of books might have "extra" stuff, or a certain cover, or a bonus story in the back, or a "25th anniversary edition preface" ... etc.  

The takeaway is that different ISBNs of the same book can be different, and PBS assumes that the person requesting the book is requesting that ISBN specifically. 

So, you have to post the ISBN you have.