If 3 books are requested, can I cancel just one??  Or will that cancel the whole order?

If you do not have them ordered in a special, there is nothing wrong with cancelling just one of the books. And no, in no situation will cancelling one of X number of books cancel the entire order. Your order will simply become an order for two books rather than 3. And of course, you can only do this if the wrapper hasn't been printed yet. After that, it becomes more problematic to cancel one or more of the books.

If someone requested the 3 books from you, then I'd suggest that you PM the person letting them know you are cancelling the one book.  When you cancel, the person's order for that book will automatically be passed on to the next sender.  Sometimes people will order extra books from your shelf, to make a bulk order, that they don't want enough to order separately.  By PMing them, you give them the option of canceling the order for that book.