How do I

Filter By Friends' Bookshelves

I checked it and nothing happened.

Also, is there a way to check 1 friends wish or reminder list to see if they have any of the books I want

From my reminder list I clicked Filter By Friends' Bookshelves.  No books were listed so I added book from a friend's bookshelf to my reminder list.  Then I searched again and it worked.  So it works but none of my friends have the books.  

When you post a book from your account you can see if any friends are wishing.  You can look at any wishlist if they are public . You can 

Filter By Friends' Bookshelves 

I checked it and nothing happened.

Once you check the box, the books remaining on the list will be on a Friend's Bookshelf. If you click the Order this Book button from that list, the request will go to that Friends list Bookshelf instead of the FIFO system.