I've always been very careful about returning a book I have borrowed from a friend. In fact, when I bring one home, I have a small white sticker I place on the back with my friend's name on it.
On the other hand, I have six or seven bookshelves scattered around the house that I have no idea who lent them to me.
