If you listened to an audiobook, do you say you "read" it?
My vote is listening does not equal reading. Reading is active, where you digest the content in your own internal voice; listening is passive, digesting content in someone else’s voice.
Likewise vice versa, where if you read the transcript of an interview (or deposition), for example, you wouldn’t say you listened to it.
ETA: And there is nothing wrong with saying “I listened to” a book — it’s a very common activity, especially for people who spend a lot of time commuting.