Thanks for the report. In the next update, we will ensure that quotes checked with whispers, will have response with whispers. That will prevent people from spamming the channel.
Currently twitch does not have any API endpoints to check the list of users timed out, neither does it publish the timeout duration when someone is timed out. So it is virtually impossible to keep an ignored list in sync with the users that are timed out/banned.
We are thinking on what is the best way to let you and your mods manage this listing. Hopefully, the whisper -> whisper will solve your immediate concern.