To be fair, this happens to me also in Debian Gnome, but very very rarely. On Windows, it keeps me from ejecting the drive until after I try it 4-5 times. Every time. In any case, lsof can help identify the culprit.

lsof | grep DRIVENAME should give which program is keeping it.

Same when you try to remove an external drive.

This is strange. You may have already solved your problem, but have you checked your apt sources list that it points to the right source (stable / oldstable) or (bookworm/trixie)?