New paper: In a small study with 32 people who were learning science from an AI tutor that was confidently wrong, the participants took over around two of three of its false informations, and they felt almost as sure about them as about the true facts.
Their trust in the AI did barely drop, what suggests that sometimes it's not the AI who bends to the user, but the user who bends to the AI.