The latter is pretty close. With his plays, Brecht's aim was for the audience to remain uninvolved so they would be able to judge the characters behaviour - not forgive them, out of sympathy, for whatever they did, but rather be able to cooly pronounce moral judgement. Where Aristoteles saw the cleansing from emotions by engaging them, catharsis, as the point of plays, Brecht wanted a moral instead of an emotional effect on the audience; he wanted to show a world in which his characters did terrible things, in which they failed morally, and he didn't want the audience to accept these things and this behaviour but rather reflect on them and decide that the world had to be different, that they had to change the way things were [Brecht was a Marxist, and you can see the anti-capitalist influence very much in Trier's Brechtian movies, as well]. All of this, so Brecht argued, could only be achieved if the audience did not suspend their disbelief, preventing identification with the characters and an awareness that they were watching a play.
Brecht called this "epic theatre", and his most important tool was the "Verfremdungseffekt", an alienation of the audience from the play that he would achieve by deliberately breaking the illusion.
Whether Brecht actually achieved his aims or not can be argued about - obviously, von Trier didn't manage to achieve this effect, at least where you are concerned (but I am inclined to agree) - but the techniques of the "Verfremdungseffekt" had a lasting effect on the way plays were staged in the theatre.