No, Valjean is not going to argue with him.

'I am not trying to mislead you, Bahorel, though I understand why you would think so. I have looked after Cosette, it is true, but she is no longer in my keeping. I am allowed to see her a little, and that is enough for us both.'

He will not question where Bahorel is getting his information - it can only be the book, but he does not want to know details - and he will not point out the most obvious lie of all; he is not a good man, and everything in his life must be coloured with that in mind. Especially this.