Bahorel notices the way Javert pales, and falls silent, and combines it with what he knows of Joly and feels the first outlines of a realization.

...And also definite confirmation that Javert doesn't even know what a wretchedly bad actor he is.

But all Bahorel allows to show is minor irritation. "Joly's said nothing. He treats the rest of us as spies when it comes to his precious Infirmary." How very dare you, Joly. Bahorel is visibly offended. Treachery. Etc. "But gossip says plenty. If you were the grand spy you claim to be, you wouldn't marvel that people can know what they didn't see. Even here at Milliways, good Madame Hearsay is a magician. A room full of people seeing a man who needs two other to carry him--why, that is a stage for her. That any patron passing by for an eye-blink can understand through her offices what you cannot gather firsthand--that should hardly surprise you anymore." ...Wait, is that insult too complicated? Well, Bahorel can throw out more if he needs to. Javert distracted and angry and snapping is much more potentially useful than Javert sitting silent, if no more pleasant.