It's... a little hard to explain, since I don't fully understand it myself. But Esme has "Authority" over this part of Purgatory – somehow, she managed to become powerful enough on her own that Invidia itself recognized her power and now she more or less has control over all of it.

I don't think she has control over everything here, or she could just snap her fingers and get rid of us, but she can easily protect herself against any attacks that come her way. So violence against her just wouldn't work.