The 30-year-old took a flight back home to the US shortly after falling out with fellow judge Tulisa Contostavlos over their acts live on air.
However bosses are said to be unhappy about her decision as she won't have a lot of time to spend with her three acts - Janet Devlin, Sophie Habibis and Misha Bryan.
A source said to UK newspaper The Sun: 'As the contest heats up, she's left the girls in the lurch all week. It's very arrogant.'
Tulisa, 23, was reportedly planning to make up with Kelly after the show after she called one of Kelly's acts 'mean' live on air.
A source added: 'She was expecting to bury the hatchet with Kelly on Sunday night. Kelly was having none of it.
'She had a major sense-of-humour failure - and was trying to suggest Tulisa was deliberately trying to destabilise her. Tulisa's not going to go crawling to Kelly - she's too stubborn for that.'