The 30-year-old singer famously fell out with co-judge Tulisa Contostavlos live on air when Contostavlos made a comment about one of her acts, Misha B.
It's claimed Kelly has found it difficult to get over the feuds and has been feeling down.
A source told UK newspaper Evening Standard: 'While everyone is aware of the panto element of the judges and the bust-ups, Kelly was not and did not realise it was something she had signed up for.
'She is refusing to take part in them and it is causing a lot of friction between her and the producers. She joined to help find and develop new music talent.'