The 40-year-old singer is a judge on the British talent show along with Louis Walsh, Kelly Rowland and Tulisa Contostavlos.
It's claimed one contestant got so upset by his comments that he stormed out and threatened to commit suicide.
A source told UK Newspaper The Sun: 'It's like Gary's evil twin turned up to judge. He's making Simon look like Mary Poppins.
'When he told singer Pebbles she looked like a man at her Birmingham audition, you could have heard a pin drop. She was mortified and immediately complained.
'There have been several complaints over Gary's dressing-downs. One singer said they felt like killing themselves after coming off stage. Gary has been asked to tone it down by contestants.'