The 34-year-old will not have his existing contract renewed for a second series after receiving 'terrible reviews', according to reports.
An insider told British newspaper the Daily Mirror: 'Steve keeps saying that living in Los Angeles is a dream come true. He loves the weather and the women, and he loves working on X Factor - it's just a shame that the US public has not warmed to him.
'He got terrible reviews from critics when he started and things haven't got better. There have been mistakes and he just doesn't seem to have the knack for dealing with contestants.
'The executives at Fox have already decided not to renew his contract and so, short of a miracle happening, it is pretty much a done deal that he will not take part next season.'
It is thought Steve could be replaced by singer Nicole Scherzinger, who took up a role on the judging panel alongside Paula Abdul, L.A. Reid and boss Simon Cowell after Cheryl Cole was fired from the show.
An X Factor spokesperson added: 'No discussions have been had and no decisions will be made about any new contracts until the New Year.'