The 37-year-old singer and Take That star is believed to have been asked to replace Simon Cowell on the talent show, but said no because the money wasn't good enough.
A source told Star magazine, 'With Simon leaving to judge the US show, he's convinced Robbie would be the perfect successor.
Rob's likeable, opinionated, and funny. But money has been a stumbling block. He wants more of an incentive to put himself out there on live TV every week.
'He also wants a baby in the near future, and that's going to be tricky enough to fit in with the upcoming Take That tour as it is.'
The insider added that while his wife Ayda enjoys their high profile, Robbie prefers to shy away.
'Ayda loves the limelight, and there are rumours she's in talks for the next series of The Real Housewives Of Beverly Hills. But Rob prefers the idea of a quiet life,' the source added. 'If he had his way, he'd move to a desert island and live like a recluse.'