The 37-year-old singer rejoined the man-band last year after quitting the group back in 1995, but has apparently only signed for one album.
Now Gary Barlow is believed to have scheduled a meeting for January - once his work on the X Factor is finished - to work out if they will continue with Robbie or go on as a four piece.
A source told The Mirror, 'Once The X Factor is out of the way, Gary will call the band, their management and record labels together to decide whether to stay as a five. They had a great reunion, but Robbie's solo sections caused a lot of controversy and his potty mouth put noses out of joint in the audience.
'Now their future is up in the air. Rob wants to continue, but they only signed up for one tour and album and it remains to be seen whether that will be extended.'