Simon Cowell, Cheryl Cole, Dannii Minogue and Louis Walsh have not yet agreed to take part for another series despite the fact auditions are soon set to start.
The show's host Dermot O'Leary has also failed to sign a deal to return in his role.
It is thought Walsh and Minogue will return but Cole and Cowell are rumoured to be leaving to join the new US show when it launches later this year.
A source said to UK newspaper The Sun: 'This is causing producers a headache as the contract process always takes some time to hammer out.'
A spokesman for ITV said: 'We're excited about this year's X Factor and full details will be announced in due course.'