Simon is expected to leave his role as a judge on the UK version of the show while he launches it in America, and fellow judge Cheryl Cole has gone with him in the hope of landing a job in the US.
Sharon Osbourne, Gary Barlow, Robbie Williams and Lily Allen are all rumoured to have been approached to replace Cowell and Cole, but Dermot believes it needs someone with 'alpha' qualities.
He told The Sun, 'I don't think we need a nasty judge. I think we need an alpha, but an alpha could be anyone. I'd love Lily Allen to be sat in Simon's chair.'
He added, 'As a judge, anyone from Take That I'd be over the moon with. But I still hope Cheryl and Simon will be back.
'I've no idea who I'm working with, which is weird, because we're starting quite soon. I'll just have to do it myself.'