Cole, 27, told Walsh to give her act Katie Waissel a chance after his criticism of her performance on the first live show on October 9.
However he replied: 'I have an opinion. It's not all about you, Cheryl!'
Cole looked annoyed at his comments and their spat continued on ITV2 show The Xtra Factor, which followed the main show.
The show's presenter Konnie Huq asked Louis a question from a fan that said: 'Do you get jealous because Cheryl gets all the attention?'
He replied: 'Oh she always gets the attention, she's the nation's sweetheart.'
Walsh continued his rant about Cole in the new issue of the X Factor magazine.
He said: 'I'm fed up listening to her! It's not her show.'