The 30-year-old welcomed daughter Monroe and son Moroccan on April 30, but now has to leave his little family to go and host America's Got Talent at venues across the country.
Talking about having to pull himself away, he told Life & Style magazine, 'It's quite difficult. Probably one of the hardest things I've had to do.
'I really didn't think about how difficult it was going to be because my wife has always been on the road.
'She's extremely understanding when it comes to that. I didn't think it would be harder than before until it actually happened.'
He added, 'All I want to do is be home with my kids. But you have to get out there and work, so it's definitely a difficult task.'