Brown's working visa was refused due to his criminal history. He was sentenced to five years' probation and six months of community service for his assault on Rihanna in 2009.
The 22-year-old's managers suggested that he performed as a hologram at the Michael Forever concert - as Beyonce is scheduled to do - but organisers decided against the idea.
A source told British newspaper the Daily Mirror: 'It would put a negative slant on the day. The family wants to make sure the show is a fitting tribute to Michael.'
The With You hitmaker is the latest star to pull out of the concert after the Black Eyed Peas cancelled their headline slot earlier this week.