The pair have regularly been spotted out together over the last five months and he recently accompanied her to Baton Rouge, where she is due to start filming new movie Breaking Dawn.
However Joe's mum Denise is said to be worried about their relationship and it is claimed she isn't Ashley's biggest fan.
A source said to US website Hollywood Life: "Normally he's the one calling the shots, but now Ashley's the one leading the relationship.
"She's edgier and older than Joe's other girlfriends and his mum isn't thrilled about the situation."
Joe, 21, has previously dated stars such as Taylor Swift and Demi Lovato who have been younger than him but Ashley is two years older at 23.