It depends how far we go with the term 'casual'. If we are talking the people that only know Conor, Jones, Khabib level and never watch Fight Nights ever, then yes, of course.99% casuals don’t know who Glover is.
100% hardcore fans love and respect Glover for being everything we want in the sport: heart, skill, humility, class
But if we are talking the casual bar audience who get exposed to UFC but don't really follow it, then I think you are seriously underestimating how much exposure he has got.
This was Glover's 7th UFC main event. For comparison, the UFC's Fight Night workhorse, Cowboy has had 9. Nate Diaz has had 8. I would be very surprised if any other non-titlist has been in more. And he has been in 6 co-mains. He was co-main in the 2nd biggest PPV ever, Conor-Diaz 2.
So 2/3 times GloTex has been main or co-main in his UFC career. The UFC certainly seems to think he has a certain level of drawing value and just by pure exposure, there's probably a lot of relative casuals who flick on the UFC who would know him.