Had the same thing on my 88 LT500.The shop I deal with up here machined the crank journal a bit then fitted a sleeve over the journal and machined it to the proper size to fit the bearing.No issues since.I went that route as unlike the 250,s no one makes a new crank for the 500,s.Likely some members here may have a known good crank they,d part with so you could always ask around.

No way the crank seal would hold the bearing inner race and cause that.Likely the crank journal it just worn and if the bearing race has been slipping on the crank and slopping around its likely pretty worn and out of round.