It quiets down a lot when I pull the clutch in, and that's where the sound seems to be coming from.
@LT250RW--Yeah I know. Fortunately yesterday was my last ride of the season, so I'll definitely be getting a set of washers and the bearings soon.
Thanks for the help.