TSM is fun, but not accurate.  I have two friends that challenge me, and I beat them back as badly as I can.  I have noticed that TSM needs a tweak.  The way I understand TSM to work is like this:

Total track time divided by number of laps = average lap time 

You race your friends in TSM according to the above.  This does not represent your friend accurately.  You are racing the AI, with all it's dirty tricks and stupid tactics.  

The way to know how you are getting beaten(or not), or how you are able to beat your friends(or not) is simple, sort of...

GoPro yourself racing the TSM challenge race.  Send the video to your friend.  

Am I way off track here?  (Pun)

:)