How about this twist, you time travel back to when you are six, but you have no recollection of what happened in the future, including anything that happened in your life. You just have a vague memory that you were older once and time travelled back in time.
So you have no memory other than you think you are an old time traveler? This ends up with you medicated in a looney bin or under a bridge talking to pigeons.