Married man question?

I have known this younger man for about 18 years or so. He is now 40 and just got married to his 27 year old girlfriend. He broke up with his former girlfriend five years ago, (he was seeing me during this time). He met this younger woman and recently married her. When he met her he said he couldnt see me again, we just make love and it is great, we have not dated actually since I met him years and years ago. During these 18 years I dated and had relationships, but we always communicated. Two years ago, he came to my back door at three am to tell me he was moving to SC for one year. He kissed me passionately and left. Since I found out he was married, I have been very down for some reason, thinking I would never see him. Lo and behold, he called me last week, asked to come over and oh boy it happened again. Now he thinks this is going to be a regular thing. He told me a while back that he loves me and that he loves his wife and everybody. I feel that at times he is callous, cocky and very flippant and cavalier about relationships. What should i do, i have a strong attraction to him. I know it is wrong to see him.