Without going into details, yes I have been in this situation (fairly similar specifics to your scenario) and felt the same random resurgences of sadness/etc for quite a few years after the incident. I wish I had some magic advice that would help you move forward but I don't. For me, it was pure perseverance, putting my head down and continuing to acknowledge that I had forgiven him and had decided to trust him. It's certainly a painful thing to have to deal with, but when I would feel sad about it, I would remind myself of all of the things we had discussed, the sincere remorse he expressed for doing what he did and (most importantly in my opinion) all of the good, happy things that had happened between that time and the present. If it eats at you regularly, makes you feel depressed for long periods of time or that sort of thiing, therapy isn't a bad idea. But for me, dealing with literally maybe a day of feeling crappy about it and then rebounding again, time was really the only cure. Good luck 
Comment