When it's time to call off a fling, it's time to call it off: no use in delaying the inevitable. The longer you let the discomfort and weirdness simmer, the more difficult it'll be to live with your conscience: not to mention the hurt feelings on behalf of the other person.