When you feel that he doesn't value you anymore, this would be the right time to let him go.