Yeah, I was wondering the same thing. I don't think it's because the girl died specifically on the island. But, maybe because Oliver was sleeping with the girl and Slade had that secret love for her, that he later professes to her as she was dying, Oliver did nothing to protect her better or prevent her death? Perhaps Slade feels he would have been better for her and he would have prevented her death. Though, I don't think he was in any shape to protect anyone in his condition at the time.