The prime minister’s biggest risk right now is from some of his own supporters.