Chancellor Angela Merkel of Germany thought she had fended off a mutiny within her own conservative alliance after returning from marathon talks with fellow European leaders on Friday with an agreement on how to handle migration.

But instead, the rebellion escalated late on Sunday, further weakening Ms. Merkel, who was once seen as a rock of European politics and the guardian of the liberal world order.