Conservative voters quit party over same-sex 'marriage' Act