Family breakdown seriously damages children's development, says former Conservative leader