Justice may, as Hume and Smith seemed to believe, entail nothing other than refraining from harming the life, liberty, or property of others.
But morality is more expansive, and requires more than merely adhering to a concept of negative rights.
The reason this conception of justice is useful is because just governments shouldn't concern themselves with moral matters more broadly. Protection of negative rights is completely at odds with enforcement of positive rights, and this is true whether the involvement is in enforcing religious lifestyle preferences to replace that undertaken by consensual adults, or secular redistribution of wealth and moral busybody regulations on individuals and their businesses.
Theocrats and progressives, it's all the same. They're as close as socialists and fascists, and just as many people misconstrue them as being on opposite ends of a spectrum rather than a half step away.
Libertarians focus heavily on rights rather than duties, because rights should matter more in the political realm of life.
But not all life is politics, and there are individual duties that matter, laying completely outside the realm of just law. And if society fails to broadly accept those duties, it makes it much harder to make a convincing case for rights.
Justice is only one portion of morality, and can only be expected to do its' thing if the other bits are broadly tended to as well.