Polities face two competing risks with chief executives. If given too free a hand, they can commit atrocities with impunity.* But if the law gives too little freedom of action, it can be manipulated by successors to regularly punish predecessors purely for ideological or partisan purposes.
And there is no law or legal interpretation that can guarantee against both of these risks simultaneously. Only a healthy political culture can.
Unfortunately, the growth of executive power in the US, combined with our growing partisan divide and growing intensity of hatred of the other side, makes it harder to maintain the political culture that can find its way between these two dangers
*I do think we have given presidents too free a hand, and that's part of why unilateral executive action has continued to expand.