I mean... An increase in population can affect housing availability when supply is limited, as there's more people chasing a more limited amount of essential goods. Like... If zero new housing was created and the population expanded, individual housing would be more expensive because of the imbalance.
But neither population nor housing are fixed pies, and we're pretty far from running out of available land as a natural limit.
Steel-manning the immigration argument, it makes some sense if you can't change the first bit. If new housing construction is significantly restricted and there's a large population increase from immigration, that's upward pressure on prices.
Now, there's other factors that make it less simple. Without immigration, we're essentially falling below replacement rates.
Immigrants tend to need different types of housing. An aging population means different types of housing needed for an aging population (think everything from downsizing to retirement communities). But it's not a crazy worry to have or anything.
I just think that opening up new housing construction by removing impediments would make the immigration concern less relevant, and given the types of immigration we have, it could even put downward pressure on construction costs.