SpringArm3D margin not working

Godot Version

v4.2.2.stable.mono.official [15073afe3]

Question

I have a third person camera attached to a SpringArm3D. And it mostly does what I want and expect, but the margin does not seem to do anything. No matter what i do, the camera keeps clipping into objects and jittering.

Player Collision layer is set to 2
Enviroment Collision layer is set to 1

Collision Mask: 1
shape: none (Will get to this later)
Spring lengh: 5m
Margin: 1m

I can change the margin as much as I want, nothing changes. If I add a shape, and specify the margin on that shape, it works. But when assigned a shape the behavoir changes, and allows other objects to be between the player and camera as long as the camera position is not inside the object.

I also tried assigning the margin via script, but it did not help.

I have not seen any other issues about this, and as such I am assuming I am doing something wrong. I probably need a shape for it to work properly