Adding a PhysicalBoneSimulator3D to my CharacterBody3D's Skeleton makes them behave strange

Godot Version

4.6.stable

Question

Hello!
I am currently making a game where NPC-controlled characters fight eachother
The NPCs are on their own collision layer, set on the Characterbody3D itself, which is a different layer from the terrain layer

But now I added a PhysicalBoneSimulator3D to the skeleton which is part of the NPC scene tree, and it causes the NPCs to be able to pass through the floor and they also are no longer able to be detected by Area3D nodes which are checking for their collision layer

The bones are all set to the same layer as the NPC
I also tried both Jolt and GodotPhysics

EDIT:
this is all before the PhysicalBoneSimulator3D is even set to active or started

Thank you!