Weird fog issues with Rendering 3d Viewports

Godot Version

godot 4.2.2


I have a portal that is a viewport of a camera somewhere else in the scene, and it is crucial it is as seamless as possible. If I enable fog (either volumetric or regular), there is a very noticable discoloration between the viewport mesh and the regular mesh. See youtube link for what I mean.

What can I do to solve this? I would really like to keep the fog for the atmosphere.