Modifying the PitchScale of a pitch shift effect on a bus causes the bus to be distorted

Godot Version

v4.3.stable.mono.official [77dcf97d8]

Question

I have a function that simulates a slowdown effect in my game through several means, most of it visual, but I decided to try and modify the audio bus “Master” to get a slowdown effect on all audio being played by adding a pitch shift to it then modifying that with the exact same values as I use to change the Engine’s Speed.

private void ModifyEngineTimeScale(double value)
{
    Engine.TimeScale = value;
    AudioManager.Instance.ModifyBusPitch((float)value);
}

This code is called with a Tweener:

public void SlowdownTime(float forSeconds = 1, bool alsoZoomIn = false,
    double withZoom = 1.5, Node2D onObject = null)
{
    ScreenManager.Instance.SlowdownEffect(forSeconds, alsoZoomIn, withZoom, onObject);
    
    Tween tween = GetTree().CreateTween();
    tween.TweenMethod(Callable.From<double>(ModifyEngineTimeScale), 0.25, 1, forSeconds);
}

And finally, the class where “ModifyBusPitch” lives looks like this:

using Godot;

namespace Predation.Scripts.Managers;

public partial class AudioManager : Node
{
	public static AudioManager Instance;

	private AudioEffectPitchShift _pitchShift;
	public override void _Ready()
	{
		base._Ready();
		Instance ??= this;

		_pitchShift = new AudioEffectPitchShift();
		AudioServer.AddBusEffect(AudioServer.GetBusIndex("Master"), _pitchShift, 0);
	}
	
	public void ModifyBusPitch(float speed)
	{
		_pitchShift.PitchScale = speed;
	}
}

Now, this works, however, after the first time a “slowdown” effect happens, and after the pitch is first modified then brought back to it’s original state, the audio becomes distorted and sounds like it’s more “distant” than it should be. I printed the _pitchShift.PitchScale value constantly, and it goes from 1 which is the default, down to what I needed it to, then even after it gradually came back up to exactly 1, the audio was no longer the same as it was before. Does modifying the PitchScale also modify something else under the hood that I don’t know about?

I attached a “video” file here to show what’s happening. I cut the visuals out as they’re not important. The first bit of the video shows what things should sound like, then in the middle I use the slowdown effect, and the last part of the video / audio shows what the distortion is like.

This is still an issue for me, anyone knows why it happens or how to avoid it?