Figured it out. Thanks for the reply!
var input_dir = Input.get_vector("move_left", "move_right", "move_up", "move_down")
var direction_ground = Vector3(input_dir.x, 0, input_dir.y).rotated(Vector3.UP, $camera_pivot/Camera3D.rotation.y)
Be sure to use the actual rotated node. So if you only rotated $camera_pivot/Camera3D at maybe 45deg for an orthogonal view, you must use the rotation.y of that node.
$camera_pivot/Camera3D.rotation.y
I wasn’t able to rotate $camera_pivot instead, it had position issues probably due to the camera being positioned differently as well. I will figure out the new correct position later.
and I found a way to make rotation smooth
velocity = lerp(velocity, direction_ground * speed, 4 * delta)
velocity.y = velocity_y
Right now I am just rotating the player model, but I can also rotate whatever I do need facing forward later.
$visuals.rotation.y = atan2(velocity.x,velocity.z)
What helped me: