It seems that the get_frame_texture() returns an AtlasTexture, with the whole spritesheet, but also the right region.
Then using get_frame_texture().get_image() returns an Image exactly as we want it.
var texture: AtlasTexture = sprites.get_frame_texture("default", 2)
var new_texture: ImageTexture = ImageTexture.create_from_image(texture.get_image())
var material = self.get_surface_override_material(0)
material.albedo_texture = new_texture
As this is expensive, I will cache those ImageTexture for later loops.