How to access SubViewport in Compositor?

[suncyto]

Godot Version

4.6.3

Question

I’m trying to access the texture of a SubViewport’s ViewportTexture from a Compositor in Godot. I am able to link the ViewportTexture to the subview @export variable. However, trying to retrieve and image via get_image returns null. As per the docs, that means the texture is invalid. Relevant code is sectioned off with obvious comments in the code below.

There are no errors thrown. Using the print statements (included in the code below), the image returned is an empty object/null. AFAIK the problem isn’t with copying the texture over or the shader file.

Is this possible to retrieve a SubViewport texture in a compositor?

For reference (not relevant to problem), I used an interlacing compositor effect as a jumping off point.

@tool
extends CompositorEffect
class_name TRSFade

const SHADER_PATH: String = "res://.../trs_fade.glsl"

@export var subview: Texture2D
@export var factor: float
var sbtex: RID

var rd: RenderingDevice
var shader: RID
var pipeline: RID

func _init() -> void:
	effect_callback_type = EFFECT_CALLBACK_TYPE_POST_TRANSPARENT
	rd = RenderingServer.get_rendering_device()
	RenderingServer.call_on_render_thread(_initialize_compute)

func _notification(what: int) -> void:
	if what == NOTIFICATION_PREDELETE:
		if shader.is_valid():
			rd.free_rid(shader)
			rd.free_rid(sbtex)

func _initialize_compute() -> void:
	rd = RenderingServer.get_rendering_device()
	if not rd:
		return

	var shader_file := load(SHADER_PATH)
	var shader_spirv: RDShaderSPIRV = shader_file.get_spirv()
	
	shader = rd.shader_create_from_spirv(shader_spirv)
	if shader.is_valid():
		pipeline = rd.compute_pipeline_create(shader)

func _render_callback(p_effect_callback_type: EffectCallbackType, p_render_data: RenderData) -> void:
	if rd and p_effect_callback_type == EFFECT_CALLBACK_TYPE_POST_TRANSPARENT and pipeline.is_valid():
		var render_scene_buffers: RenderSceneBuffersRD = p_render_data.get_render_scene_buffers()
		if render_scene_buffers:
			var size: Vector2i = render_scene_buffers.get_internal_size()
			if size.x == 0 and size.y == 0:
				return

            # relevant part where issues happen
			if sbtex.is_valid():
				rd.free_rid(sbtex)

			var image: Image = subview.get_image()
			print("null? ", image)
			print("null? ", image == null)
			if image == null:
                # this branch will always be reached
				image = Image.create_empty(size.x, size.y, false, Image.FORMAT_RGBAH)
			else:
				image.convert(Image.FORMAT_RGBAH)
				image.clear_mipmaps()

			var format: RDTextureFormat = RDTextureFormat.new()
			format.width = image.get_width()
			format.height = image.get_height()
			format.format = RenderingDevice.DATA_FORMAT_R16G16B16A16_SFLOAT
			format.usage_bits = RenderingDevice.TEXTURE_USAGE_CAN_UPDATE_BIT | RenderingDevice.TEXTURE_USAGE_STORAGE_BIT | RenderingDevice.TEXTURE_USAGE_CAN_COPY_FROM_BIT | RenderingDevice.TEXTURE_USAGE_CAN_COPY_TO_BIT
			sbtex = rd.texture_create(format, RDTextureView.new(), [image.get_data()])
            # end of problematic part

			@warning_ignore("integer_division")
			var x_groups := (size.x - 1) / 8 + 1
			@warning_ignore("integer_division")
			var y_groups := (size.y - 1) / 8 + 1
			var z_groups := 1

			var push_constant := PackedFloat32Array([
				factor,
				0,
				0,
				0,
			]).to_byte_array()

			var view_count: int = render_scene_buffers.get_view_count()
			for view in view_count:
				var uniform_set: RID

				var color_buffer: RID = render_scene_buffers.get_color_layer(view)

				var dest_buffer := RDUniform.new()
				dest_buffer.uniform_type = RenderingDevice.UNIFORM_TYPE_IMAGE
				dest_buffer.binding = 0
				dest_buffer.add_id(color_buffer)

				var vp_buffer := RDUniform.new()
				vp_buffer.uniform_type = RenderingDevice.UNIFORM_TYPE_IMAGE
				vp_buffer.binding = 1
				print(sbtex.is_valid())
				vp_buffer.add_id(sbtex)

				uniform_set = UniformSetCacheRD.get_cache(shader, 0, [dest_buffer, vp_buffer])

				var compute_list := rd.compute_list_begin()
				rd.compute_list_bind_compute_pipeline(compute_list, pipeline)
				rd.compute_list_bind_uniform_set(compute_list, uniform_set, 0)
				rd.compute_list_set_push_constant(compute_list, push_constant, push_constant.size())
				rd.compute_list_dispatch(compute_list, x_groups, y_groups, z_groups)
				rd.compute_list_end()

[mrcdk]

EDIT: nvm, looks like it possible! check @normalized post below How to access SubViewport in Compositor? - #5 by normalized

Old wrong answer

Godot uses an acyclic graph when rendering so it records and re-orders the drawing commands internally. There’s no guarantee that one viewport is rendered before another one unless something forces an specific order like one viewport using a ViewportTexture from another viewport. The only safe way to access a ViewportTexture is after the RenderingServer.frame_post_draw is emitted.

You can read more about how Godot prepares and executes its rendering in this blog post:

GPU synchronization in Godot 4.3 is getting a major upgrade – Godot Engine

CompositorEffect_render_callback() runs in the render thread which happens before that signal as it’s part of rendering the viewport the compositor effect is attached to so, at that point, accessing a ViewportTexture will be invalid. You should also try to avoid doing too much work on this callback to avoid stalling the render thread.

I don’t think it’s possible to access a ViewportTexture in a compositor effect.

[normalized]

Subviewports are always rendered before the main viewport.

You shouldn’t have any problems accessing subviewport texture from a compute shader. Just pass its RID to the shader as an uniform.

[suncyto]

Thank you @mrcdk and @normalized for responding.

I believe that I now understand how different render ‘pipelines’ (a pipeline per viewport) is managed at the discretion of the engine. That means a compositor effect, which can modify passes done in a single ‘pipeline’, can’t cross over to other ‘pipelines’ reliably. (I use pipelines in quotes because I’m not sure what else to call them, but I mean the overall render pipeline for a viewport).

I’m assuming there is no way to do what I want to do then? That is, to access a ‘color layer’ of a different camera? This is doable with screen shaders but it doesn’t work for me as effects done by the shader aren’t able to be read by another post-processing compositor I have.

And to respond to @normalized specifically, the subviewport cannot be accessed in the compositor script as far as I’ve tried so its RID cannot be used. Plugging the RID of the non-null ViewportTexture, as in…

var vp_buffer := RDUniform.new()
# I changed it to a sampler. This shouldn't affect the problem
vp_buffer.uniform_type = RenderingDevice.UNIFORM_TYPE_SAMPLER_WITH_TEXTURE
vp_buffer.binding = 1
vp_buffer.add_id(sampler)
vp_buffer.add_id(subview.get_rid())  ## here!!

errors out saying that this RID is not a valid texture RID. Could you elaborate on your idea of a solution?

[normalized]

How can I access the unlit scene in either GDShaders or Compositor for PostProcess effects?

So here’s the setup. Runs smooth as butter in the editor. I excluded all boilerplate error checks and thread safety stuff for brevity. Once you get it running you can bring all that back in if needed. Editor: [image] Runtime: [image] Scene: [image] Script in scene root (updates camera transform/fov and SubViewport texture rid in compositor effect): @tool extends Node3D func _process(dt) : var vp_main: Viewport if Engine.is_editor_hint(): vp_main = EditorInterface.get_editor_view…

[suncyto]

Thank you @normalized. You’re linked post is quite helpful and pretty applicable to my situation. More specifically, the RenderingServer.texture_get_rd_texture function is good to know.

One odd thing that I’ve been trying to figure out is that the viewport texture always appears brighter when processed by the compositor. If I don’t pass srgb in the texture_get_rd_texture, it appears erroneously bright, but even if I do pass srgb = true, no matter if I use a sampler or an image2D, the viewport image is still ever so slightly brighter.

This isn’t a problem if used in a screen-shader, or displayed on a TextureRect, but it’s always slightly brighter when used in the compositor and it doesn’t seem to be related to camera/subviewport setups as far as I can tell. Do you know why this happens by chance? (and if possible how I can fix it).

Another thing I tried from your post is use_hdr_2d but that accomplishes the same effect as getting an RD texture as an sRGB (as in, the slight brightness is still there).

[normalized]

Enabling use_hdr_2d for the subviewport should linearize the texture. Make sure that you’re not doing any srgb/linear conversions elsewhere.

[suncyto]

Allow me to clarify my issue. In the below examples, I’ve rendered the real camera on the left half, and the subviewport texture on the right half.

When passing srgb = true to RenderingServer.texture_get_rd_texture →

When enabling use_hdr_2d on the subviewport →

When using neither of the a above options →

The brightness difference in the first two images is the issue I was referring to in the previous reply. To re-iterate on some facts:

• The two cameras are essentially the same (same position, same camera parameters, same environment object).
• The compositor is given the RID of the texture via RenderingServer.texture_get_rd_texture(subview.get_texture()).
• The compositor passes the texture as a sampler uniform (RenderingDevice.UNIFORM_TYPE_SAMPLER_WITH_TEXTURE') as per the code I posted in the previous reply.

Afaik there are no other active settings that could be affecting color space. In fact, all the settings in the subviewport are default parameters, and the subviewport camera matches the parameters of the main camera.

[normalized]

What happens if you render the viewport texture through a regular shader?

[suncyto]

A similar splitscreen test using gdshader (on a subviewport that has use_hdr_2d = true) will show the correct image (no differences), while use_hdr_2d = false will display similarly to the 3rd image in my previous reply (what i assume is misinterpreting sRGB as linearized).

Following images are in order of what I mention directly above:

Below shows if I hide the shader and bring back the compositor effect.

[normalized]

Try with image uniform type.

[suncyto]

Using image2D instead of a sampler yields the exact same result unfortunately.

If it helps, I made a minimally reproducible version of my code (see below) that still shows the odd behavior of mismatching textures. It includes a gdshader version of the splitscreen, and separate versions of the compositor (for image2D and sampler2D).

If you run the project as downloaded it will show an example where 1) use_hdr_2d = true on the subviewport, 2) the image2D version of the compositor effect is running. Other elements are hidden/disabled.

ZIP File of the minimally reproducible version.

One thing that isn’t the same as my original code in this version is that turning on use_hdr_2d while using the gdshader version makes the subviewport portion of the screen darker, when it originally corrected a previously brighter image in my original code. I’ve looked through and I haven’t been able to pinpoint what changed though.

[normalized]

Image pixel format in the compute shader should be declared as rgba16f, not rgba8.

[suncyto]

This doesn’t seem to be the problem. The outcome doesn’t change if you modify the datatype in the shader.

[normalized]

Set the tonemap mode in the environment to linear (and exposure to 1.0).

[suncyto]

That was it! I had experimented around with them at some point. Even then, I don’t see why they wouldn’t render properly in the subviewport texture copy. My last question about this would be how come this discrepancy is there (sounds like a bug but I don’t know enough about the rendering internals just yet to know for sure).

Even so, thanks again @normalized for lending your ear to so many of my questions.

[suncyto]

For future reference, I also had problems with inaccurate shadows. I fixed this by changing the property Positional Shadow Atlas > Size in the SubViewport object.

[normalized]

Tonemap is a post processing effect. It happens after renderer and compositor are done with their jobs.

You had an environment with a non-linear tonemap assigned to both cameras. So it was done once for the subviewport, and then again for the main viewport that composited an already tonemapped image. This effectively applied the tonemap twice on the subviewport image.

So if you still want to use tonemapping, you should render the subviewport without tonemapping (or any other post-processing adjustments) and only do them for the main viewport i.e. on the final composit only. You’ll need two separate environments. The one that’s assigned to the subviewport camera should do the illumination related stuff and the one assigned to the main viewport can then only do post-processing adjustments.