This reminds me of my issue where, when the virtual keyboard on a mobile device pops up, the entire viewport briefly resizes, distorting the size and shape of the controls (assuming you are using scaling, which you essentially must). There’s no way to “freeze” the viewport resolution in this situation and, on Android anyhow, the Android app setting that should ignore resizing doesn’t work. It could be fixed if there were a way to tell the viewport to maintain its current size and not automatically scale.
Similarly, here, could it be as simple as adding a toggle to containers to maintain their own size and stop auto-adjusting based on their children sizes and positions?