I likely have some of the same inspirations as you although mine is perhaps more in the Skyrim realm as I have played Skyrim VR a lot and wanted to do some kind of VR project and play with interactions in VR. Without repeating what I have posted about mine, just playing with the capabilities of Godot has made me convinced it is perfectly possible to make a nice and rich VR world in it. The art assets will always be sub-par to whatever AAA studios would have so I have to be sober in this department, and most of them downloaded from SketchFab for a very very long time except for some specific use cases that i want to try out things.
I am also a developer who generally dont like frameworks and plugins as they often limit what I want to do instead of help me. Yes sure you can quickly get something running but after a while the limitations often surface and you then likely need to figure out how to modify it or get the people who made it to expand on it (which is very hard). So in my case I just made my own procedural generation to test out what I could comfortably run at 72 hz refresh rate and was quite surprised at how performant Godot is in the 3D department. In time I will pre-generate worlds that can be edited also to have some artistic control where I feel it is needed/wanted since its supposed to live on a multi user server anyway (likely coded in Nodejs).
Like you I am exploring this as a fun hobby project and have no plans to release anything either at this point. If I see it becomes something fun and solid I can reconsider but that is years away.
I will likely upload some kind of video to showcase some elements just to share progress in the near future.