if tileset for a terrain is 3x3, it works as mentioned in most tutorials. But mytileset terrain loks like this, consisting of 5x5 tiles. I tried many patterns for bit-mask, but none of them work. They give weird random results.
As of now, 5x5 auto-tiling isn’t possible in Godot.
There are alternatives, though:
you might like to use an external program for autotiling, such as Tiled (which has Godot 4 export built-in), or LDtk (more versatile than just autotiling. You’ll need a Godot addon to import from LDtk, though.)
You could place the tiles manually, rather than depending on autotiling. This is much more tedious, but takes away the limits of Godot’s implementation of autotiling.
I want this feature too, so I submitted an issue on Godot proposals to implement it natively (#12303).