Found an answer on the Discord from HeraldOfFire.
“You want the texture origin setting.”
If anyone else encounters this:
Texture Origin formula is: - (larger tile size in px / 2) + half base tile size in px
So, if base tile/grid size = 32x32, and you make a tile that is 4x4 (so 128x128px), formula is: - (128 / 2) + 16 = -48
Also, if you’ve already created collisions, you will likely have to re-position the polygon.