You don’t have enough tiles. You want a tile to handle each permutation when you go to paint them.
You aren’t using the right approach to the bit mask. Think about it as painting the “walkable” regions. (For example, in the image you linked the small red square should be filled in.)
Consider watching this video for an overview of the concepts:
Before attempting to build your own tile set from scratch I recommend finding an image online of one that someone else has created with all of the required tile shapes, apply a bit mask to it so you can see it painting correctly, and then paint over the image with your own artwork.