Bitmasks work by trying to match colour to colour, and non colour to non colour. In the example you gave, the only bitmasks that were marked for the grass tiles were:
Which means they'll only be selected as an appropriate tile when surrounded by tiles on all sides. Edge tiles will look like this (on a 2x2):
[ ][ ] [X][ ] [X][X] [ ][X]
[X][X] OR [X][ ] OR [ ][ ] OR [ ][X]
When the autotile goes to place an edge tile, it checks for the appropriate bitmask. If it doesn't find any that fit, which in this case it won't, it instead selects whatever the default, icon tile is set to.
If I'm creating an autotile where I want the edge tiles to be the same as the middle tiles, I use 5 of the same image. One for the middle, and one for each of the four edges. If I want to have multiple, random middle tiles, then I'll just add more, different images, and set their bitmask to the "middle" setting, aka the first one I wrote.