Just remove the same index you assigned... So, change your code to something like (untested):
var sprites = [
"res://Img/black.png",
"res://Img/green.png",
"res://Img/griss.png",
"res://Img/orange.png",
]
func _color():
var index = randi() % sprites.size()
$Sprite.texture = load(index)
sprites.remove(index)
Note, that you need to use the current size of the array to select the next element, since the size changes with each removal...