This probably isn't the optimal method, but it's easy to set up at least. The scene should look like this:
root
mask_sprite1
mask_sprite2 ...etc
BackBufferCopy
background
building_sprite1
building_sprite2 ...etc
unit_sprite1
unit_sprite2 ...etc
The mask sprites are in the same positions as the corresponding building sprites and have the same textures. The BackBufferCopy is set to copy the viewport. The mask-sprites are drawn with this fragment shader:
COLOR = color(1.0, 0.0, 0.0, tex(TEXTURE, UV).a);
The unit-sprites are drawn with this fragment shader:
uniform float outline_thickness = 2.0;
uniform color outline_color = color(1.0, 1.0, 1.0, 1.0);
color col;
if(texscreen(SCREEN_UV).r < 1.0) {
col = tex(TEXTURE, UV);
}
else {
float alpha = 4 * tex(TEXTURE, UV).a;
alpha -= tex(TEXTURE, UV + vec2(TEXTURE_PIXEL_SIZE.x * outline_thickness, 0.0)).a;
alpha -= tex(TEXTURE, UV + vec2(-TEXTURE_PIXEL_SIZE.x * outline_thickness, 0.0)).a;
alpha -= tex(TEXTURE, UV + vec2(0.0, TEXTURE_PIXEL_SIZE.y * outline_thickness)).a;
alpha -= tex(TEXTURE, UV + vec2(0.0, -TEXTURE_PIXEL_SIZE.y * outline_thickness)).a;
col = color(outline_color.rgb, alpha);
}
COLOR = col;