I've created a basic theme with these settings:

Where the font is a DynamicFont with these settings:

I've set this as the default theme under Project Settings > Gui > Theme > Custom.
I created a scene with a single Label, with no configuration beyond its text and position.
When I run the scene though the word "Error" with black lettering appears behind the label.

In the editor the label doesn't use the new font when I set the theme until I restart the project. After that I also see "Error" behind the label in the editor.
Is this a bug in Godot? An error in my theme settings? A bug in the font's TTF?