After looking into this, I found that the TranslationServer (as of now) seems to be always disabled in the editor.
This is because in editor_node.cpp there is a call: TranslationServer::get_singleton()->set_enabled(false)
After commenting that out and rebuilding Godot I am now able to see translations. I'm not sure why it has to be disabled in the editor though.
UPDATE: I found out that Label's set_text method instead of storing the key will store the translation. This means with the above modification, Labels will lose the correct key value in their text property and change it into the translated value. This is why the TranslationServer is disabled in the editor. The better solution though would be to have Label store the key and only translate when rendering (or possibly have a local cache of the translation)