Implement handwriting recognition in Godot

Question

I am trying to make a Mandarin learning game for my kids. They hate writing the Chinese characters, so I'm thinking whether I can make a game where the player writes Chinese characters, and if their writing matches (within a range of accuracy) the actual writing, then they can (i dunno) kill the monsters.

How could I go about doing something like this?

For example, is it possible to train (as in machine training kind of train) the game what the acceptable handwritings are for each character, and then use the model trained to score any writing by the player to give an accuracy, and if the accuracy is above 70%, then give green light to kill the monster?

Comments

I don't think you can do this with Godot.But you can use python opencv library to recognize characters.You can make some server on python which gets data(images) from your godot application ,then sends back the result.

---

Hi
I can not help you on the godot side, but if you are new to NeuralNetworks the following links might help. If you already know NeuralNetworks you can skip my comment.

As general introduction I found 3Blue1Brown on youtube rather helpfull:
https://www.youtube.com/watch?v=aircAruvnKk

Codelabs lets you use external hardware (you need a gpu) and gives you hands on tutorials for a betters start.
https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist#0

Maybe you will find a pretrained network for your problem, but if not there you can design, train and export a new network. For the training you will need a big amount of labeled data, in your case lots of pictures of Chinese characters with notes what character this is. I am sure you will find such a database if you search for one on the internet.

Answer 1

Handwriting Gestures

First break your game into 3 groups

• Gesture Recognition
  
  
  • This nice and simple app
• Character Identification
  
  
  • Either OCR
  • Compare Images using RMS
  • Compare hasables
• Reward System
  
  
  • Totally up to you

Happened across this post and remembered your question since i liked it and was able to make exactly what you wanted by modifying that script and compiling the tesseract-lib using GDNative even for Android as well, but know that there are also pre-build binaries you can use.

I had alot of fun doing this so i feel like spoon feeding you would be robing you of that Joy and i also can't remember all the steps i took.

P.S. Use Line2D instead to draw you get much better results

Comments

My version of the ChalkBoardApp i use the Thread for character recognizing functions which are excluded here

extends ColorRect

export var pen_color = Color(1, 1, 0)
export(float, 1, 50) var pen_size = 5
export(float, 1, 10) var pen_delay = 1

onready var viewport = Viewport.new()
onready var board = TextureRect.new()
onready var pen = Line2D.new()
onready var thread = Thread.new()
onready var thread_timer = Timer.new()

var mouse_pos = Vector2()
var tex : Image

func _ready():
    var rt = get_rect().size

    viewport.set_size(rt)
    viewport.set_usage(Viewport.USAGE_2D)
    viewport.set_clear_mode(Viewport.CLEAR_MODE_ONLY_NEXT_FRAME)
    viewport.set_transparent_background(true)
    viewport.set_vflip(true)

    pen.set_joint_mode(Line2D.LINE_JOINT_ROUND)
    pen.set_begin_cap_mode(Line2D.LINE_CAP_ROUND)
    pen.set_end_cap_mode(Line2D.LINE_CAP_ROUND)
    pen.set_default_color(pen_color)
    pen.set_antialiased(true)
    pen.set_width(pen_size)

    # Use a sprite to display the result texture
    board.set_texture(viewport.get_texture())

    add_child(viewport)
    add_child(board)
    add_child(thread_timer)
    viewport.add_child(pen)

func _input(event):
    if event is InputEventScreenTouch:
        if not event.pressed:
            mouse_pos = Vector2()
            pen.clear_points()
        else:
            mouse_pos = get_local_mouse_position()
    if event is InputEventScreenDrag:
        if event.relative.length() > pen_delay:
            mouse_pos = get_local_mouse_position()
            pen.add_point(mouse_pos)

Answer 2

Implementing handwriting recognition in Godot can be a challenging task but a rewarding one. The post raises some important points on the subject, such as the use of pre-trained models and the importance of labeling data for training purposes. I would like to recommend reading the article https://realwealthbusiness.com/points-of-labeling-individual-elements-for-training-data-in-ml/ as it provides valuable insights into the labeling process.Additionally, it's important to consider the type of handwriting recognition algorithm to be used, such as Optical Character Recognition (OCR) or Neural Networks. Both have their advantages and disadvantages, and the choice will depend on the specific requirements of the project.Overall, implementing handwriting recognition in Godot is a complex but feasible task, and with the right approach, it can lead to exciting possibilities in game development.