×

Activity Summary

If you’re accessing this activity directly, did you know there are nine other activities in this series up on our website? Check out our AI page to see a breakdown of the activities and our recommended order to complete them in! Also, these activities introduce AI concepts and terminology. If you find yourself unfamiliar with any of the words in this activity, the landing page also has a glossary of AI terms. Happy space-station-fixing!

To recap: You and your group-mates are astronauts and scientists aboard the Actua Orbital Station. Unfortunately, your station just got bombarded by magnetic rays and your electronics have begun to shut down! The only one who can save you is the orbital station’s AI, DANN. DANN stands for Dedicated Actua Neural Network, and it’s gone a little loopy. Brush up on your technical skills, learn about AI, and save yourself and your crewmates!

DANN’s visual core is almost complete, thanks to your hard work in “Sort Things Out: Exploring Image Classification”. But we aren’t done yet! Before the magnetic storm, DANN was a state-of-the-art AI system that you could interact with using your voice. But during the shut-down, DANN’s audio core was knocked offline. The problem is, it doesn’t look like we can access the audio core without DANN’s help first. Our mission specialists think that you might be able to use DANN’s visual core to communicate, but the visual core isn’t working either! We need to train an image classification model to recognize our hand shapes as commands, and then make sure they’re refined and ready to go in “What Machines See: Digging into Machine Vision”!

In this activity, participants will train an AI model to complete an image classification task. They will use Google’s Teachable Machine platform to classify images of hand gestures that can be used to control a computer program. Participants will create datasets to use for training the AI model and in doing so, will learn about how AI programs understand the information that they’re presented with.

 

Activity Procedure

Our mission specialists think that you might be able to use DANN’s visual core to communicate, but the visual core isn’t working. We need to train an image classification model to recognize our hand shapes so that we can get access to the audio core and reinitialize it. Reinitializing one of DANN’s cores is a fairly straightforward process, but the steps have to be completed in a certain order. Most of the steps use specific commands:

  1. DISENGAGE core safety interlocks.
  2. DISMOUNT core from station operating system.
  3. SHUTDOWN core.
  4. REINITIALIZE core.

The commands within each step are executed using the keyword in all-caps. For example, to complete the first step, we need to send a “DISENGAGE” command. If a core is successfully reinitialized, it should be automatically engaged by the station’s operating system. You’ll need a way to give these four commands, in order, and we can’t use DANN’s audio systems to do it.

Activity 1: Training your AI model

The two pre-trained models encountered in “Sort things out: Exploring Image Classification”, while impressive in their own way and potentially useful elsewhere, don’t seem like a good fit for what we need. They can identify many different objects with varying degrees of success, but they aren’t currently set up for recognizing the hand shapes that you will be using as commands. It’s looking like you might need to train your own model to help you reinitialize DANN’s audio core. Luckily, the mission specialists included instructions to help you through the process. They read:

IN CASE OF EMERGENCY, A CLASSIFICATION MODEL CAN BE TRAINED USING TEACHABLE MACHINE.

  1. LOAD TEACHABLE MACHINE
  2. DEFINE CLASSES
  3. CREATE TRAINING DATA
  4. TRAIN MODEL
  5. TEST MODEL

AFTER MODEL IS TRAINED AND TESTED, IT CAN BE CONNECTED TO AN INPUT CORE.

Load Teachable Machine

Teachable Machine can be accessed online here: https://teachablemachine.withgoogle.com/train/image. That link will bring you to Teachable Machine’s image classification model. Teachable Machine has two other models: a body pose-based classification model and an audio-based classification model. We will explore these in later activities. For now, you should focus on training Teachable Machine’s image classification model to recognize the hand shapes that we’ll be using to communicate with DANN.

Define classes

Defining classes is a key part of a classification task: you need to tell your model what you are looking to identify (e.g. “I think this image is in Class A, and not Class B or C”). As humans, we can rely on a number of sensory inputs (as well as our own memories) to help us complete classification tasks. An image classification AI model, on the other hand, only has the data from the images that you’ve shown it to make its guesses as to which class an image might fit into. It can only make informed guesses. 

Remembering back to our initial briefing, you should recall that there are four commands, one for each step in the core reinitialization process. We need the image classifier to recognize each one:

  • DISENGAGE
  • DISMOUNT
  • SHUTDOWN
  • REINITIALIZE

These will be the four classes that you will need your classifier to recognize, but you should also add a class for when the model doesn’t recognize any of these classes: an UNKNOWN label.

  1. In Teachable Machine, rename any existing classes (Class 1 and Class 2, by default) to the names of the commands.
  2. Add the remaining classes, by clicking on “⊞ Add a class” below your existing classes, for each of the remaining commands and “UNKNOWN”.
  3. When you are done, you should have five boxes, one for each class. They should be to the left of the middle box entitled “Training”.

Create training data

For each of the four commands, you will need to come up with a unique hand shape.

(Facilitator note: ensure that all shapes chosen are appropriate and not offensive. If in doubt, provide predetermined options for participants to choose from).

 You will be teaching your AI model to recognize these shapes through a process called training. Your image classification model learns by looking at visual examples of each class, so that it can begin to tell the difference between them. Rather than relying on existing images of your chosen hand gestures, which may be difficult to find, you will create the training images using Teachable Machine’s built-in webcam function. The webcam captures images that the model can then use for training. The mission specialists included some instructions on how to proceed with this phase:

  1. Define and/or agree upon a hand shape for each of the four commands.
  2. Sketch these shapes in the space provided.
    • Note: these shapes don’t have to convey any actual meaning (e.g., your shape for DISENGAGE doesn’t have to somehow communicate the idea of disengaging). The only requirement is that the shapes need to be distinct enough that the program can tell them apart easily.
    • In Teachable Machine, for each command class:
    • Click on the “Webcam” button in the class box. You may be asked for permission to use your webcam. After training, unless you specifically save your Teachable Machine project, the images gathered will be discarded.
    • Record images of your hand shape. You have two options for recording:
      1. Use “Hold to Record” to gather images yourself or have another person press “Hold to Record” for you. You should record for 3 to 5 seconds.
      2. Click on the gear icon (⚙️) to access the recording settings and turn “Hold to Record” off. This will enable a timed recording mode that you can use to record your images for a fixed duration after a small delay, suggested to be 3 to 5 seconds.
    • If you have group mates or a friend nearby, repeat the recording process with them also making the hand shapes.
  3. For your UNKNOWN class:
    • Repeat the recording process but for hand shapes that do not resemble your command classes. The purpose of this class is to provide your model with a non-command class guess option. Make sure you make lots of different hand shapes for this label, as it is meant to be the catch-all for any other gestures!

Train Model

Now that you’ve created examples of what you want your AI model to recognize, you can have your model examine the images in each of your classes and begin distinguishing between them. To do this:

  1. Click on the “Train Model” button in the box labelled “Training”.
  2. Wait for training to complete. After a short while, below the training button, you should see a timer counting up and a number out of 50. Your training is complete once that number reaches 50 out of 50, but, depending on the number of training images you have, this might take a few moments.
  3. When training is complete, the “Preview” box should have an “Output” section which displays your trained model’s classification of your webcam image and its confidence in its classification.

Test Model

You need to check if your model’s been trained well enough for you to use it to communicate with DANN’s visual core. This means that your model should reliably detect each hand shape that you trained it on and classify everything else as UNKNOWN. The mission specialists have included some questions to consider for evaluating if your model is functioning well:

Does your model accurately classify, for each of your command classes, your specified hand shapes…

  1. …when made by you or other group members used in the training data?
  2. …when made by other people who were not part of the training data?
  3. …when made against a different background from the training data? (A different location, angle, or different light levels?)

If yes to all of the above, your model is ready for use. If not, consult the section on troubleshooting, below:

  1. Does your training data include any images that are not good representations of the hand shape for the class that they are in? What do you do if there are some images that shouldn’t be in there?
  2. Does your model work on some command classes but have difficulty recognizing specific command classes? Looking at the hand shape specification and training data, could you hypothesize why this might be?

You also notice a note that’s been included below the evaluation and troubleshooting questions:

If evaluation results are not satisfactory, a model can be re-trained once more training data has been added. In many cases, additional training data, or better training data, will solve issues of model reliability and accuracy. A model that can reliably pass at least one of the evaluation questions may still be loaded into the visual core at the discretion of the user. Reliable performance, however, cannot be guaranteed.

Activity 2: Using your AI model

To get your trained model from Teachable Machine into DANN’s visual core, you will first need to export it from Teachable Machine.

  1. Once your model has been trained, you will need to go to the “Preview” box on the right and click “Export Model”. This will pop up an export menu.
  2. By default, the correct settings have been selected, however just in case they have been changed, confirm the following:
    • The “Tensorflow.js” tab is selected.
    • Under the “Export your model:” header at the top, “Upload (shareable link)” is selected.
  3. You should be able to click on “Upload my model”. Click “Upload my model”.
  4. It may take a few moments (10-15 seconds in good Internet conditions) for your model to be uploaded, but once it’s complete, you should now have a link under “Your shareable link:”. This link should be in the following format: https://teachablemachine.withgoogle.com/models/xx-xxxxxx/

This the URL to your trained model. We can use this to connect your model to DANN’s visual core!

Activate the window below and paste the URL for your Teachable Machine model where the window prompts you to. This will load your AI model into DANN’s visual core.

Pretty sketch where the model URL can be pasted. This step will also describe steps to exporting your model. The sketch will use the provided model and require a hand shape to be successfully detected for an amount of time before accepting it as a command. Confidence threshold will be adjustable.

  1. Once your model is loaded, you can adjust the confidence threshold. Any images below your threshold will be ignored by the operating system. The timing threshold indicates how long you have to hold the shape for it to be recognized.
  2. Using your hand shapes, you must enter the commands to re-initialize the visual core (in order).
    • The core has a failsafe to help limit incorrect inputs, so you must hold each hand shape until the system has confirmed that your input has been registered.

Reflection & Debrief

As you enter the last command, you feel a wave of relief as the screen displays the text “REINITIALIZATION IN PROGRESS”. Time slows as you wait for the reinitialization process to conclude, but eventually, a new notification appears: “AUDIO CORE ONLINE”.

While you’re eager to move on to working on the newly reinitialized audio core, first we must think back to the questions that we used to evaluate our trained models:

Does your model accurately classify, for each of your command classes, your specified hand shapes…

  • …when made by you or other group members used in the training data?
  • …when made by other people who were not part of the training data?
  • …when applied in a different environment from where the training data was generated (e.g. a different part of the room, a different background for the webcam, different levels of light)?

{ Individually, reflect on / As a whole class, discuss } the appropriateness of these questions for confirming that your model is adequately functional, specifically:

  1. What is different between these three questions? Do you think the differences are important? If so, why? If not, why not?
  2. What might the answer to each question tell us about how well our model is trained?

Considering the task as a whole:

Other questions to reflect on:

  1. Do you think shapes or gestures are a good way to interact with a computer system? Why? Why not?
  2. Not including voice control (which will be addressed in a subsequent activity), what other ways could you improve the interactions with the station’s systems and DANN’s AI?
  3. How could visual recognition of hand gestures be applied to real world problems? Where could it make things easier?
  4. Can you think of any AI used in the real world that have similar functions to Teachable Machine? How are they useful?

Extensions & Modifications

How might you adapt the time, space, materials, group sizes, or instructions to make this activity more approachable or more challenging?

Extensions 
  • Expand the training data for your image classification model. How many different gestures can you train it to recognize? Provide each label with more training data. How does that affect the quality of the model? Are there any significant changes? Does the average confidence go up?
  • Have someone else test your trained model. Does it recognize their commands? What could you do to improve that recognition?
Modifications 
  • If a participant is struggling with the activity, focus on one gesture to begin with. See if the program can distinguish between two possibilities: The gesture is being made, or it isn’t. Create data for those two options, and train the model.
  • In a virtual environment, each participant is able to run through the activity on their own. Ensure that they have access to the proper links. 
  • If participants wish to combine training data, Teachable Machine allows you to zip input images and add them to a google drive. From there, another computer can install them directly into Teachable Machine.

Downloads

This website uses cookies to ensure you get the best user experience. By continuing to use this website, you consent to our use of cookies.
Accept