Robot Copy Human

Overview

This blog talks about the development of movement detection and then the passing to the robot to perform the action.

Background

After the basics were understood of detecting the movement of the user and then sending it to the robot, then creating the range of detectable movements was the next step.

Activity

Based on the previous work complete of using a use case to run the IR codes. This could be extended to allow for every IR code. More have since been gathered from research on the internet. Including new moves by combining current IR codes and joining them together to perform actions the original robot couldn't perform at once. This totals to 21 different actions. However all these will not be used, although are available if need be.

A list of movements that were required was compiled seen below:

• Left arm up and down
• Right arm up and down
• Arms up together
• Walk
• Turn Left and Right
• Turn arms in and out

Each move was first detailed by observing a human performing the actions and comparing the limitations of the robots angle of movement. The human observation also took in to account what joints could be used to detect their movements, For example moving arms up with not be usful by using the X axis rather comparing the Y axis of the wrist with another joint that stays stationary on the Y axis, such as the spine as seen in the code below.

 skeleton.Joints[JointType.WristRight].Position.Y > skeleton.Joints[JointType.Spine].Position.Y

To stop the arm from continualy trying to move up if the user leaves the arm in the upright position a system needed to be developed to detect if an action si being performaed as to not sent the IR codes again and again. to do this, a array of all actions was made that would state whether an action was being executed already. By using the same numbering system currently being used to send the IR code variables  These could be used as identifies of the array fir each action. The list below is the actions and their number as integers for use in identifying the array slot. This show the extra command that had been implemented nearer the bottom.

 int IRSTurnRight = 01;
        int IRSRightArmUp = 02;
        int IRSRightArmOut = 03;
        int IRSTiltBodyRight = 04;
        int IRSRightArmDown = 05;
        int IRSRightArmIn = 06;
        int IRSRightHandSweep = 07;
        int IRSRightHandStrike2 = 08;
        int IRSWalkForward = 09;
        int IRSWalkBackward = 10;
        int IRSTurnLeft = 11;
        int IRSLeftArmUp = 12;
        int IRSLeftArmOut = 13;
        int IRSTiltBodyLeft = 14;
        int IRSLeftArmDown = 15;
        int IRSLeftArmIn = 16;
        int IRSStop = 17;
        int IRSWakeUp = 18;
        int IRSBurp = 19;
        int IRSRightHandStrike = 20;
        int IRSNoOp = 21;
        int IRSHigh5 = 22;
        int IRSFart = 23;
        int IRSLeftHandStrike = 24;
        int IRSLeftHandSweep = 25;
        int IRSWhistle = 26;
        int IRSRoar = 27;
        int IRSArmAllTheWayUp = 28;
        int IRSArmAllTheWayDown = 29;
        int IRSLeftArmAllTheWayUp = 30;
        int IRSLeftArmAllTheWayDown = 31;

Once this was established it was possible to check if the action was currently in motion and prevent it from running again by including it in the IF statement alongside the joint locations. This also allowed the program to detect if the action has been perfomed and then if the users arms have lowere then in the send the IR code that lowers the robots arms. The full code can be seen below.

if (skeleton.Joints[JointType.WristRight].Position.Y > skeleton.Joints[JointType.Spine].Position.Y
      && activeCommand[IRSRightArmUp] == false
      && skeleton.Joints[JointType.WristRight].Position.Y < skeleton.Joints[JointType.ShoulderRight].Position.Y)
  {   //Run action

      Thread thread = new Thread(send);
      thread.Start(RSRightArmUp);
      //thread.Start(RSRightArmUp);
      activeCommand[IRSRightArmUp] = true;
      

  }
  else if (!(skeleton.Joints[JointType.WristRight].Position.Y > skeleton.Joints[JointType.Spine].Position.Y) && activeCommand[IRSRightArmUp] == true)
  {   //Stop

      Thread thread = new Thread(send);
      thread.Start(RSRightArmDown);
      activeCommand[IRSRightArmUp] = false;
      

  }

This was then done for all other movements. Each one has taken a large amount of alpha testing. However the walking was the most complex to develop to achieve a simple way of doing it.

To allow the user to simulate walking (walking on the spot) the Program had to detect that the user was move both legs over a period of time unlike all the pre-existing movements that used a one fix point this had to have 2 joints they moved over time (the knees). The knees were chosen as the anchor point as they move the further forward when walk after human observation. A timer system was needed to insure that the 2nd leg movement follow swiftly after the 1st otherwise stop the robot. To achieve this a timer run on an independent thread could have been used however I felt this was over complicating matters. As there was already a timer in place that ran a method every 500 milliseconds. An integer is simply incremented every 500 milliseconds. By using the integer to time how long between leg movements, it was possible to stop the robot if no leg movement has happened in the last second. The code below show it checking the Knee Z axis and also if the turn left/right action is running as this caused issues during testing. I then check how long between the last movements and depending on the time it send the appropriate IR code.

if ((skeleton.Joints[JointType.KneeRight].Position.Z > (skeleton.Joints[JointType.KneeLeft].Position.Z + 0.12)) 
|| (skeleton.Joints[JointType.KneeLeft].Position.Z > (skeleton.Joints[JointType.KneeRight].Position.Z + 0.12)) 
&& (activeCommand[IRSTurnLeft] == false && activeCommand[IRSTurnRight] == false))
  {
      
      if (count < 2 )
      {
          Thread thread = new Thread(send);
          thread.Start(RSWalkForward);
          activeCommand[IRSWalkForward] = true;
          commandLog.Text = "Walk\n" + commandLog.Text;
          label1.Content = "Walk";
          countTF = true;
         
      }

  }
  else
  {
      if (count > 1)
      {
          //stop robot
          label1.Content = "stopped";
          Thread thread = new Thread(send);
          thread.Start(RSStop);
          activeCommand[IRSWalkForward] = false;
          count = 0;
          countTF = false;

The next stages are fine tuning the movements such as minimizing the time it takes to run through a method by modified the Arduino code.