Kinect To Arduino

Overview

This blog is outlining the progress made connecting the Kinect application to the arduino and showing results based on a users actions.

Background

Development of the Kinect to Arduino has been in full swing recently. Has the arduino will be instructing the robot what to do from data received from the Kinect application, testing has been in production to create a way of doing this.

Activity

Now the Kinect application can connect to the the arduino is is time to send the data when a user is performing an action. To this the Kinect had to detect the user doing something. I decided it would on the distance of the right hand from the Kinect. This would be using the Z axis.
The z axis is a value to 15 decimal places and testing purposes this number is rounded up to 4 decimal places using the code below.

            double RightHand = first.Joints[JointType.HandRight].Position.Z;
            double multiplier = Math.Pow(10, Convert.ToDouble(4));
            double current = Math.Ceiling(RightHand * multiplier) / multiplier;
            label4.Content = current;

Firstly the kinect has to store the current position of the hand so it can be acted upon. This is stored in a variable called "current". A variable that will be used to store the previous position is checked to see if it has been used. See code below

 if (prev == 0)
            { 
                prev = current;
                return;
            }

The default value of "prev" is 0. Resulting in the current position to previous for use in the next frame. The next frame will then process the current position and check to see how much it has changed from the previous position. It will then change the content of a label to the direction the hand is moving.

if (prev >= current + 0.01 || prev <= current - 0.01)
            {
          
                if (prev < current)
                {
                    //back
                    String mes = "back";
                    send(mes);
                    label2.Content = "back";
                    prev = current;
                }
                else
                {
                    String mes = "forward";
                    send(mes);
                    //forward
                    label2.Content = "forward";
                    prev = current;
                }
            } 

There is a buffer of 0.01 either way because the Kinect sensor is so sensitive it is very hard to keep a hand so still to not change the value. The buffer allows for this and will only change if the value exceed the 0.01 threshold.  The "send(mes)" runs a method that is detailed below.

public void send(String message)
        {
            TcpClient tcpclnt = new TcpClient();
            ConState.Content = "Connecting.....";
            try
            {
                tcpclnt.Connect("192.168.0.177", 23);
                ConState.Content = "Connected";      
                String str = message;
                Stream stm = tcpclnt.GetStream();
                ASCIIEncoding asen = new ASCIIEncoding();
                byte[] ba = asen.GetBytes(str);
                stm.Write(ba, 0, ba.Length);
                tcpclnt.Close();
            }
            catch (Exception)
            {
                ConState.Content = "Not Connected";
                return;
            }
        }

This send the message to the Arduino using the TCP stream outlined in a previous Blog post.
This successfully sent to Arduino and was monitored on the serial monitor. However an issue that has become apparent the fact that the messages are sent in byte and must be joined together to form a string on the arduino to perform actions if for example "back" was sent across and an if statement condition was used. A solution may be to use Single letters to send to arduino for decryption. This will be detailed at another time.
For now the arduino is receiving a different message depending on the action performed by the user on the Kinect.


The Next Step

The next step is to develop the gestures for the Kinect to recognise to control the robots different movements. The robot is still not acquired but will be soon (hopefully) including the power devices to power the wireless device and Arduino without the need to USB Power.