Realistic Human Motion Preservation-Imitation Development on Robot with Kinect

At most, motion generation on robot is usually done through complex computation in off-line mode and straightforward method. In straightforward method, the operator drives robot to certain pose either with moving manipulator tool-tip with hand or remotely. Once the desired pose achieved, the current pose is saved to memory. However, these methods are time consuming. An easy and quick approach is by imitating an object motion to robot with sensing devices. There have been numerous efforts for motion imitation either by using position sensitive detector (PSD) or by using stereo camera. However, a calibrated pre-condition should be done initially, which is not possible for natural movement. Here, this paper proposed motion preservation by capturing human motion naturally through Kinect and then reproduced human motion on humanoid robot simultaneously. In addition, the motions are also preserved in database for later used on robot motion generation and teaching as well. Furthermore, the robot motions are developed to run smoothly and close to human eye ability. The proposed method has been validated in experimental results by capturing and reproducing human motion on robot in rate of 20Hz with340us computation cost for each process.


Introduction
The need for human motion preservation especially in Indonesia is unvoidablle.There have been numerous claims on National traditional dances by other country while teenagers are starting to loose appetite in learning national cultures such as traditional dance.For that purpose, there have been national robot competitions on national dancing performance since three years ago.Here, the dancing should be carried out by humanoid robots since this kind of robot is more similar to human being.Also, interesting challenge in humanoid robot is how to design trajectory control for each joint.Obviously, the robot should have generated motion in memory.
There have been many ways to generate robot motion.For industrial need, robot motion is generated though complex computation to achieve high precision.This is done in offline mode through engineering software.Other method uses straightforward approach.Here, the operator drive robot to certain pose either by moving manipulator tool-tip with hand or remotely.Once the desired pose achieved, the current pose is saved to memory.However, these methods are time consuming especially for long-duration motion.Each pose may takes in minutes.Thus, these methods are not effective for more than hundred poses.Inspite of that, an easy and quick approach is by imitating an object motion to robot with sensing device, or often called motion capture [1][2][3].There have been numerous efforts for motion imitation either by using position sensitive detector (PSD) or by using stereo camera.Unfortunately, initial preparation is required such as mounting detector on object along with cabling or calibrating camera position for each measurement.This is not possible for natural movement.
Programming by demonstration, or also known as imitation learning or motion capture is an answer to a problem how to control humanoid robot joints.This method offers flexibility and easy modification of the robot control.Humans as the actor would make a move, and then the robot as an agent will perform the movements performed by humans.The system is very useful to train the robot to make movements and in the future can be used to create a natural humanlike motion of humanoid robot motion that can do housekeeping, dangerous work, and even entertain humans [4][5][6][7][8].
Some researches on robotic imitation previously havebeen done among others.Hsin-yu et.al developed human motion imitation to humanoid robot through 13 mark points with webcam [2].Nakaoka joint research group also performed traditional dances imitation to robot [9].Eight cameras were used to obtain three dimensional informations of 30 mark points on actor.Similar research was done through markless tracking by Marc Rosanes to imitate the human arm movement on the Bioloid robot with TOF capture device Camera: Swiss Ranger 3000 [10].Generally, all of those were using multi-camera as motion capture devices, which is too costly either in hardware and computation proces due to complex algorithm.Moreover, preparation for environment, mark points and multi camera calibration should be done initially before capturing human motion.
In this paper, realistic motion preservation by capturing human motion through Kinect is proposed.Kinect that usually used along with Xbox games has a stereo camera system.Jing Tong et.al.havedone scanning 3D full human bodies with kinect [3].However, capturing full human bodies parts are not necessary for human motion which is time consuming.With certain library, some of human ankles are captured and extracted along with depth value.Later, the captured human pose is translated to quarternion form of each human joint.This is done naturally without marking human body.This paper used Robot Operating System (ROS) as work platform.In addition, the extracted human motion is preserved in database for later used on robot motion generation and teaching as well.In order to show the imitation ability, an upper part body with 4DOF on each hand are developed.Each DOF is driven with Dynamixel AX-12 servo.The rest problem was to realize realistic motion on these servos through parameter adjustment on servo and modification on ROS library.Here, the operation process is extended to run in rate 20Hz, which is close for human eye ability up to 25Hz.
The rest of this paper is organized as follows; section 2 describes the proposed method and systems used torealize the realistic motion perservation and imitation.Later, tracking process, mapping process and and database design are explained in section 3. Afterwards, section 4 described the experimental results along with analysis.Finally, concluded in section 5.

The Proposed Method
In order to realize the realistic motion perservation and imitation, we proposed markless tracking human motion with Microsoft Kinect and inverse kinematic method to mapping human arm motion to humanoid robot.All process of tracking and mapping worked in Robot Operating System (ROS).Here, we describe the details of robot, kinect and ROS.

Humanoid Robot
The used robot in this paper is humanoid robot consisting of two arms with each arm has four degrees of freedom with dynamixel AX-12 actuators in each joint.The advantages of ISSN: 1693-6930 Realistic Human Motion Preservation-Imitation Development on Robot with Kinect (Abdul Muis) 601 this actuator is to have a feedback position, speed, voltage, temperature, and load.Here is a picture robot modeland its arm movements that used in this paper.

Figure 1. Robot Model and Dynamixel Position
Robot movement is regulated directly by the PC via USB2 Dynamixel.Identification of the configuration of the robot models will be described in the next section.As a source of power supply in this robot, we used a computer power supply.

Microsoft Kinect
Microsoft Kinect is an accessory for Microsoft's Xbox 360 platform.Kinect is motion sensing unit device that was developed to interpret human gestures.It has sensors connected to a motorized pivot based.This device has a RGB camera, depth sensor, and multi-array microphone.With these sensors, Kinect can provide the ability to capture full-body 3D motion, facial recognition and voice recognition.Kinect has a good performance, small dimensions, and open source libraries that are able to make Kinect used on multiple OS platforms (Windows, Linux, and Mac), and also kinect is cheaper than other capture device.In this paper, human motion will be captured using kinect with markless method.With the skeleton tracking software from the Open NI, three-dimensional vectors of the joints in humans are achieved.

ROS (Robot Operating System)
ROS (Robot Operating System) is a framework for the development of robot software.ROS was originally developed by the Artificial Intelligence Laboratory Stanndford, but in 2008 the further development carried out by Willow Garage with more than 20 institutions collaborating in the development of ROS.Currently ROS has supported the development of various robots.
ROS provides a standard OS (Operating System) such as hardware abstraction, low level device control, implementation of common functions of an OS, message-passing between processes and package management.This architecture is based on a graph in which a process is located on a node that can send and receive multiplex sensors, controls, knowing the state, actuators, and other posts.Library of ROS-based UNIX systems (preferably Ubuntu Linux support, while other variations such as Fedora and Mac OS X is still a "experimental").
Thispaper developed a rob_mocap, ros-package that can perform real time saving features of imitation and repetition with the motion that has been imitated.

Research Method
There are three steps for doing imitation learning in this paper.First, we did human tracking to get important skeleton vectors.Second is mapping human motion to robot, joint position and velocity control.Finally, designed database motion, and then extract database to repeated movements that have been made.

Tracking Process
To track the human skeleton, openni_tracker node from stack openni_kinect in ROS is employed.This node will provide three-dimensional vector of 15 points on the human body.To perform tracking, someone needs to stand up and do "surrender" pose in front kinect (raise both hands).Here is pose calibration picture and 15 vector openni_tracker published from this node.

Mapping Process
In mapping process, firstly, kinematics model of the robot must be identified which used to find the inverse kinematic solution of the model later.Figure 4 shows the kinematic chain of the robot's right arm.θ 1 , θ 2 , and θ 3 angles are the rotation angle of the shoulder.And then, θ4 angle is the rotation angle of the elbow.Forward kinematic matrix G from the shoulder to the hand is obtained by multiplying the transformation matrix from T 1 to T 4 .With the matrix T is 4x4 dimension homogenous matrix transformations from a frame that represents the frame rotation by a 3x3 matrix Rgand frame translation is shown by the 3x1 dimensional matrix, tg.
From the model at Figure 4, each frame transformation matrix obtained as follows T2= Rotz(θ2) T3= Rotx(θ3).transl(L1,0, 0) T4= Roty(θ4).transl(L2,0, 0) So the transformation matrix from the shoulder to the hand is: Cartesian coordinates of the hand from the shoulder frame indicated by the elements g14, g24, and g34 in the matrix G.The vector v sh can be expressed in the equation.
In the tracking process with kinectwe got some important vectors (v sh , v se , and v eh ).So that, by use the forward kinematic equations above, we got the inverse kinematic solution which we used to transform human movement to robot.The steps to find the inverse kinematic solution presented as follows.
1) Find θ 4 Angle Solution θ 4 is an elbow internal angle, which is the angle formed between v se (shoulder_elbow) and v eh (elbow_hand).θ 4 angle can be found by using the dot product vector equation.In Euclidean geometry, the angle formed between the two vectors is the dot product of vectors that expressed in an orthonormal basis related to the length and the angle formed between the vectors.The relationship is represented in Figure 5.

2) Find θ 1 and θ 1 Angle Solutions
To find solutions if angles θ 1 and θ 2 can be used forward kinematic equations from the shoulder to the elbow.The position of the elbow from the shoulder is the function of three first joints, ie.(8) Expanding the matrix equation above will yields the following inverse kinematic solutions for θ 1 and θ 2. (10) 3) Find θ 3 Angle Solution θ 3 angle is the internal rotation angle of the hand.The solution of this angle can be found using the equation in xx after obtaining the solutions of θ 1 , θ 2 , and θ 3 .

  
In order for the imitated movements in real time, itneed to map each joint angular velocity.Speed control block diagram is shown in Figure 6.

Motion Database Design
Motion database is needed in order the robot can be able to repeat the results of imitation motions.In this paper, human motion is saved in a csv extension by writing in the following format.With that format, less memory is saved because there is no need to keep the speed memory of each joint.Velocity of each joint can be derived from the substraction of the current and previous angle position divide by time difference.

Validation Invers Kinematic Solutions
There are 7 DOF in human arm (3 DOF in hand), to make robot motion looks like human natural motion at least there are 4 DOF in robot arm.In other research, some robots are only having 3 DOF in their arm.Error of validation invers and forward kinematic using 3 DOF in arm is so high.This sectionprovides validation testing of inverse kinematic solution to test the equations that have been obtained previously.Testing is done by taking images of vectors in the following positions.Once these vectors are obtained, the inverse kinematic solutionis computed like the steps explained previously.And then, performed forward kinematic process and compared the results of the forward kinematic vectors which generated by the solutionwith reference vectors.The compared results show the errors of solution vectors and reference vector are 0%.

Real Time Imitation
Process of real time imitation can be seen in Figure 10.The experiments were realized in 20Hz with single loop computation mapping process tooks about 340us.Since the process is close to human eye ability, there is no significant different between human motion and generated motion on robot.Thus, the imitation looked realtime.The robot motion was done through inverse kinematic mapping process.In addition, some weakness conditions of multi-solution inverse processes were anticipated to make robot follow human the provided kinect capture library in ROS only support up to 5Hz.

Motion Record and Generation
The results of recording motion database and generated motion are shown in Figure 11.The recording motion data are shown in reference position (qref).While the generated motion data on robot are shown in result position (qres).Although there is a slight delay in generated motion due to actuator response, those figures showed the robot was able to imitate the preserved motion.Mainly, the generated speed is limited to the actuator behavior.The utilized servo in this paper is driven through serial communication with 115kbps, where each command for each servo took almost 1ms.Here, having higher quality actuator would extend the motion generation performance.Nevertheless, generated motion on robot in this paper is sufficient since the generated motion still close to human eye ability.Also, the results are better than results as shown in references [1], [2], [9] since this paper used kinect with computation cost less than 0.5ms while others used multi cameras that took more computation cost on processing images.

Conclusion
A realistic motion preservation and imitation are realized in this paper.The preservation is done by capturing motion through Kinect and saved the extracted motion to database.At the same time, the upper part of humanoid robot imitated human motion.By tuning Robot Operating System, the proposed method has been validated in rate of 20Hz which is close to human eye ability.Here, the ROS rob_mocap package that has been developed, successfully the challenges of the imitation process well humanoid robot capable of performing database extraction and storage and motion that has been done by humans.Also, extraction of inverse kinematic solutions, real time process of imitation, and the motion database extraction of the robot has shown good results.
The used model robot consists of two arms with each arm consists of four degrees of freedom.Configuration of the model robot in normal position (each joint at 0 o ) is shown in the figure below.

Figure 4 .
Figure 4.Comparation Robot and Human Right Arm

Figure 6 .
Figure 6.Position and Speed Control Block Diagram

Figure 8 .
Figure 8. Testing Pose of Validation Inverse Kinematic Solution

Figure 10 .x
Figure 10.Real Time Imitation Result

Table 2 .
Validation Inverse Kinematic Solution Result Realistic Human Motion Preservation-Imitation Development on Robot with Kinect (Abdul Muis) 607