In order to recognize the actions of human’s lower limbs, a novel action recognition method based on a human joint was proposed. Firstly, hip joint was chosen as the recognition object, its y coordinates were as recognition parameter, and human action characteristics were achieved based on filtering and wavelet transform. Secondly, an improved self-organizing competitive neural network was proposed, which could classify the action characteristics automatically according to the classification number. The classification results of motion capture data proved the validity of the neural network. Finally,an action recognition method based on hidden Markov model (HMM) was introduced to realize the recognition of classification results of human action characteristicswith the change direction of y coordinates. The proposed action recognition method needs less action information and has a fast calculation speed. Experiments proved the method hada high recognition rate and a good application prospect.