Most of the human detection methods are using HOG (Histogram of Oriented Gradients). In the case of fixed camera environment, it is possible to make background model using GMM (Gaussian mixture model) and easily extract motions using background subtraction. However, it is difficult to recognize pedestrians among extracted motions. In this paper, we propose an efficient coarse-to-fine pedestrian detection framework which combines motion detection and HOG cascade to make a faster pedestrian detector. Firstly, motion detection is used as the coarse detection in order to reduce the area of interest to be covered by the pedestrian detector. Then HOG cascade which detects pedestrians is executed only on the blobs or ROIs selected from the coarse detection. The experimental results on PET2009 768X576 dataset show that proposed method of which processing speed is 11.46 fps is 7.5 times faster than HOG and 2.2 times faster than HOG cascade.