As a well known filter in the category of order statistic filters, median filter has been widely employed in the fields of image processing, computer graphics and computer vision etc. This paper proposes an approach for implementing the median filter with varying kernel sizes at each different pixels. The proposed algorithm consists of two kinds of histograms. One is column histograms for all columns, the other is a kernel histogram for kernel window. The column histograms are updated according to the scales when moving to the next row, while the kernel histogram is updated using column histograms when moving to the next pixel. Since the kernel sizes are varying at each pixel, the updating of column and kernel histograms uses operations of addition and subtraction depending on the scales. Compared with the brute-force implementation, the experiments show that the proposed algorithm is very fast and effective.