This paper proposes an efficient way of background modeling and elimination for extracting foreground information from the video, applying a new block-based statistical feature extraction technique coined as Block Based Quantized Histogram (BBQH) for background modeling. The inclusion of contrast normalization and anisotropic smoothing in the preprocessing step, makes the feature extraction procedure more robust towards several unorthodox situations like illumination change, dynamic background, bootstrapping, noisy video and camouflaged conditions. The experimental results on the benchmark video frames clearly demonstrate that BBQH has successfully extracted the foreground information despite the various irregularities. BBQH also gives the best F-measure values for most of the benchmark videos in comparison with the other state of the art methods, and hence its novelty is well justified. © 2017 IEEE.