Brain tumor detection is the prior diagnosis process to separate the regions having tumor cells from the regions having normal brain tissues. Artificial Intelligence (AI) algorithms, particularly Deep Learning (DL)-based methods have shown remarkable progress in the domain of medical image analysis by applying the concept of object detection. Most of the earlier object detection methods follow the classification-based approach, which uses a two-stage process to select the interesting regions in the image first and then classifies objects present within those regions by using Convolutional Neural Network (CNN) separately, hence not suitable for real-time applications. On the other hand, regression-based methods predict the class as well as bounding box co-ordinates in one run and apply the entire CNN to the whole image, making the process faster. In our proposed system, we are using a two-fold approach; where in the first fold we are adopting a detection framework by considering You Only Look Once (YOLO) model, which uses the concept of regression-based learning mechanism. In the second fold, we are adopting a transfer learning-based training process, where a model trained on one task is re-purposed on a second related task and preferable for small dataset. We have also created a custom dataset containing nearly 1100 Magnetic Resonance Imaging (MRIs) containing both High-Grade-Gliomas (HGG) and Low-Grade-Gliomas (LGG). As part of this study, we have adopted different versions of YOLO models like YOLOv2, YOLOv3, and YOLOv4 and used them for the task of brain tumor detection. © 2021, The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.