Breast cancer is one of the major threats to the human being. Early identification can prevent some of the premature deaths. Manual methods are sometimes very tedious and time consuming. Moreover manual diagnosis can be prone to error. Automated analysis can reduce the overhead of the manual diagnosis and reduce the error. In this work, an automated computer assisted framework has been proposed to analyze and detect the type of the disease from the current condition of the breast. Histological slides have been used for automated diagnosis. SIFT based feature selection and extraction method has been used followed by a Bag-of-Features method. The extracted features are classified by a metaheuristic supported Artificial Neural Network. Three models have been compared in terms of accuracy and obtained results are reported in a comprehensive manner. © 2018 IEEE.