Recommender systems based on Collaborative Filtering (CF) techniques have achieved great success in e-commerce, social networks and various other applications on the Web. However, problems such as data sparsity and scalability are still important issues to be investigated in CF algorithms. In this paper, we present a novel CF framework that is based on Bordered Block Diagonal Form (BBDF) matrices attempting to meet the challenges of data sparsity and scalability. In this framework, general and special interests of users are distinguished, which helps to improve prediction accuracy in collaborative filtering tasks. Experimental results on four real-world datasets show that the proposed framework helps many traditional CF algorithms to make more accurate rating predictions. Moreover, by leveraging smaller and denser submatrices to make predictions, this framework contributes to the scalability of recommender systems.