Image retargeting is a technique of adapting images to different aspect ratio for various display terminals such as cellphone, personal digital assistant (PDA) and projector. This paper proposes a method to deal with the problem using visual saliency. The method includes two steps: computing a saliency map, and modifying the image to suit different aspect ratio. A novel visual saliency model is used. The model indicates the weighted sparse coding residual as visual saliency. In the step of weighted resizing, a fast algorithm is proposed. It is a fast image adaptive process with no obvious distortion. The experimental results indicate the robust and effectiveness of the proposed method with different images.