Surface defect detection is very important to ensure product quality, but most of the surface defects of industrial products are characterized by low contrast, large variation in size and category similarity, which brings challenges to the automatic detection of defects. To solve these problems, this paper proposes a defect detection method based on convolutional neural network. In this method, a backbone network with semantic supervision is applied to extract the features of different levels. While a multi-level feature fusion module is proposed to fuse adjacent feature maps into high-resolution feature maps successively, it significantly improves the prediction accuracy of the network. Finally, an encoding module is used to obtain the global context information of the high-resolution feature map, which further improves the pixel classification accuracy. Experiments show that the mean intersection of union (mIoU) of the proposed method is superior to other methods on a standardized defect detection dataset of steel strip (NEU_SEG, mIoU of 85.27%) and a magnetic-tile defect dataset (mIoU of 77.82%).