In the study of TCPs, Zou et al. ^[9] proposed a novel computational approach to address the era in which a painting was drawn by using the appearance and shape features extracted from the paintings. Gao et al. ^[10] extracted features from two aspects, the first is color details and hue angle, and the other is angle features such as stroke, shape, and line. Then, scale-invariant feature transform (SIFT) feature detector and edge detector are fused to fully describe the visual differences in important areas. Sheng et al. ^[11] applied computer vision technology to study different applications of Chinese painting, such as the sentiment classification of Chinese paintings via feature recalibration of deep network aggregation. Fan et al. ^[12] discussed the visual complexity of Chinese paintings from the aspects of line, texture, light, and shade, and selected six color levels for statistical quantification, thus proposed to measure the visual complexity and richness of works. Cohen et al. ^[13] proposed an unsupervised semantic segmentation algorithm for paintings based on domain adaptation. In short, the current research on TCP image segmentation is mainly based on traditional machine learning algorithms, but the research on semantic segmentation methods based on deep learning is not sufficient.

Feature representation is the key step of the segmentation algorithm, and the quality of feature representation has a direct impact on segmentation results. Different researchers proposed different feature representation models. For example, the U-Net^[14] realizes scale prediction by feature combination through skip connection. The single shot multibox detector (SSD)^[15] method realizes multi-scale fusion by detecting features of different layers at the same time. A top-down architecture with lateral connections feature pyramid network (FPN)^[16] is developed for building high-level semantic feature maps at all scales by leveraging the inherent multi-scale pyramid hierarchy of deep convolutional networks. Kirillov et al. ^[17] added boundary detection information in the process of clustering to improve the accuracy of the model. Visual geometry group network (VGGNet)^[18] effectively enhances the abstract expression and refinement capture ability of the network for image features by introducing a deeper network hierarchical structure and adopting a smaller filter unit. DeepLab series^[19-20], on the basis of retaining and developing the core technology of dilated convolution, optimize the ability to capture and utilize multi-scale features of images and refine the segmentation results gradually. Entropy^[21], in the field of communication, is expressed in mathematical language as the expectation of the amount of information that can be obtained from all possible values of a variable, i. e. , from all possible events. In recent years, many scholars have applied information entropy to the field of image processing. Such as, Celik^[22] proposed a spatial entropy-based global and local image contrast enhancement algorithm by considering the distribution of spatial locations of gray levels of an image. Zhou et al. ^[23] proposed an image feature description method based on local entropy, which solved the problem that the traditional feature description method SIFT would lose some important information when describing image features at a single scale. Chen et al. ^[24] proposed an automatic salient object detection algorithm based on maximum entropy estimation. Considering that the color of Chinese painting images has the characteristics of halo and gradual change, the existing convolutional network is difficult to effectively extract the positional relationship features between Chinese painting instances and the whole, so the FPN^[16] structure is used for multi-scale feature fusion, and the instance information entropy representation is formed for the feature representation of Chinese painting instances segmentation.

Instance segmentation, which assigns different labels to separate instances of the same class of objects, is the most challenging problem in the field of image segmentation. Semantic contour detecton (SCD)^[25] can be regarded as the earliest instance segmentation algorithm, which combines the segmentation task and the detection task for the first time. Instance segmentation based on object detection first generates the detection region of each instance object, and then predicts the mask of each region by using the detector,such as faster region-based convolutional neural network (Faster R-CNN)^[26], etc. ResNet^[8] is a method that regards instance segmentation as a task of first detection and then segmentation. The main steps of this method are to first extract features from images by ResNet +FPN, then scan these feature maps to generate proposals, and finally generate feature maps on regions of interest and output semantic segmentation masks. This approach can be traced back to deep mask^[27], which uses a sliding window to generate instance masks. Liu et al. ^[28] proposed an adaptive pooling method based on the instance segmentation framework to improve information retention. Mask region-based convolutional neural network (Mask R-CNN)^[29] is a simple and effective instance segmentation method with good segmentation performance on a single object. Segmenting objects by locations (SOLO)^[30] views the task of instance segmentation from a completely new perspective by introducing the notion of instance categories, and achieves better accuracy on the single-shot instance segmentation. PolarMask^[31] formulates the instance segmentation problem as predicting the contour of an instance through instance center classification and dense distance regression in a polar coordinate. BlendMask^[32] can effectively predict dense per-pixel position-sensitive instance features with very few channels, and learn attention maps for each instance with merely one convolution layer, thus being fast in inference. In recent years, there have been a series of studies on the improvement of Mask R-CNN. For example, Cai et al. ^[33] proposed Cascade R-CNN, which improves the performance of the model by gradually improving the prediction and adaptive training. In the CenterMask^[34] model, an anchor-free instance segmentation method is proposed to alleviate the saturation problem in the VoVNet model. Object scale inconsistency is one of the difficulties in instance segmentation, and multi-scale feature fusion is a good solution^[35]. Although the above methods have achieved good results in their respective fields, these methods are difficult to take into account the large and small targets at the same time, and they all rely on the mask candidate regions generated by the original image, and do not make full use of the features extracted by the input network of a large number of images, so it is difficult to meet the accuracy requirements of TCP image segmentation.

TCP belongs to the visual art of artificial modeling on a plane by using lines, colors, and structures. Therefore, color and lines are two core elements for TCP creation. Considering the characteristics of intermittent discontinuous contours and haloed boundary colors in the TCP image, the regression direction and distance of each sampling point are calculated to get the contour entropy reflecting the boundary information of the instance. The three-dimensional (3D) color information and gradient of each sampling point are calculated to obtain the color entropy reflecting the pixel color distribution of the instance. The color entropy and the contour entropy constitute the instance information entropy of the instance representation.

1) Contoure entropy

Considering the characteristics of intermittent discontinuity of contours and haloing of boundary colors in TCP images, a mask generation algorithm using contour entropy based on contour regression in the subsequent mask branch network is proposed.

The process starts with the input of the feature maps of each size obtained in the feature extraction phase to the subsequent mask branch to generate the heat map. Then, the center coordinates of the instance object are located by the target detector center-based object detection network (CenterNet)^[36]. Finally, the initial contour is obtained by regressing the centroid bias, i. e. , calculating the offset of the contour points with respect to the centroid coordinates, based on the feature maps and centroid features.

The adjustment of the initial contour requires an iterative process of random sampling, relative position judgment, and calculation of moving distance. First, a number of points e₁,e₂,... ,e_n,... ,e_N are randomly selected from the initial contour, and for a particular point e_n, the classifier judges the relationship between this point and the instance position. That is, the classifier predicts whether the points are inside or outside the instance by randomly selecting pixels around the boundary of the real instance and extracting fine-grained features with respect to the boundary relative position input. Then, the high-resolution feature maps obtained from the feature extraction stage are further encoded through a convolutional layer to form the position vector p of the point e_n.

Finally, the vector p is input into the multi-layer classifier to iteratively adjust the initial contour, so as to finally obtain the contour entropy vector p_f, which represents the position of the instance boundary points.where d represents the moving direction and w represents the point moving distance. The move distance w is related to two factors, one is the instance size and the other is the confidence of the relative position of the point e_n. The larger the area of the instance object, the larger the moving distance and the faster the convergence rate. The confidence degree of relative position of the point e_n is obtained by the classifier. When the point e_n move once, the classifier will update the corresponding position of the point e_n accordingly, thus obtaining the next regression moving distance w iswhere k is the given coefficient, A is the shift amplitude, and M(·) is the multilayer classifier. The schematic illustration of the contour point regression is given in Fig. 2.

2) Color entropy

The one-dimensional (1D) gray-scale entropy reflects the aggregation characteristics of the distribution of image pixel values, but the color space and contour information of an image contain more image information. Instances in different kinds of TCP have different tones and structures, such as flowers and birds in flower and bird paintings both have rich and vivid colors, but flowers for instance have similar internal colors, and bird instances have more varied internal colors, so the color features play an important role in identifying different instances.

In the Lab color space of digital image I, since two color channels a and b do not interfere with each other, the color entropy in color space information of image channels a and b is calcutated aswhere the i represents the gray value of the color channel [ - 127, 128], and c (i) represents the proportion of pixels with gray value i in the a channel and b channel. For a picture, in addition to color richness, primary color is an important color feature. The dominant color ratio of the image and the information entropy of the image affect each other, the higher the dominant color ratio, the lower the color information entropy, and vice versa. In the Lab color space, the dominant color ratio can be computed aswhere R_L,a,b denotes the color characterization of the image, H_L,a,b and M_L,a,b denote the information entropy and dominant color percentage of the image on the L, a, and b channels respectively.

Since TCP images usually have the characteristics of gradual halo in color, the extracted image features are fused to 1/4 size and then input them into the position branch. The specific process is to divide the image into n ×n regions first, and then calculate the quantization of color features belonging to that location for each subregion, so as to fuse the color space features with the image spatial structure to obtain the feature vector of n × n × [R_L,R_a,R_b]. The variance of the four neighboring regions above, below, left, and right of each region block is calculated for R_L, R_a, R_b of different regions respectively, to obtain the color entropy E, which reflects the correlation degree of the color space.where R_L, R_a and R_b denote the color radius within different color channel regions, respectively. ,, and denote the average values of the color radius in the corresponding regions, respectively. α, β and γ denote the corresponding weight coefficients.

3) Information entropy

In image processing, entropy can estimate the average information content of an image^[37]. The information entropy formula is expressed aswhere p(x_i′) shows the probability of occurrence of event x_i′. From Eq. (6), it can be seen that information entropy has non-negativity, cumulativity, and monotonicity, which makes it widely used in probability theory, astrophysics, images, and other fields as a measure of effective information.

In this paper, inspired by the concept of information entropy^[38], the information entropy theory is introduced to calculate the information entropy for each spatial element, which generated by the base segmentation network according to the contour entropy p_f and the color entropy E. By calculating the information entropy for each spatial element of the feature map generated by the base segmentation network, we can determine regions with high certainty and regions with low certainty in an image, so as to establish connections between them and to enable information to flow from elements with low entropy to elements with high entropy to reduce the uncertainty of the elements to which information flows.

For an input TCP image I, let define {I(x,y)|0≤ x≤H-1,0≤y≤W-1}, where H and W denote the length and width of image I respectively, and assume that I has a dynamic range of [I_d,I_u], where I(x,y)∈ [I_d,I_u]. The feature map h_c = softmax(p_f) and h_E = softmax(E) denote the feature maps generated by the base segmentation network. Using the feature map h_k(x,y), which generated by the base segmentation network according to the contour entropy vector p_f and the color entropy E, the information entropy measure S(h_k) is computed as

The information entropy measure S(h_k) is used to compute a discrete function f_k according to f_k =. The discrete function f_k is further normalized by, anddenotes cumulative distribution function.

The mask branch and position branch constitute the core components of the SSC-Net architecture. The color entropy and contour entropy are input into the mask branch and position branch of the network structure respectively, so as to realize TCP instance segmentation integrating spatial structure.

1) Mask branch

The mask branch uses contour regression to predict the mask for each instance. Sampling points are randomly selected on the edge during each iteration, and the regression direction and distance of each sampling point are calculated to achieve boundary prediction. It is responsible for predicting the mask containing instances at each position. The structure of the mask branch is shown in Fig. 3.

The mask branch divides the image into n × n = M grid regions, each of which can be understood as an instance position class. Starting from the top left, mark from left to right and from top to bottom. If an instance is in the (x, y) position class, the nx +y channel will predict the mask belonging to that instance. If n is set too large, the predicted mask for an instance will increase, resulting in redundant computation. If n is set too small, a grid area may contain multiple objects, which means that one channel predicts multiple instance masks, losing the meaning of instance position mapping and affecting the utilization of position features. We set n = 8 according to the experimental comparison.

The 5-layer feature map is input into the mask branch, and the position coordinates of each instance (x, y) are input into the last two channels of the layer. The coordinates need to be normalized to between [ -1, 1], the number of channels becomes 256 +2 =258 channels. Take the number of grids set at the current layer, and the output feature size is W × H ×n ×n. Where H is the image height and W is the image width. By combining spatial structure features with multi-scale features, the problem of instance overlap is solved to a certain extent and the accuracy of the model is improved.

The loss function of the mask branch iswhere N_Pi is the number of predicted instances, and p_n′and g_n′ denote the contour point locations of predicted instances and ground truth, respectively. Considering the efficiency of the model, 128 sets of sampling points on each instance boundary are selected for regression.

2) Position branch

The position branch obtains the composition of example semantic categories and instance positions in the image by feeding the color entropy of the current region to the feature map channel and the classifier. Unlike prediction mask, semantic information and instance location information are less affected by feature map scale. Therefore, the position branch does not need to input all the 5-layer features of the backbone network, but through up-sampling and down-sampling, the features are unified to 1/4 size, and divided into n ×n =M grid areas.

In position branch, there are multiple cases of different semantic categories in a position block, in order to reduce the computation, the instance is only kept with the highest confidence, i. e. , a position region only belongs to one instance. When there are multiple instances in a region, the instance with the largest percentage of masks in the region will determine the semantic category and region composition determination of the location region. First, the color entropy of each neighboring region is calculated by Eq. (7), and the image feature maps are encoded using the same fully-connected layer and then jointly input to the classifier to construct the difference layer. Then, the Euclidean distance of the two regions is calculated using the feature encoding, color entropy, and semantic category information, and the confidence that the two regions belong to the same instance is output using the sigmoid activation function after the single-node fully connected layer. The instance location prediction process is shown in Fig. 4.

After completing the prediction of all positional semantic categories and instance compositions in the image, the loss function is calculated by comparing with the ground truth to determine the proportion of instances in the ground truth, so that a positional region only belongs to one instance. The position branch loss function L_p consists of two parts, the classification prediction and the position prediction, is shown aswhere M is the number of location regions, L_c is the semantic category prediction loss, L_l is the location prediction loss, and λ is a given coefficient.

3) Combinatorial output

The ResNet+FPN network extracts 5-layer of features such as 1/4, 1/8, 1/16, 1/32, 1/64 of the image, and inputs them into the position branch and mask branch for prediction. The prediction marks the channel mask belonging to a certain instance as positive samples, and the others as negative samples. Then, the final instance mask prediction result is obtained by adding the weights. Finally, the prediction results are upsampled to the original image size to complete instance segmentation.

In the position branch, each spatial position class represents an area, and this branch predicts the composition of the area blocks for each instance. In the mask branch, each spatial position class maps to an instance mask prediction. By combining the outputs of the two branches through spatial position class correspondence channels that do not belong to instance predictions are filtered out, and masks belonging to the same instance are linearly combined by weighting and selected by threshold. The final segmentation prediction is obtained. Unlike the two-stage model that first performs object detection, the position branch is composed by predicting the instance location, improving the model's perception of instance shape and overall spatial structure of the image. It is also associated with the mask branch through spatial position classifications, and the linear addition of multiple channels of masks further improves segmentation accuracy. The final model is represented aswhere N_r is the number of location regions for an instance, N_i is the number of instances of the image, is the weight obtained by quantizing the prediction confidence in the position branch with a specific step size, and is the mask prediction belonging to the instance channel.

The loss function L_s of the model consists of two-branch loss, L_s is shown aswhere L_m is the loss of the mask branch and L_p is the loss of the position branch.

The software environment for SSC-Net training is Ubuntu 18. 04 operating system, Python 3. 8 language environment, PyTorch 1. 9. 1 deep learning framework, OpenCV image processing library, CUDA 11. 1 platform. The hardware environment is 12-core CPU 2. 5 GHz, single NVIDIA RTX 2080Ti 11 GB card. Different models are trained for different iterations (18 000 or 36 000 or 90 000) with an initial learning rate of 0. 01.

The experimental dataset mainly comes from various book scans and digital collections of different museums. The dataset consists of 1 000 images from three categories of traditional Chinese realistic painting: figure painting, flower and bird painting, and animal painting. Considering factors such as the color and layout integrity of TCP, while maintaining image semantics, data augmentation is carried out using methods such as flipping, scaling, and cropping. After data augmentation, the dataset was increased to 3 000 images. All images in the dataset are more than 1 000 pixels of width and height, and an aspect ratio of 1.03. The images were divided into training and testing sets in an 8∶2 ratio. Use the EISeg annotation tool to annotate the dataset into 5 categories: person, horse, bird, flower, and dog. The annotated data is saved in JavaScript object notation (JSON) file format.

The AP of a category and the mean AP (MAP) of all categories indicators are used to qualitatively evaluate the experimental results^[39]. For the convenience of experimental explanation, different categories of APs are calculated for algorithm performance evaluation. In this paper, the AP₅₀ and AP₇₅ represent APs when intersection over union (IoU) is equal to 50% and 75% , respectively, AP_S represent targets with pixels not greater than 32 ×32 pixels in the segmentation area, AP_M represents the objects with pixels greater than 32 ×32 pixels but less than 96 ×96 pixels in the segmentation area, AP_L represents objects with pixels not less than 96 × 96 pixels in the segmentation area.

The procedure for checking the validity of the contour regression method of the mask branch is as follows. First, the feature map extracted by backbone is input into the mask generation module of Mask R-CNN to obtain n × n instance mask predictions based on pixel regression, and then multiply and add them with the position branch to obtain the output result. Finally, the contour regression method validity test of the mask branch is obtained. The verification results are shown in Table 1. It can be seen from the experimental that the contour-based regression has a significant improvement effect on the objects with large area, and the overall segmentation effect is improved by 2%.

Table 2 shows the experimental results of the effectiveness of the position branch. Instead of assisting the segmentation results through the instance position composition prediction, the final result is composed of the mask filter output of each channel directly in the mask branch. It can be seen from the experimental results that the position branch has a high improvement in accuracy, especially for the instance object with large area.

The number of contour regression points is a factor of the image experimental results. Too many sampling points will cause the model bloat and reduce the efficiency, while too few sampling points will not cover the image fine. Table 3 shows the experimental results of the number of contour regression points. According to the experimental results, the SSC-Net model has obvious improvement effect when the initial sampling points are increased, and then the growth is slow and approaches saturation. Therefore, 128 points are finally selected for contour regression in this paper.

ResNet^[6] is used as the backbone. While keeping other conditions constant, experiments are conducted using the two most common network structures of ResNet, ResNet-50 and ResNet-101. Table 4 shows the experimental results of different backbone on different target categories. It can be seen that when choosing ResNet-101 as the backbone network, the AP reaches 55.682%,which is 1.786% higher than ResNet-50 is used us the backbone network. Therefore, ResNet-101 is chosen as the backbone of the feature extraction network.

In order to verify the influence of the introduction of FPN on the detection accuracy of different size targets under different scale information, Table 5 shows the influence of whether FPN is connected in the model on the output effect under the condition of other network structures unchanged. Among them, when removing the network with FPN structure, the feature map extracted by ResNet-101 is sampled to the same size as before and input to the position branch and mask branch. It can be seen from Table 5 that FPN structure fusion of multi-scale features has significantly improved the segmentation results, and AP has increased by 8. 431% on average. From the APs and AP_L results, it can be seen that the access of FPN structure improves the detection accuracy of small and large objects at the same time, because the FPN structure realizes the prediction and fusion of multi-scale features. Therefore, the ResNet-101 +FPN combined structure is used as the backbone network of the model in this paper.

The number of instance location classes is an important parameter in the ResNet + FPN model. If n is set too large, it will not only cause an increase in calculation and training difficulty, but also make the final prediction effect worse because of the excessive number of channels. However, setting n too small will make insufficient utilization of spatial features and decrease segmentation accuracy. The experimental results are shown in Table 6. According to the experimental results, when n is less than 8, the accuracy of the model changes significantly with the increase of n. When n is greater than 8, the accuracy of the model grows slowly and there is a loss to fall back. Therefore, considering the accuracy and parameter quantity, n = 8 is selected in this paper, that is, the image is divided into 64 position regions for the experiment.

Fig. 5 shows the location channel feature maps output by the SSC-Net model on each location channel, and it can be seen that the location mask of SSC-Net model performs well in capturing the spatial location of the instances. It shows that the location branch of SSC-Net achieves the acquisition of the instance location in the image by feeding the color entropy of the current region into the feature map channel and classifier.

In order to verify the superiority of SSC-Net model on TCP dataset, ResNet-50 and ResNet-101 are used as backbone networks, respectively, SSC-Net with the state-of-the-art algorithms that perform instance image segmentation, including Mask R-CNN^[29], SOLO^[30], BlendMask^[32], PolarMask^[31]. The experimental result is shown in Table 7. It can be seen that SSC-Net is 1. 414% higher than the Mask R-CNN model with the best performance among the comparative algorithms, which illustrates the effectiveness of SSC-Net. Compared with the same algorithm, the performance of ResNet-101 as the backbone network is better than ResNet-50. In terms of algorithm efficiency, a lightweight version of SSC-Net outperforms the other comparison algorithms with of MAP 53. 89% at 25. 8 FPS evaluated on a singleNVIDIA RTX 2080Ti card.

Fig. 6 shows the qualitative experimental results of SSC-Net model, which shows that the SSC-Net model can achieve effective instance segmentation of the specific objects of Chinese painting, especially it also has good robustness on Chinese paintings with large object scale changes.

For example, the SSC-Net model can effectively segment the objects information such as zhinü, children, horses, and dogs in different original Chinese paintings. The qualitative segmentation results further verify the effectiveness of SSC-Net model for TCP data segmentation.