Existing tag signal detection algorithms inevitably suffer from a high bit error rate (BER) due to the difficulties in estimating the channel state information (CSI). To eliminate the requirement of channel estimation and to improve the system performance, in this paper, we adopt a deep transfer learning (DTL) approach to implicitly extract the features of communication channel and directly recover tag symbols. Inspired by the powerful capability of convolutional neural networks (CNN) in exploring the features of data in a matrix form, we design a novel covariance matrix aware neural network (CMNet)-based detection scheme to facilitate DTL for tag signal detection, which consists of offline learning, transfer learning, and online detection. Specifically, a CMNet-based likelihood ratio test (CMNet-LRT) is derived based on the minimum error probability (MEP) criterion. Taking advantage of the outstanding performance of DTL in transferring knowledge with only a few training data, the proposed scheme can adaptively fine-tune the detector for different channel environments to further improve the detection performance. Finally, extensive simulation results demonstrate that the BER performance of the proposed method is comparable to that of the optimal detection method with perfect CSI.