With the advent of large scale-surveys the manual analysis and classification of individual radio source morphologies is rendered impossible as existing approaches do not scale. The analysis of complex morphological features in the spatial domain is a particularly important task. Here, we discuss the challenges of transferring crowdsourced labels obtained from the Radio Galaxy Zoo project and introduce a proper transfer mechanism via quantile random forest regression. By using parallelized rotation and flipping invariant Kohonen-maps, image cubes of Radio Galaxy Zoo selected galaxies formed from the Faint Images of the Radio Sky at Twenty-cm (FIRST) radio continuum and the Wide-field Infrared Survey Explorer (WISE) infrared all-sky surveys are first projected down to a two-dimensional embedding in an unsupervised way. This embedding can be seen as a discretized space of shapes with the coordinates reflecting morphological features as expressed by the automatically derived prototypes. We find that these prototypes have reconstructed physically meaningful processes across two channel images at radio and infrared wavelengths in an unsupervised manner. In the second step, images are compared with those prototypes to create a heat map, which is the morphological fingerprint of each object and the basis for transferring the user generated labels. These heat maps have reduced the feature space by a factor of 248, and are able to be used as the basis for subsequent machine-learning (ML) methods. Using an ensemble of decision trees we achieve upwards of 85.7% and 80.7% accuracy when predicting the number of components and peaks in an image, respectively, using these heat maps. We also question the currently used discrete classification schema and introduce a continuous scale that better reflects the uncertainty in transition between two classes, caused by sensitivity and resolution limits.