Microstructure quantification is an important step towards establishing structure-property relationships in materials. Machine learning-based image processing methods have been shown to outperform conventional image processing techniques and are increasingly applied to microstructure quantification tasks. In this work, we present a 3D variational autoencoder (VAE) for encoding microstructure volume elements comprising voxelated crystallographic orientation data. The VAE is used to encode a training set of volume elements with an equiaxed polycrystalline microstructure with random texture. We show that the model generalises well to unseen microstructures and, also, some edge cases with textures, grain sizes and aspect ratios outside the training distribution. Structure-property relationships are explored through using the training set of volume elements as initial configurations in various crystal plasticity (CP) simulations. Microstructural fingerprints extracted from the VAE, which parameterise the volume elements in a relatively low-dimensional latent space, are stored alongside the volume-averaged stress tensor response to applied random average deformation gradient loading paths, resulting from CP simulations. This is then used to train a recurrent neural network (RNN), which acts as a surrogate model for the CP simulation. Previous work on RNN-based surrogates for CP have been shown to perform well when correlating between stress and strain, but microstructure dependence is often excluded. Here, we show that suitable fingerprinting has the potential to enable microstructure-dependent CP surrogates. Given a statistically equivalent volume element and load path, the trained RNN can accurately predict the resulting stress response. Once trained, the RNN offers significant speed up over running a CP simulation and can ultimately serve as a tool for upscaling volume element responses to a component scale simulation.