Friend recommendation from user trajectory is a vital real-world application of location-based social networks (LBSN) services. Previous statistical analysis indicated that social network relationships could explain 10% to 30% of human movement, especially long-distance travel. Therefore, it is necessary to recognize patterns from human mobility to assist the friend recommendation. However, previous works either modelled friendships and check-in records by simple graphs with only one connection between any two nodes or ignored a large amount of vital spatio-temporal information and semantic information in raw LBSN data. To overcome the limitation of the simple graph commonly seen in previous works, we leverage heterogeneous multigraph to model LBSN data and define various semantic connections between nodes. Against this background, we propose a Heterogeneous Multigraph Contrastive Learning (HMGCL) model to capture spatio-temporal characteristics of human trajectories for user node embedding learning. Extensive experiments show that our method outperforms the state-of-the-art approaches in six real-world city datasets.