Defining relevance is a topic in its own right. You typically collect either explicit human judgments or implicit judgements from behavior, and then you use those judgments to train a relevance model. But the former approach generally assumes that relevance is an objective function of the query and the document, and the latter approach relies on the validity of the implicit judgments. As information scientist William Goffman said, the relationship between the document and the query is necessary but not sufficient to determine relevance. Still, you have to start somewhere.