Correcting for Selection Bias

You are conducting a survey where participation is voluntary. Therefore, survey participants may be different from the population of interest. If you had many characteristics of the survey participants and the population, how would you infer population means from the sample? You can run a predictive model (the model doesn't matter for this argument) for survey respondents answers based on their characteristics. Then, using that model, you would predict the answer of each member of the population based on their characteristics. The population mean is the average of those predictions. Since the sample size is limited, you cannot use all the characteristics. So, which characteristics should you use? Those that predict selection into the survey best or those that predict answers best?  For simplicity, suppose that you are to choose one characteristic only. Consider the two cases below

Suppose that gender is the most important determinant of self-selection into your survey e.g. only 20 percent of survey participants are female while half of your population is female. However, after investigating your sample, you realize that survey responses are not correlated with gender when tested at 5% level*. In this case, you should not use gender to infer population means. For simplicity, let's suppose that the sample means by gender are equal. Using gender, your predicted population mean will be exactly equal to your sample mean.

Suppose that 40 percent of survey participants are white while 50 percent of the population is white. So, race is not as important of a self-selection predictor as sex was. However, suppose that the answers of whites are 100 while the answers of non-whites are 0. In this case, the inferred population mean will be equal to 0.5*100=15 while the sample mean is 0.4*100=40. 

 Is there a sensible systematic approach? A simple algorithm might be the following:

1. Rank important self-selection determinants e.g. run a random forest or decision tree model to predict being in the sample and choose the most important features.
2. Find the most important features that move the response. 
3. Use the intersection of 1 and 2 to adjust for self-selection bias.

* Not paying attention to the statistical significance or confidence intervals, the researcher may introduce more bias than the selection bias. Suppose that there is no actual bias caused by self-selection. For example, only 20% of participants are female but the responses are orthogonal to gender. You put a lot of weight on the determinants of selection into survey (gender here), you notice that the point estimates of the sample mean are different for males and females, but you don't pay much to the fact that this difference is not statistically significant. Can you see why the researcher will introduce bias in this case?