The whitemouth croaker is one of the most important demersal resources of southeast and southern Brazil, and it is caught mostly by bottom gillnets. This fishing system has the advantage of a unimodal distribution of size selectivity, where the optimum length corresponds to the mode of the distribution. This study is aimed to calculate the selection factor of gillnets from fishery studies and monitoring programs, and to estimate a generic curve that represents the retention size probability of the most commonly used mesh sizes for catching whitemouth croaker. Data on capture frequencies using 70, 90, 100, 110, 120, and 130 mm mesh were obtained from the states of São Paulo, Santa Catarina and Rio Grande do Sul between 1998 and 2012. The selection factor calculated corresponds to 3.73. Based on the general equation of retention size, we estimated a modal length of 261 mm for the 70 mm mesh, 336 mm for the 90 mm mesh, 373 mm for the 100 mm mesh, 411 mm for the 110 mm mesh, 448 mm for the 120 mm mesh and 485 mm for the 130 mm mesh. The modal lengths estimated in this study were consistent with previous estimations obtained by experimental approach. We discuss the appropriate mesh sizes with respect to the biology of the species, along with the possibility of considering gillnet selectivity to improve fishery catch.