Yeah, people often say this, but I don't know if I see any realistic difficulty there. Polygamy carries historical connotations of patriarchal abuse (especially, for the American progressive, whose frame of reference includes the history of Islam or LDS) that are inconsistent with progressivism, to which patriarchy is an anachronism. Plus, at least from a legal perspective, the gay rights movement in America has proceeded by emphasizing equality and equal rights to participate in the institution of marriage; legal challenges have proceeded under the equal protection clause. The only distinction between SSM and "traditional" marriage, SSM advocates hold, is based on the sex of the people involved. But a polygamous relationship has another critical difference: a difference in the number of participants. It's an essentially different institution, not the same institution with a sex-based difference (which sort of differences progressives tend not to recognize as legitimate). So conclusions that SSM opens the door to polygamy have always struck me as poorly reasoned. The premise of such arguments is that once you change anything from "traditional" marriage, you can change anything else, but that reasoning is insufficiently attentive to the fact that the people who are in favor of changing the definition of "traditional" marriage have certain values too, and polygamy is just not in line with them.
Now, if the point is simply that once you untether marriage from its religious definition, it could, theoretically, be enlarged to include polygamous relationships, that's a perfectly valid argument. But as a historical prediction, I think it's silly.