Direct Link to File. 2269 words, 12 minute read, 8 paperback pages

…presumably someone else has already written about this and given it a name, I just don’t know what that is. nor (apparently) do I know what terms to search on. –S.H.

Abstract

We hypothesize the existence of a (perhaps unintentional and/or unnoticed) feedback loop for online platforms who are capable of predicting and influencing people’s behaviors, in which there is an incentive or “lower energy state” whereby platforms influence people toward what we might term “model conformity,” i.e. to become more like the sorts of people that the platform’s algorithms can predict well – “changing the data to better fit the model” in a sense. In the case of systems that use classifications (rather than continuously-varying outputs), the drive toward model conformity seems likely to take the form of influencing people to become more strongly members of a class, with fewer people “in the middle” or near threshold-boundaries between classes. We term this process “better-binning” [for lack of a better term right now], in the sense that in involves making the “bins” in which people are placed into more descriptive features for use in models predicting the people’s behavior. For binary classifications, this would imply the development of increased polarization. This incentive for “better binning” of people could be at work in addition to more well-documented polarizing outcomes of the attention economy such as “echo chambers” and amplification of extreme content; further, better-binning may contribute to radicalization but this is not a necessary outcome. This essay describes the hypothesis of the “model conformity” feedback loop and the case of “better binning” in particular, and explores some of the implications.

(…if I ever get around to writing the full essay. This is currently more of a diary-essay; not sure I want to do the work to turn this into a ‘real’ essay. It may depend on how much has already been written about this idea, which is unlikely to be original!)

Intro / Context

The ability of social media platforms to modify people’s opinions and behaviors over time, at scale, has been described in numerous treatments [cite? e.g. “The Great Hack”?] – which, by the way, others have responded that these claims (e.g. of influencing elections) are largely overblown manifestations of techno-determinist thinking, but let’s leave such objections aside for a moment. Most recently this combination of modeling and influencing was depicted popularly in the Netflix documentary “The Social Dilemma,”, which is largely about the unintended consequences of the advertising monetization model of social platforms and the “attention economy” this business model gives rise to. The movie shows a dramatization of algorithms assigning probability scores for how likely different types of content are to generate engagement in the viewer and keep the viewer’s attention.

(Note that they may not predict how likely the viewer is to actually buy the product being advertised, but I digress…. No I’ll digress more. Facebook could demonstrate value to its customers, the advertisers, based on not just showing people ads, but in getting people to buy the advertisers’ products. …they probably do that already. But the movie doesn’t discuss such a metric, only the metric of engagement. But I digress ;-) )

The Incentive Toward a Feedback Loop

As (I think it was) Shoshana Zuboff says in the movie, “the strength of these companies is the certainty of their predictions.” Now, if you are someone who is firmly within a particular kind of group, then one can imagine that the company’s ability to model and predict your behavior is probably stronger than if you were on the boundary between two different groups.

So, why wouldn’t the company(‘s algorithms) use their “nudge” influence to try to push you toward one side or another? That way, future predictions about you can be more certain.

Whether you’re pushed in the direction of, say, “conservative” or “progressive” isn’t as important for the present discussion as whether you’re pushed toward being more conservative or more progressive, because these establish more certain class membership, which in turn makes the class/category a better descriptor of you.

And given the mathematical structure of how this works,

(show a sigmoid function), …

TODO: say more about this, or remove the graph! LOL

this means radicalization, and increasing polarization. End of story. It’s in the company’s interest to have you less on the fence, because then it’s easier to predict with certainty, which then allows them to demonstrate to their customers (the advertisers) how good their models are.

In this sense, the kind of polarization we see in our society may not necessarily be “just” an effect of the attention economy or “echo chambers,” it may also be , via what Cathy O’Neil would call a “feedback loop,” an effect of the model self-optimizing to make it “better” and more valuable to customers (advertisers) and to the company itself – actually, not so much optimizing the model itself but using methods of influence to optimize the users.

Note that I am not talking about merely trying to nudge those on the fence toward your particular candidate or product. I am talking about how the system works better when categories are populated more uniformly, i.e. when everyone is easy to classify, and that by using influence and “nudging,” online platforms can cause that to happen. This is not about nudging toward a particular action, rather it is about trying to make the boundaries more well-defined (with fewer “gray” areas), i.e. more discontinuous.

The idea of nudging toward “extremism” is related but not necessarily required. The example of the binary political spectrum powerful and highly relevant though. It kind of depends on how the classification is happening. See my writing on “Kinds of Kinds and Methods of Classification” for more on this.

(…I can say more and edit this if there’s not already a serious body of work on the topic. I’m not sure what search terms to use to find out!)

“Yea, but what would this actually look like in practice? Providing a more concrete example of how this mechanism might play out would make this a stronger essay.” True. I’ll try to think about that.

Objections / Misunderstandings / What-This-Is-Not

  1. A possible misunderstanding is that by “better binning” we might mean to adjust the bins so they better fit the people we have. No, that is “standard” machine learning, just as it is “standard” to make the model conform to the people being modeled. We’re not talking about making the bins better fit the people as they are, we’re talking about a feedback loop of making the people better fit the bins as they are. (to be redundant:) By “better binning” we mean adjusting the people so that they better fit the bins we have – so that the bins are better descriptors of them, i.e. that people within each bin are more homogeneous than they were previously, for the purposes of making the classification data about them more useful for subsequent predictions.

  2. This is not about deliberately trying to influence “swing” voters or to “get people off the fence” to buy your product or vote for your candidate. Rather it is that having people on any fence is “bad for business” because they are not as easy to predict as those who fit more strongly within a class. And thus it is harder to reliably influence them, and thus it is harder to justify charging your customers (the advertisers) more money based on the (supposed) power of your model. To make this personal: the fact that I do not identify with either Republicans or Democrats is bad for business, because I may like an ad about saving the environment, but may dislike an ad about promoting gun control. “Ah we’ll just make categories for these preferences too.” But the point is, whatever the categories are, it’s better for business if they clearly apply – or clearly don’t apply – to me, because the act of categorization eliminates any continuous “gray” area that might more closely match “where I’m really at”. And regarding continuity, see the next point.

  3. “To ‘solve’ the polarizing influence, we could just create an extra ‘middle’ category.” Maybe you could. Such a remark is not a real objection to the idea of the existence of the better-binning feedback loop, rather it seems to be intended intended as a means to try to mitigate polarizing effects of better-binning. Better-binning is not a claim about what designers could (or should) do. The hypothesis of the better-binning feedback loop is about the existence of an incentive for “modeler-influencers” (i.e., those who not only predict but are also capable of also influencing behavior) to try to make people better fit into “molds” for which the models yield strong predictions. Adding a middle category to a binary classification would replace a boundary in the middle with two boundaries on either side, and better-binning would imply moving people either more toward the center, or the left or the right: you would have three poles instead of two. Perhaps three poles – or four, or more – would be better than two for the sake of society. This amounts to imposing a more finely-grained classification scale, changing its granularity. The limit of such a scale is a continuous function which could still fall prey to the overall incentive of making people more like what the model’s good at: making people “fit on the line” better, removing “noise”. Even though the term “better binning” is about classification, the incentive is the same.

    Thus my use of “model conformity” as a general term, and “better binning” being a special case of model conformity for classification systems. Other terms may be more appropriate or catchy. I do find the case of classifications to be most profound, and merits the use of its own kind of term; “better-binning” may be lame, and perhaps some other term could be used like “class conformity” (which some may find more descriptive and a better parallel with “model conformity”), “convergence” or “forcing conformity” or “elimination of gray areas and outliers”, “strengthening classifications”, these are all possible. Back to the granularity of the classification scheme: we might assume that the granularity that gets used is either the granularity that is required by the reality (binary choices, go-or-no-go), by the customer (only wants certain bins), or produces the strongest predictions. Thus changing the granularity of the classification scheme is not necessarily a factor: it may not be possible, or there may be a strong dis-incentive to do so.

  4. A key requirement for the mechanism (besides that the platform’s algorithms can influence behavior as well as predict it, which is the essence of the feedback loop) is that the models are valued on the basis of the accuracy of their predictions – but aren’t all models, by construction? See the point made by Zuboff mentioned above. This mechanism could be viewed as a special case of “if the data doesn’t fit the model, then the data is wrong,” where now we say “if the data doesn’t fit the model, then change the subjects (of the model) so that they produce data that fits the model.”

  5. As I said earlier, but perhaps worth repeating: This is not about the company trying to influence you toward behaviors that align with their goals or serve their economic interests, such as a health insurance company sending you ads to try to get you to stop smoking or eat healthier. That’s interesting, but what I’ve been talking about would be more about the company trying to make you more like whatever (similar) sorts of people there are for which the company’s models have tended to generate highly accurate predictions.

  6. Finally, this is really not about the particular definition one uses for the word”nudging,” that some people get very animated about. For some, a “nudge” is only imperceptible and only directed toward positive goals of human flourishing. In this essay/writing, I’m simply talking about any means of influencing behavior, whether one wants to use the word “nudge” for it or not.

Support / Evidence?

The idea of a feedback loop driving “model conformity” and the case of “better binning” in particular may seem plausible, yet is there any evidence that it actually occurs? I have absolutely no idea. Further, I have no idea how to even begin to investigate this (and/or whether I should take the time to, given my other commitments right now).

Are there even any anecdotal quotes from either creators of social platforms (such as those in the documentary) indicating an awareness of the incentive toward driving model conformity in users, or of actual users describing such an influence? I have no idea. …So far nobody I’ve talked to has heard others talk about this.

It seems likely that there would be also confounding factors making it difficult to distinguish the action of a “model conformity” dynamic vs. other dynamics. How would one distinguish this? I have no idea.

I can imagine an “expert friend” shrugging and saying, “Yea, this sort of dynamic is interesting to consider and it’s possible, but it would be really hard to show…”

References?

…yea if I want to turn this into a real essay I’ll do references. But/and this recent paper that I haven’t read yet [1] may be relevant. (Featured in Evgeny Mozarov’s The Syllabus.) Or maybe this new paper about the benefits of ambivalence [2].