Elon is Right: Twitter Should Open Up the Algorithm

1

The main character on Twitter this week, yet again, is Twitter itself. 

As everybody knows, Elon Musk wants to buy the company, and one of the ideas he’s pushing is to open-source the algorithm that decides how tweets are ranked in all of our timelines. Whether or not the sale goes through (seems unlikely), I actually think this is a great idea. In fact I would go further and argue Twitter should not only open-source their algorithm so we can all see how it works, I think they should create an open marketplace for algorithms where anyone can build their own, and use algorithms created by others.

For example I’d want to try an algorithm that attempts to prioritize nuanced conversations about important topics. Maybe someone else would want algorithms to find mind-expanding threads, savage dunks, or thirst traps of hot new snax. 

Here’s how it could look:

(The background images I used for the algorithms are credit to some cool people playing with DALL·E 2 on Twitter!)

Would this transform society? No. Would it “unlock Twitter’s true potential”? Probably not, tbh.

But I do think it would be a solid step towards making Twitter a better and more fun place for everyone, and I think it could rebuild some marginal trust in Twitter as an institution, by demanding less of it. It would give more control to users, and move Twitter back towards its pirate roots of operating more like an open protocol, while still protecting against the downsides of full decentralization.

I’ve been fascinated with this idea for a couple years now:

Nathan Baschez@nbashaw

It would be cool if twitter created a marketplace where developers could create custom algorithms to sort and filter tweets.

Seems like there are all sorts of interesting possibilities we’re missing out on because experimentation is so constrained.

September 7th 2019, 4:58pm EST

8 Retweets87 Likes

At first glance opening the algorithm to transparency and choice might seem far-fetched, but then again, a few weeks ago the guy who invented Twitter said he is also into the idea! (Close observers may note the connection here to Bluesky, a private entity funded by Twitter to build an open social networking protocol.) So perhaps there is hope?

jack⚡️@jack

The choice of which algorithm to use (or not) should be open to everyone

Elon Musk@elonmusk

Twitter algorithm should be open source

March 24th 2022, 7:04pm EST

2k Retweets22k Likes

Whether you love or hate Elon Musk, and regardless of your position on the ‘free speech’ vs ‘safety from abuse’ debate, I think the idea of a marketplace for algorithms should be taken more seriously. There are two main advantages. First, from a purely commercial and self-interested perspective, opening up their algorithms to transparency and choice will help Twitter scale by strengthening their network effect. Second, from a societal point of view, this would be a meaningful step towards rebuilding trust, which is at an all-time low.

Let’s begin by looking at what’s in it for Twitter.

2

Allowing anyone to develop algorithms would be a good idea for Twitter because it abides by the classic law of technology: “commoditize your complement”.

It sounds fancy but the basic idea is simple: if you’re selling a thing (like, for example, a smartphone), and there are other things that go well with that thing (like apps) then it’s a good idea to have as high a quantity and quality of supply of apps as possible. As the supply of your complement increases, generally so does demand for your product. Therefore, it’s logical for smartphone companies to try to commoditize—i.e. make widely available for as cheap as possible—their complement, which in this case is smartphone apps. 

This phenomena is widely observed across all sectors of the economy: 

• Cheap gas increases demand for driving, car-buying, etc
• Cheap interest rates increase demand for projects that require lending (like home-buying)
• Cheap and vast quantities of data increases demand for machine learning algorithms 
• Cheap, VC-subsidized food deliveries increases demand for local restaurants
• Cheap web browsers increases demand for websites (Netscape aimed to increase sales of their server software by open-sourcing their browser)

In Twitter’s case, I think it helps to think of their ranking algorithm as a “complement” to their real product, which is their network of users and tweets. If they can increase the quality and quantity of options people have for accessing that network, then the network becomes more valuable.

The fascinating thing about the economic concept of a ‘complement’ is that there’s kind of a blurry line between it and a ‘sub-component’. For example, are steering wheels complements of cars? Intuitively you might say “no, it’s just a sub-component of a car.” And it makes sense because it’s hard to imagine a car without a steering wheel. But what about car stereos? Air conditioning units? Rear-facing cameras? If the supply of any of these increases in quality or decreases in cost, it’s likely to increase demand for cars in just the same way as more obvious complements like gasoline would.

In reality, products exist within complex ecosystems of interlocking parts, and there are many types of complements that exist on a spectrum that ranges from ‘intrinsic to the thing, and impossible to unbundle’ to ‘sold and used completely separately’. If we put something core like the steering wheel on one end of that spectrum, perhaps on the other end of the spectrum we’d find non-obvious complements like the Michelin dining guide, famously created by the tire company to increase demand for road trips. Clearly, Twitter’s algorithm is closer to the ‘steering wheel’ end of the gradient than the ‘Michelin guide’ end.

There is a risk here, though. When you start to blur the lines between complements and sub-components, there’s a danger of going too far. You don’t want to commoditize your core! For example, if you sell a car that is differentiated by the driving performance, you might be happy to benefit from a commoditized media unit, but not happy with what happens if you commoditize your drivetrain. 

The key is to protect what you’re really selling—what’s most important to end-users that only you provide—and commoditizing everything else.

So, in Twitter’s case, what is core? Is it the algorithm? I don’t think so. What’s most important for Twitter is that people have to send their tweets to them in order for other people to see them, and that if you want to see tweets from people you know you have to go to Twitter to get them. The highest strategic imperative is to remain the central gate that communication must pass through. This is what gives them the right to serve ads. As long as they don’t commoditize the database of tweets and users, then it makes sense to increase the supply of algorithms that users rely on to get value every day from that database.

3

If Twitter did decide to build a marketplace for algorithms, here’s how I’m imagining it could work:

Right now in Twitter there is a sparkle icon in the top right corner of the app. When you click it, you see the option to switch back and forth between Home, which shows algorithmically-sorted feeds, or Latest, where you get a simple reverse chronological timeline.

This is a natural place to put a button where users can discover more algorithms. For increased visibility, Twitter could also put samples of algorithmic rankings in the Explore tab (would love this to replace the lame topics like “Tech Personalities” that are currently in that tab). They could also give users the option to display which algorithm they spend the most time with in their profile page. 

How would these algorithms work? They would be machine learning models that take in a bunch of tweets and spit out an ordered version of those tweets. I can imagine all sorts of fun experiments people could do here. For example, could you create an algorithm that downranks tweets that seem like shallow outrage, while promoting tweets that seem nuanced and thoughtful? What would an algorithm look like that is maximally aggressive in filtering out spam and bots? How about one that attempts to target harassment and abuse? For example, what if tools like Block Party could integrate more deeply into their users’ Twitter experience? These algorithms needn’t be limited to the main timeline, they could also sort and filter our notifications tabs for us, too.

For sure only a subset of power-users would want to manually find or switch between algorithms, so part of the goal here could be to discover new techniques that could be integrated into Twitter’s main, default algorithm. It could function almost like a sort of Netflix Prize, where they open sourced lots of their data and offered a $1m bounty for anyone who could create the best algorithm for recommending movies to users. Netflix was able to use the prize to get an algorithm that was 10% better than their own—and that was in 2009, with relatively primitive ML techniques, and a relatively sparse data-set.

But I think perhaps at a deeper level there may not be “one true best” algorithm for Twitter. Different people clearly want different things at different moments of their lives and in different moments of their day. Right now there’s only one way of curating this type of experience: following people and hoping for the best. I bet algorithms could do a better job. You can almost think of them like supercharged lists that can be personalized for each user.

People are already hacking together this kind of functionality on their own. When the Ukraine war started, I followed a few lists curated by thinkers I trust, but wished they had more flexibility—I found the same accounts who tweeted the most kept crowding out the rest, and I’m sure I missed a lot of interesting tweets that didn’t make it onto the list timeline. More recently, I created an alt account focused on F1 racing and Dodgers baseball—the recommendations are much better there. Outside Twitter, Blake Robbins from Benchmark has done the same on YouTube so he can get better algorithmic recommendations for educational content.

All of Twitter’s best features have come about this way: making the thing people are already doing easier. It’s where hashtags and retweets came from, and I think there’s a similar opportunity here—not just for Twitter, but really any algorithmic content platform.

4

So let’s say Twitter implemented this, and some developers started writing their own algorithms, and some of those algorithms were a hit with users. Perhaps they figured out a fair way to compensate developers based on usage to create an incentive to develop these algorithms. What would happen next?

Each person’s Twitter experience would be more individualized and suited to the kind of thing they want. More types of people would be able to use Twitter more, and have a less frustrating experience. It would become a more flexible, more open platform for public conversation. Great!

So, what could stand in the way? I think there are probably three main risks:

First, it might be hard to get developers to build algorithms. It might be hard for Twitter to pay developers enough per user to make it worth it. Would people develop algorithms for other reasons? Perhaps. Obviously lots of groups have an incentive to shape what narratives people are exposed to, but I don’t think many users would opt-in to overtly propagandistic algorithms developed by organizations pushing a narrative. 

Then again, maybe they would? Your primitive superstition is my great religion, as they say.

Second, even if there were a pretty good supply of algorithms created by developers, maybe nobody would use them. Like, I wonder what percentage of Twitter users even know what an algorithm is? I genuinely have no idea. And even if people do know what it is, or could be educated, then how much motivation would they have to try one? I do think it’s possible to work “selections from this featured algorithm” into people’s Discover tab, or maybe even the Home tab, and I think people would intuitively get it, but what do I know? Worst comes to worst, Twitter could throw it into Twitter Blue, their subscription product for power users, or maybe TweetDeck, but that would seem like a bit of a failure to me.

The third risk I see is more on a societal level. Some might make the argument that this would only increase polarization by sorting people into ever more distant filter bubbles. Right now we all already construct our own filter bubbles based on who we follow, this could take it one step deeper by integrating the filter bubble into the algorithm itself.

But I don’t think this is a real risk. We’re already there, and if anything, too much proximity is making the mutual contempt worse, not better. Sometimes when you’re stuck in a bad relationship dynamic and you can only see the worst in the other person, it’s best to take a break and get some space, rather than be in each other’s faces fighting all the time. A lot of days it feels like that’s exactly what’s happening on Twitter. Maybe all that fighting is hurting more than it’s helping.

5

This brings me to my final point: the effect I think this change could have on societal trust.

I do think Elon is right to note that trust is at an all-time low. As he put it, “Civilizational risk is decreased the more we can increase the trust in Twitter as a public platform.”  Algorithmic transparency and choice wouldn’t be a silver bullet, but I think it would make things marginally better. And I can’t think of much else that would have any substantial impact. It’s just a hard problem.

For better or worse, a large percentage of the population is suspicious of Twitter. Personally I am not in that camp, but I can relate to what it might feel like. Some people would say you can’t reason with these people, and think caring about their concerns or that giving an inch means supporting everything they advocate for, but I don’t think that’s a healthy perspective. The reality is, their trust is lost. I don’t think it helps anything to throw our hands up and say nothing can be done, or to tell them they should suck it up and leave if they don’t like it. We should be engaging in good faith, even if it doesn’t always feel like they are.

Paradoxically, the most powerful way to rebuild trust is to stop asking for it, and start giving people transparency and control instead.