I worked on this problem for more than a decade, as co-founder of what became the Center for Humane Tech, and in other roles.
One way to break it down is different users, but as Ivan notes towards the end, we all have Marl in us. So, another way to get at the same thing is to exclude certain engagements from the metrics — the things we click on but would not reflectively endorse as meaningful. You get a lower engagement number that represents meaningful choice, rather than just "revealed preference" / engagement.
This is what I work to align LLMs with at the Institute for Meaning Alignment[1], and Ivan is helping! I also have a paper[2] on the difference between revealed preference and meaningful choice.
(It's also worth noting that this process of enshittification doesn't just happen in software. Markets and voting also have this revealed preference vs meaningful choice problem. So, making this distinction is a chance to upgrade all of our large-scale systems.)
"(It's also worth noting that this process of enshittification doesn't just happen in software. Markets and voting also have this revealed preference vs meaningful choice problem. So, making this distinction is a chance to upgrade all of our large-scale systems.)"
I agree. The alignment problem is not AI specific. We need to align our economy in general. The reason is simple: Our economy as a whole is a reinforement learning AI that is maximizing an objective function (profit).
And, as Joe says, there is a lot to learn about the economy from AI (and CS more generally).
Yes, Lucas critique etc. But that doesnt mean that we cant use eg taxes to change economic incentives. We only need to be reasonably sure that we steer in the right direction. Then we can adapt the details every year when we pass a budget.
2. Hobbyist builder that do sell out (at a good price) can use the capital to build even more hobbyist tools, instead of becoming a VC or retire on an island somewhere.
Hobbyist builder that fails both 1 & 2 is just another kind of Marl.
Doctorow has studied this phenomenon under the term of "enshittification". His analysis is different though. Instead of look at the psychology of the individual user he takes a more systemic view and looks at the economic incentives. In a nutshell, an online service can either serve the user well or maximize revenue but not both. Would be interesting to take a deeper look at these seemingly opposing but probably complementary analyses and to compare "Enshittification" with "The Tyranny of the Marginal User".
It just occurred to me that Quora could be a good example of enshittification. I used to like it, even spent quite some time on it writing and answering. But last time when I had a question, I was not only linked to a page that didnt have any answer to my question, Quora also tried to make me, quite aggressively in my view, read all kind of sensational ugly low quality stuff on totally unrelated questions ... I dont think I will return there any time soon.
The power users on Quora have been complaining about this for several years. The consensus is that Quora built a great product until they had to monetize, and then they shifted to focusing on engagement and ruined the site.
For example, it used to be against the rules to post memes. Now I get constant spam from memes in my feed, no matter how many spaces I block.
Thanks for this. From what you say, Quora is a clear example of enshittification. You refer to a consensus: Has this been documented? Is there a good reference in case I want to argue this in the future?
interesting ... paying for questions is of course stupid ... in particular in times of AI where you can have bots asking questions. Is it known what Quora does about bots asking questions?
Do you know whether somebody has tried to build a decentralized version of Quora? Of course, the problem of quality control remains difficult also in a decentralized setting. But if there is enough anger about Quora, maybe there could be enough critical mass to try sth new?
There is an official Quora Questions bot that's run by the company, but lots of people complain about the quality of the questions that it produces.
If you mean users writing bots, I don't think they do much to prevent it. That might explain why the site gets overwhelmed with endless variations of questions like, "What screenshot deserves 469317+ views?"
I don't know of anyone working on a decentralized Quora. How exactly would that work? And what problems do you think it would solve?
I think Apple did a pretty good job of developing and evolving their software while serving pretty much everyone, including the "marginal user". Maybe the article is more about social software than software in general?
agreed! notably Apple isn't primarily a software company, they make their money off hardware (fine, the integrated hardware-software package), and their business model allows them to derive (some) value from serving their power users well. They're still far from ideal; e.g. I would pay more for an iPhone that encouraged app providers to serve my intentions rather than competing to hijack my attention, but the market doesn't clear.
Interesting point: maybe Apple is producing quite good software because their incentive is in the hardware, not getting our atttention and personal data :)
Apple is racing to the bottom too, what with the idiocy of a new silent switch etc, the new Settings app in Ventura, inability to fix the clusterfuck that is anything to do with subscriptions and iCloud, etc etc. We're getting there, man.
But yes, I think this is primarily about sites and mobile apps. An interesting question is games. I don't think they are, on the whole, getting stupider... yet?
iCloud and all Apple’s online services were much worse before. I think they are streamlining this over time. But yes it’s still far from perfect. Much accidental complexity left.
Regarding games, that depends. But I have kids and I definitely see a lot of games pushing the instant reward principle way too far, to encourage addictive behavior, which is a disaster.
Yes, I agree. I see a lot of games pushing instant rewards and micro transactions as a sort of gambling or status mechanism that, to my mind, is pretty disastrous to good game development.
Apple is a really interesting case if only because of Apple Music, an app that has simultaneously become more complicated and less useful over time. It exists in some insane netherworld where its target user cares neither about app responsiveness nor about any particular feature, but sure wants _a lot of stuff_. It's like mirror-universe Evil Marl.
Apple Music is still iTunes, a shitty jukebox app that Apple bought 20+ years ago and has been Scotch-taping stuff onto ever since.
The fact that you can't even bookmark shows or playlists from the service is absolutely pathetic. And the UI doesn't even show you WHICH SHOW IS PLAYING. I mean... WTF? It actively cripples your effort to consume content.
But then, so did Apple's removal of the headphone jack from its best-selling music player. And so did the pathetic battery life that resulted from Jony Ive's brainless mania for one "idea:" "thinness."
I actually like Apple Music because it's simple and does what I need it to do: replicate digitally the experience of walking to my record shelves, picking out an album, and putting it on. I don't create or use playlists. I don't want to use the app to "discover" anything new - I use freeform and college radio and word of mouth for that. In a sea of apps that seem to be constantly screaming at me to try features that I have no use for, I find the simplicity of Apple Music refreshing.
Actually, Jony Ive's brainless mania for TWO ideas; thinness and a smooth, opaque IO. College humor did a few sketches with Jony Ive, the smooth voiced British accent, menacing Tim Cook, who wants the product to be marginally useful.
I'm a freelance writer and I hated, loathed, and despised Apple's butterfly keyboards. It was kind of like trying to type on glass, only without the haptic feedback you get when you type on your phone which makes it...not too horrible, really.
I despised that POS every minute I used it. I bashed the shit out of it hard enough to press the internal components of the computer into the bottom of the case, denting it outward. I have to admit, the keyboard continued to "function" despite that, of course in the same offensively defective manner. Did you see the Wall Street Journal's great takedown of the thing, where the writer published the whole article without correcting any of the keyboard-induced errors? You could flip switches to eliminate each kind of error, until the article became legible: https://www.wsj.com/graphics/apple-still-hasnt-fixed-its-macbook-keyboard-problem/
My solution was to take the old Apple aluminum Bluetooth keyboard (the one with the rounded back where the batteries went) and just sit it on my MBP. It straddled the POS keyboard perfectly. I was overjoyed when that computer (an employer-bought computer, fortunately) finally just stopped booting up and I could get the M1. LIFE-CHANGING.
Oh yeah, and let's not even go into the goddamned emoji bar that Apple forced on its "pro" users instead of function keys.
I didn’t say I was “impressed”. Just that they make pretty good software. Apple Notes Pages, Keynote, or the Camera, Wallet and Apple Pay apps on the iPhone, for example.
Fair fair. As a software guy I was just curious to learn which apple software’s specifically are appreciated by people. Product professionals do not really rate apple to high when it comes to software because. Thanks Nicolas, appreciate the response.
I would kill... KILL to resurrect OKC with some developers - only instead of marketing and binary questions, do so with UX Research question formation in practice. Call it OKKupidating or something. You think 2016 was cool? You'd have loved it in 2010.
The questions were shit though, and the user generated ones were awful. But get a few dozen UXRs on it? Mercy - what we could do!
Before OKCupid, I thought Nerve was pretty good. It seemed to attract a funnier, smarter class of people. They had some sentence-completion exercises that were occasionally funny and provocative.
OKCupid had a similar edge over Match, which has always been more middle-of-the-road.
Just (finally) read this, which touches on many of the same things. It's a good read, and overall aligns with Doctorow's observation that an online service can either serve the user well or maximize revenue, but never both.
Personally, I've just been turning off more and more of these services and figuring out that life is pretty good without most of them. In fact, it's better when I'm not spending a good chunk of it trying to squeeze my brain into Marl's preferences. It doesn't solve the larger problems, but at least I'm happier...
"Since [...] economic incentives operate on the margin, a company with a billion-user product doesn’t actually care about its billion existing users. It cares about the marginal user - the billion-plus-first user - and it focuses all its energy on making sure that marginal user doesn’t stop using the app. Yes, if you neglect the existing users’ experience for long enough they will leave, but in practice apps are sticky and by the time your loyal users leave everyone on the team will have long been promoted."
Implicit here is a choice of time unit. Do I care about the marginal user today or the marginal user(s) over the next year? Choosing a larger time unit those who will leave the platform come into focus again.
Thus, afaiu, the bigger issue is that the economic incentives at work here discount the future too much. This problem of discounting the future too much is a much larger problem. For example, it is at the heart of why we do not tackle seriously climate change and biodiversity loss.
The industry has taken us back to mainframe computing of the 60's, this began in 1997 with pc games with ultima online, lineage and everquest, they started removing multiplayer networking code and rebranding pc games in order to steal them. You're 26 years too late buddy.
They are removing plaintext root access to our computing devices, when the iphone was launched in 2007 and android shortly thereafter, apple and google made sure to transfer ownership of the device and software to themselves to jack up software prices.
Yes and no. If Marl used all the buttons and settings, spent time and money on great apps with useful content and demanded self-moderation tools, management and investors would be thrilled to give Marl more complexity and tools because it’s selling well.
"digital nightclub" is so funny yet sadly accurate
You nailed the problem: as long as promotion relies on short-term performance metrics, trying to build anything for the long run becomes career suicide.
I agree with the points made. I would include Substack itself in substandard software. The search feature is subpar -- both sitewide and within a particular substack. Also, wish they would support paying for single articles and, ideally, some kind of bundled subscription.
This has been such a painful death blow to the Internet. It went from a fun place that we could come together and discuss ideas or share funny memes.
Now it's just stale, forced, oversaturated and over socialed. Too many "content creators" are just normal people desperate for attention who never touched the old internet and got the original concept. Internet, to them, is just a place to get money and attention from.
Wow. Well put. The scariest thing is, this translates even to domain-specific apps such as Navionics Boating. I use it every time I go out, because, somehow, they've not yet managed to touch the charts and rendering and it just works, better than any of the competitors. But, the rest of the interface is like a Fisher Price toy. You want to add a waypoint based on a specific lat/long you got out of a pilot book? There is no such thing as "Add waypoint" in the UI, nooo, you enter the lat/long in "Search" and then tap on something or other to add it as a waypoint.
This attitude manifests itself throughout the application's UI, as if, indeed, the application is optimized for "Marl’s tolerance for user interface complexity is zero.".
I worked on this problem for more than a decade, as co-founder of what became the Center for Humane Tech, and in other roles.
One way to break it down is different users, but as Ivan notes towards the end, we all have Marl in us. So, another way to get at the same thing is to exclude certain engagements from the metrics — the things we click on but would not reflectively endorse as meaningful. You get a lower engagement number that represents meaningful choice, rather than just "revealed preference" / engagement.
This is what I work to align LLMs with at the Institute for Meaning Alignment[1], and Ivan is helping! I also have a paper[2] on the difference between revealed preference and meaningful choice.
(It's also worth noting that this process of enshittification doesn't just happen in software. Markets and voting also have this revealed preference vs meaningful choice problem. So, making this distinction is a chance to upgrade all of our large-scale systems.)
[1] https://meaningalignment.org/
[2] https://github.com/jxe/vpm/blob/master/vpm.pdf
"(It's also worth noting that this process of enshittification doesn't just happen in software. Markets and voting also have this revealed preference vs meaningful choice problem. So, making this distinction is a chance to upgrade all of our large-scale systems.)"
I agree. The alignment problem is not AI specific. We need to align our economy in general. The reason is simple: Our economy as a whole is a reinforement learning AI that is maximizing an objective function (profit).
And, as Joe says, there is a lot to learn about the economy from AI (and CS more generally).
Couldnt we change the objective function that is training this superintellgience via reincforcement learning?
Yes, Lucas critique etc. But that doesnt mean that we cant use eg taxes to change economic incentives. We only need to be reasonably sure that we steer in the right direction. Then we can adapt the details every year when we pass a budget.
1. Hobbyist builder doesn't have to sell out.
2. Hobbyist builder that do sell out (at a good price) can use the capital to build even more hobbyist tools, instead of becoming a VC or retire on an island somewhere.
Hobbyist builder that fails both 1 & 2 is just another kind of Marl.
Maybe Mr. Hobbyist Builder signed a contract never to develop a similar tool again?
I think the maker of Apollo (a reddit client) is an example of 1. Would appreciate it if you can give some examples of 2.
Doctorow has studied this phenomenon under the term of "enshittification". His analysis is different though. Instead of look at the psychology of the individual user he takes a more systemic view and looks at the economic incentives. In a nutshell, an online service can either serve the user well or maximize revenue but not both. Would be interesting to take a deeper look at these seemingly opposing but probably complementary analyses and to compare "Enshittification" with "The Tyranny of the Marginal User".
It just occurred to me that Quora could be a good example of enshittification. I used to like it, even spent quite some time on it writing and answering. But last time when I had a question, I was not only linked to a page that didnt have any answer to my question, Quora also tried to make me, quite aggressively in my view, read all kind of sensational ugly low quality stuff on totally unrelated questions ... I dont think I will return there any time soon.
The power users on Quora have been complaining about this for several years. The consensus is that Quora built a great product until they had to monetize, and then they shifted to focusing on engagement and ruined the site.
For example, it used to be against the rules to post memes. Now I get constant spam from memes in my feed, no matter how many spaces I block.
Thanks for this. From what you say, Quora is a clear example of enshittification. You refer to a consensus: Has this been documented? Is there a good reference in case I want to argue this in the future?
Sorry, I don't know of a single place that clearly documents the process. But you could find some of the changes by searching on Quora itself, e.g., some of the answers here: https://www.quora.com/Why-did-Quoras-quality-degrade-so-quickly
interesting ... paying for questions is of course stupid ... in particular in times of AI where you can have bots asking questions. Is it known what Quora does about bots asking questions?
Do you know whether somebody has tried to build a decentralized version of Quora? Of course, the problem of quality control remains difficult also in a decentralized setting. But if there is enough anger about Quora, maybe there could be enough critical mass to try sth new?
There is an official Quora Questions bot that's run by the company, but lots of people complain about the quality of the questions that it produces.
If you mean users writing bots, I don't think they do much to prevent it. That might explain why the site gets overwhelmed with endless variations of questions like, "What screenshot deserves 469317+ views?"
I don't know of anyone working on a decentralized Quora. How exactly would that work? And what problems do you think it would solve?
I think Apple did a pretty good job of developing and evolving their software while serving pretty much everyone, including the "marginal user". Maybe the article is more about social software than software in general?
agreed! notably Apple isn't primarily a software company, they make their money off hardware (fine, the integrated hardware-software package), and their business model allows them to derive (some) value from serving their power users well. They're still far from ideal; e.g. I would pay more for an iPhone that encouraged app providers to serve my intentions rather than competing to hijack my attention, but the market doesn't clear.
Interesting point: maybe Apple is producing quite good software because their incentive is in the hardware, not getting our atttention and personal data :)
Apple is racing to the bottom too, what with the idiocy of a new silent switch etc, the new Settings app in Ventura, inability to fix the clusterfuck that is anything to do with subscriptions and iCloud, etc etc. We're getting there, man.
But yes, I think this is primarily about sites and mobile apps. An interesting question is games. I don't think they are, on the whole, getting stupider... yet?
iCloud and all Apple’s online services were much worse before. I think they are streamlining this over time. But yes it’s still far from perfect. Much accidental complexity left.
Regarding games, that depends. But I have kids and I definitely see a lot of games pushing the instant reward principle way too far, to encourage addictive behavior, which is a disaster.
Yes, I agree. I see a lot of games pushing instant rewards and micro transactions as a sort of gambling or status mechanism that, to my mind, is pretty disastrous to good game development.
Apple is a really interesting case if only because of Apple Music, an app that has simultaneously become more complicated and less useful over time. It exists in some insane netherworld where its target user cares neither about app responsiveness nor about any particular feature, but sure wants _a lot of stuff_. It's like mirror-universe Evil Marl.
Apple Music is still iTunes, a shitty jukebox app that Apple bought 20+ years ago and has been Scotch-taping stuff onto ever since.
The fact that you can't even bookmark shows or playlists from the service is absolutely pathetic. And the UI doesn't even show you WHICH SHOW IS PLAYING. I mean... WTF? It actively cripples your effort to consume content.
But then, so did Apple's removal of the headphone jack from its best-selling music player. And so did the pathetic battery life that resulted from Jony Ive's brainless mania for one "idea:" "thinness."
I actually like Apple Music because it's simple and does what I need it to do: replicate digitally the experience of walking to my record shelves, picking out an album, and putting it on. I don't create or use playlists. I don't want to use the app to "discover" anything new - I use freeform and college radio and word of mouth for that. In a sea of apps that seem to be constantly screaming at me to try features that I have no use for, I find the simplicity of Apple Music refreshing.
Actually, Jony Ive's brainless mania for TWO ideas; thinness and a smooth, opaque IO. College humor did a few sketches with Jony Ive, the smooth voiced British accent, menacing Tim Cook, who wants the product to be marginally useful.
Oh, I'll have to find those.
I'm so glad Ive is gone, since I need to work on Macs, and Windows is an insufferable shitshow now anyway.
I'm a freelance writer and I hated, loathed, and despised Apple's butterfly keyboards. It was kind of like trying to type on glass, only without the haptic feedback you get when you type on your phone which makes it...not too horrible, really.
Here's my favorite one, "The new iPhone is just worse": https://youtu.be/RgBDdDdSqNE?si=tR3Kx5f2jZKaPbke
I despised that POS every minute I used it. I bashed the shit out of it hard enough to press the internal components of the computer into the bottom of the case, denting it outward. I have to admit, the keyboard continued to "function" despite that, of course in the same offensively defective manner. Did you see the Wall Street Journal's great takedown of the thing, where the writer published the whole article without correcting any of the keyboard-induced errors? You could flip switches to eliminate each kind of error, until the article became legible: https://www.wsj.com/graphics/apple-still-hasnt-fixed-its-macbook-keyboard-problem/
My solution was to take the old Apple aluminum Bluetooth keyboard (the one with the rounded back where the batteries went) and just sit it on my MBP. It straddled the POS keyboard perfectly. I was overjoyed when that computer (an employer-bought computer, fortunately) finally just stopped booting up and I could get the M1. LIFE-CHANGING.
Oh yeah, and let's not even go into the goddamned emoji bar that Apple forced on its "pro" users instead of function keys.
Hi Nicolas,
Can you share specifically which software from Apple has impressed you?
Hi,
I didn’t say I was “impressed”. Just that they make pretty good software. Apple Notes Pages, Keynote, or the Camera, Wallet and Apple Pay apps on the iPhone, for example.
Fair fair. As a software guy I was just curious to learn which apple software’s specifically are appreciated by people. Product professionals do not really rate apple to high when it comes to software because. Thanks Nicolas, appreciate the response.
I would kill... KILL to resurrect OKC with some developers - only instead of marketing and binary questions, do so with UX Research question formation in practice. Call it OKKupidating or something. You think 2016 was cool? You'd have loved it in 2010.
The questions were shit though, and the user generated ones were awful. But get a few dozen UXRs on it? Mercy - what we could do!
Before OKCupid, I thought Nerve was pretty good. It seemed to attract a funnier, smarter class of people. They had some sentence-completion exercises that were occasionally funny and provocative.
OKCupid had a similar edge over Match, which has always been more middle-of-the-road.
https://news.ycombinator.com/item?id=37509507
Comments on hackernews
Just (finally) read this, which touches on many of the same things. It's a good read, and overall aligns with Doctorow's observation that an online service can either serve the user well or maximize revenue, but never both.
Personally, I've just been turning off more and more of these services and figuring out that life is pretty good without most of them. In fact, it's better when I'm not spending a good chunk of it trying to squeeze my brain into Marl's preferences. It doesn't solve the larger problems, but at least I'm happier...
https://www.amazon.com/gp/product/B093G9TS91
Bought the book. Thanks for sharing.
"Since [...] economic incentives operate on the margin, a company with a billion-user product doesn’t actually care about its billion existing users. It cares about the marginal user - the billion-plus-first user - and it focuses all its energy on making sure that marginal user doesn’t stop using the app. Yes, if you neglect the existing users’ experience for long enough they will leave, but in practice apps are sticky and by the time your loyal users leave everyone on the team will have long been promoted."
Implicit here is a choice of time unit. Do I care about the marginal user today or the marginal user(s) over the next year? Choosing a larger time unit those who will leave the platform come into focus again.
Thus, afaiu, the bigger issue is that the economic incentives at work here discount the future too much. This problem of discounting the future too much is a much larger problem. For example, it is at the heart of why we do not tackle seriously climate change and biodiversity loss.
The industry has taken us back to mainframe computing of the 60's, this began in 1997 with pc games with ultima online, lineage and everquest, they started removing multiplayer networking code and rebranding pc games in order to steal them. You're 26 years too late buddy.
They are removing plaintext root access to our computing devices, when the iphone was launched in 2007 and android shortly thereafter, apple and google made sure to transfer ownership of the device and software to themselves to jack up software prices.
https://cs.stanford.edu/people/eroberts/cs201/projects/trusted-computing/index.html
But it isn't Marl who is making decisions about the product, you said it yourself: it's management and ultimately stockholders.
Yes and no. If Marl used all the buttons and settings, spent time and money on great apps with useful content and demanded self-moderation tools, management and investors would be thrilled to give Marl more complexity and tools because it’s selling well.
Friggin Moloch I swear...
All my homies hate Marl.
"digital nightclub" is so funny yet sadly accurate
You nailed the problem: as long as promotion relies on short-term performance metrics, trying to build anything for the long run becomes career suicide.
I agree with the points made. I would include Substack itself in substandard software. The search feature is subpar -- both sitewide and within a particular substack. Also, wish they would support paying for single articles and, ideally, some kind of bundled subscription.
This has been such a painful death blow to the Internet. It went from a fun place that we could come together and discuss ideas or share funny memes.
Now it's just stale, forced, oversaturated and over socialed. Too many "content creators" are just normal people desperate for attention who never touched the old internet and got the original concept. Internet, to them, is just a place to get money and attention from.
I miss the older days.
Wow. Well put. The scariest thing is, this translates even to domain-specific apps such as Navionics Boating. I use it every time I go out, because, somehow, they've not yet managed to touch the charts and rendering and it just works, better than any of the competitors. But, the rest of the interface is like a Fisher Price toy. You want to add a waypoint based on a specific lat/long you got out of a pilot book? There is no such thing as "Add waypoint" in the UI, nooo, you enter the lat/long in "Search" and then tap on something or other to add it as a waypoint.
This attitude manifests itself throughout the application's UI, as if, indeed, the application is optimized for "Marl’s tolerance for user interface complexity is zero.".