The Ethics of Engineered Demand

The power to engineer demand arrives with a duty of care, whether or not the field admits it.

Most of us have been caught in a subscription we could not leave. Signing up took thirty seconds and two clicks, a flow so smooth it felt like a courtesy. Canceling required finding a buried page, then a phone call during business hours, then a conversation with someone trained to keep you, and at every step the path narrowed and the friction rose, until leaving cost more effort than staying for another month, which was the point. None of this was an accident or a failure of design. It was design, careful and deliberate, a mechanism engineered so that the easy direction was the one that served the company and the hard direction was the one that served the customer, and it worked precisely because someone had thought about it with skill. The asymmetry between the thirty-second entrance and the obstacle-course exit is the system itself, doing exactly what it was built to do, rather than a bug in it.

The skill in it is the troubling part. The same talent that could have made leaving as easy as joining was spent making leaving hard, and the people who built it were good at their jobs, which is to say the harm was a product of competence rather than its absence. That is the whole problem in miniature, because competence is exactly what an engineering discipline produces, and competence aimed at the wrong end produces better-engineered harm.

I start here because it is the clearest small example of something the rest of this book has been building toward without quite naming, which is that the power to engineer demand is the power to do harm, and that the field acquires the second power at the exact moment it acquires the first. Everything I have argued points toward go-to-market becoming a discipline that designs systems to act on the attention and behavior of large numbers of people, reliably and at scale, and a discipline with that power inherits a responsibility that the field has so far been able to avoid because it did not yet have the power to make the responsibility real. That avoidance is ending, and the question of what the people who engineer demand owe to the people demand is engineered upon is no longer a question the field can postpone.

The power and the duty arrive together

There is a principle in engineering so basic that it underlies the professional codes of every branch of the field, which is that the power to build carries a duty of care toward the people who depend on what you build. The engineer who designs a bridge owes a duty to everyone who will ever cross it, a duty that does not depend on knowing them or on their consent, and that duty is written into the codes of the profession as an obligation that overrides the engineer’s other interests, including the interests of whoever is paying. The reason the duty exists is the power, the fact that the engineer’s choices determine whether strangers live or die, and the profession long ago accepted that this kind of power cannot be held without the corresponding responsibility, that to claim the one is to accept the other.

The codes are explicit about this in a way worth borrowing. The foundational principle of the major engineering codes holds the safety, health, and welfare of the public paramount, a word chosen deliberately to mean that this duty outranks the others, including the engineer’s duty to the client or employer who is paying. The point of making it paramount is precisely that the pressures run the other way, that the person paying wants what they want and the engineer is tempted to give it to them, and the code exists to say there is a limit the payment cannot buy past. A discipline of demand without such a clause, without something that outranks the wishes of whoever funds the system, has no defense against being aimed at whatever pays, which is most of how the harms get built.

This is the inheritance go-to-market is walking into, and it cannot take the power while declining the responsibility, because the two come from the same source. The whole argument for go-to-market as an engineering discipline is an argument that the field designs systems with real effects on real people at scale, that it builds the apparatus by which markets are created and demand is shaped, and that this apparatus works, reliably and powerfully, on human attention and human choices. A field making that claim about its own power is making, at the same time and whether it intends to or not, a claim about its own responsibility, because power over people at scale is the precise condition under which a duty of care arises. The bridge engineer’s duty comes from the fact that people depend on the bridge. The demand engineer’s duty comes from the fact that the system acts on people, shapes what they want and choose and do, and acts on enough of them that the aggregate effect is large.

The field has been able to avoid this so far for a reason worth naming, which is that it has not been powerful enough for the responsibility to bite. A practice that depended on talented individuals and worked unreliably could cause only so much harm, throttled by the same limits that throttled its successes, and a field that cannot reliably do much of anything cannot reliably do much harm. The move toward engineering changes this, because engineering is the production of reliable power, and reliable power to shape demand is reliable power to shape it in ways that serve the people it acts on or in ways that exploit them. The duty arrives with the reliability, and the field is acquiring the reliability now.

How a designed system harms

It helps to be specific about the ways a designed demand system can harm, because the harms have a structure, and naming the structure is the start of being able to design against it.

The first is manipulation, the design of a mechanism to exploit the predictable ways human judgment goes wrong, rather than to inform it. People have well-documented cognitive biases, systematic gaps between how they decide and how they would decide on reflection, and a demand system can be engineered to exploit those gaps, to manufacture false urgency, to exploit the tendency to stick with defaults, to use social proof that misrepresents, to design choices so that the option serving the company is the one a distracted person takes by reflex. The dark pattern is the visible edge of this, the cancellation flow built to exhaust, the checkbox pre-ticked, the cost revealed only at the last step, and the deeper version is subtler, a whole system shaped around the exploitation of predictable irrationality so that people are moved in ways they would reject if they saw the mechanism clearly. The mark of manipulation is exactly that, that it works better the less the person understands what is being done to them.

The second is extraction, the design of a system against the genuine interest of the person it acts on. A demand system can be built to create relationships that serve the company at the customer’s expense, engineered addiction that keeps people engaged past any benefit to them, hidden costs that extract more than the value delivered, structures that make entry easy and exit hard so that the relationship persists on inertia rather than on continued worth. Extraction differs from a fair exchange in that a fair exchange leaves both parties better off and an extractive system is designed so that the company gains at the customer’s cost, with the design specifically aimed at the gap between what the customer would choose with full understanding and what the system induces them to do.

The third is externalization, harm done to people who are not even party to the exchange. A demand system operates in a shared environment, the common space of attention and information and trust, and it can degrade that commons for everyone in pursuit of its own ends, flooding the information space with noise, eroding the general trust that makes communication possible, consuming a shared reservoir of attention that others also depend on. The flood of machine-generated outreach is the example of the moment. Each company that fills inboxes and feeds and message channels with cheap, automated, plausible-looking messages gains a little for itself and adds to a rising tide of noise that makes every channel slightly less trustworthy and slightly less usable for everyone, including the next company and the people on the receiving end. No single message is the problem, and no single sender bears the cost, which is exactly the structure of an externality, a harm spread thin across a commons while the benefit concentrates with the actor, and the predictable end of it is a shared environment so polluted that the channels stop working for anyone, a destruction of common value that no participant intended and none can halt alone. The harm here falls on third parties and on the commons itself, and it is the hardest of the three to see, because it does not show up in the relationship between the company and any individual customer, only in the slow degradation of the environment that all of them share.

The three harms share a tell, which is that each one works by opening a gap between what the person would choose with full understanding and clear options and what the system induces them to do, and then living inside that gap. Manipulation opens the gap by clouding understanding, extraction by structuring the options against the person’s interest, externalization by pushing the cost onto people who never chose at all. A demand system that closed those gaps, that left people choosing as they would with full understanding, might make less money in the short run, and it would be a fundamentally different kind of thing to have built, and the distance between the two kinds of system is most of the moral content of the field.

The brake is coming off

These harms have always been possible, and the reason they become urgent now is the same reason the whole discipline becomes possible now, which is that the labor constraint that limited execution is lifting, and with it the limit on the scale and speed of harm.

For most of the field’s history, the capacity to manipulate or extract or externalize was bounded by the same labor limit that bounded everything, so a manipulative mechanism could be deployed only as fast and as widely as people could build and run it, which kept its harm to a survivable scale and gave the people subject to it, and the society around them, time to notice and react. Cheap, abundant execution removes that limit on the harmful uses exactly as it removes it on the beneficial ones. A manipulative pattern that once reached thousands can now reach millions, a mechanism of extraction can be deployed and personalized at a scale no team could previously staff, and the degradation of the commons can proceed faster than the commons can repair itself.

There is also a new dimension the tools add, which is personalization at scale. Manipulation used to be broadcast, the same manufactured urgency shown to everyone, which limited its power because a generic exploit only catches the people it happens to fit. Cheap, abundant execution makes it possible to tailor the manipulation to each person, to find the specific exploit that works on this individual and deploy it, which is a sharper and more invasive thing than the broadcast version, and it is newly affordable. The capacity to manipulate at scale and the capacity to manipulate precisely arrive together, and the combination is more dangerous than either one alone. The same lifting of the labor brake that makes engineering judgment more valuable makes the absence of an ethic more dangerous, because the systems that can now be built at scale include the ones that should not be built at all, and nothing in the cheap execution distinguishes the two.

This is what makes the timing matter. The power to engineer demand at scale is arriving before the field has any shared sense of the duty that should govern it, which is the most dangerous possible order, the capability outrunning the ethic. The harms will scale with the capability, and the only thing that could keep pace is a sense of responsibility developed deliberately and early, ahead of the failures rather than in response to them.

Define the duty before the failures

Mature engineering disciplines did not invent their ethics in the abstract, and many of their codes were written after disasters, in the hard light of bridges that fell and machines that killed, which is the expensive way to learn. The opportunity in front of go-to-market is to do it the less expensive way, to define its duty of care while the discipline is still forming, before the worst failures rather than after them, because a field that waits will have its ethics written for it, either by the regulators who arrive after the harms become undeniable or by the slow erosion of trust that eventually makes the whole enterprise harder for everyone in it.

I will not pretend to write the code here, because a real professional ethic is built by a field over time rather than handed down in an essay, though its shape is visible from the harms it would guard against. It would hold honesty in the mechanism as a baseline, the principle that a system should not depend for its function on the person misunderstanding what is being done to them, which alone would rule out most dark patterns. It would treat the genuine interest of the person the system acts on as a design constraint rather than an externality, so that a system designed to work against that interest would be a violation regardless of its effectiveness. And it would take account of the commons, the shared environment of attention and trust, as something the field has a collective duty not to degrade. These three fall well short of a whole ethic, and making them precise is real work, and even so they are enough to mark the direction, already enough to call into question a great deal of what the field currently does without a second thought.

The line that has to be drawn

The hard part, and I want to be honest about its hardness, is that go-to-market exists to influence, and influence is not in itself wrong, so the ethic cannot simply be a prohibition on shaping what people want. The entire function of the field is to create and direct demand, to bring people to products they did not know they wanted, and much of that is benign or positively good, connecting people to things that truly serve them, which means the line falls somewhere inside the territory of influence itself, between the kinds that respect the person and the kinds that subvert them, and drawing that line is truly difficult.

The most useful place I know to look for the line is the distinction between persuasion and manipulation. Persuasion offers a person reasons, engages their judgment, and works through their understanding, so that a persuaded person has been given something to think with and has chosen in the light of it. Manipulation works around a person’s judgment, exploiting the gaps in it, and works better the less the person understands what is happening, so that a manipulated person has been moved by something they would have rejected had they seen it. The question is one of mechanism rather than effect. Both persuasion and manipulation influence the person; what separates them is whether the influence worked through the person’s understanding or around it, whether it would survive the person seeing exactly how it worked. A mechanism that still functions when the person fully understands it is on the persuasion side of the line. A mechanism that depends on the person not understanding it is on the other, and a discipline of demand could do worse than to adopt that single test as a first approximation of its duty, because it captures most of what separates the systems that serve people from the systems that exploit them.

A pair of examples fixes the distinction. A landing page that clearly explains what a product does, shows its price, and lets the visitor decide is persuading, and it works as well or better when the visitor understands exactly what it is doing, because what it is doing is handing them the information to choose. A countdown timer that invents a deadline, manufacturing an urgency that does not exist, is manipulating, and it stops working the instant the visitor realizes the deadline is fake, because its whole power depended on the visitor believing something untrue. The first survives understanding and the second is destroyed by it, and that difference, tested case by case, does most of the work of telling the two apart.

The line is not always sharp, and reasonable people will disagree about where exactly it falls in particular cases, and an honest treatment has to admit that. There are genuine hard cases, influences that engage judgment and exploit weakness at the same time, situations where the person’s reflective interest is unclear even to them, and a real ethic has to work in that grey zone rather than pretending it away. But the existence of hard cases at the boundary does not dissolve the distinction, in the same way that the existence of dusk does not mean there is no difference between day and night, and most of what the field does that should trouble it sits well across the line rather than in the grey zone at all, on the side of mechanisms built to work because the person does not understand them.

There is a reasonable objection to all of this, the one that says adults are responsible for their own choices and that calling influence manipulation is its own kind of condescension, a refusal to credit people with the ability to look after themselves. The objection has real force, and a serious ethic has to engage it rather than wave it away, because the line between protecting people and patronizing them is real. What answers it, I think, is the nature of the gap the harmful systems exploit, which is one the person themselves would disown on reflection. The relevant test is whether the system was built to work precisely because most people predictably will not resist, which is a different matter from respecting their freedom to choose. A system designed around the reliable lapses of judgment is monetizing those lapses, and naming that is a way of taking people’s agency seriously rather than denying it.

The choice the field is making

The deepest reason to take all of this seriously is that the alternative is a world with engineered demand and no ethic to govern it, rather than a world without engineered demand, and that is the world being built right now by default. The systems are being constructed regardless, the power is accumulating regardless, the brake is coming off regardless, and the only open question is whether the people building these systems will accept the responsibility that comes with the power or will leave it to be imposed from outside after the harms have done their work.

A field that accepts the responsibility, that defines its duty of care and holds itself to it, becomes a profession in the full sense, one that can be trusted with the power it holds because it has accepted the obligations that power entails. A field that declines, that takes the engineering’s power while refusing its duty, remains something else, a hustle with better tools, more dangerous than before precisely because it is more capable, and it will eventually have its ethics written for it by people outside the work who do not understand it.

There is a self-interested version of the argument, for anyone unmoved by the rest, which is that a field’s license to operate depends on the trust of the people it acts on and the society around it, and that trust is a finite reserve that manipulation and extraction draw down. A field that spends that trust faster than it replenishes it eventually finds the whole enterprise harder, the audiences warier, the regulators more hostile, the tools more constrained, and the prosperity of every practitioner diminished by the behavior of the worst of them. Even read coldly, as pure strategy, the duty of care is the field protecting the commons it depends on, so the ethical case and the self-interested case point the same direction, and a field that cannot be moved by the first might still be moved by the second. The choice between those two futures is being made now, in the ordinary decisions of the people building these systems, most of whom have never been asked to think of their work as carrying a duty of care at all, because the field has never told them it does. Telling them is part of what it would mean to build the discipline, and it raises the final question of who, exactly, the person at the center of this discipline is, the one who holds the power and inherits the duty, and what it means to be them.

---

References

• On engineering ethics and the duty of care codified in professional codes of conduct, see standard treatments of engineering ethics and the codes of the major professional engineering bodies.
• On manipulation, dark patterns, and the exploitation of cognitive bias in designed systems, see the literature on dark patterns in interface design and the behavioral-economics account of systematic bias.
• On the distinction between persuasion and manipulation, see the philosophical literature on the ethics of influence.