AI & human flourishing
Tools Worthy of People: Our Design Test for AI That Serves Human Agency
Kenroy George · 2026-08-31 · 7 min read
TL;DR: Most AI ethics debates happen at the altitude of civilization, but AI's real effect on people is decided one product feature at a time. At Cari we run every feature through a single test: does this give a person more agency, or less. Here is what that test looks like in practice, why it matters most when governments adopt AI, and why the old optimistic science fiction had the right idea all along.
The loudest questions about AI are the biggest ones. Will it end work. Will it end truth. Will it end us.
Those questions matter, but they have a strange property: almost nobody debating them is in a position to decide them. Meanwhile, the decisions that actually shape how AI touches a human life are being made every day, quietly, in product meetings. Should this feature run automatically or wait to be asked. Should this data be stored, and who can see it. Should the model tell the user what is true or what is pleasant.
Civilizational outcomes are the sum of these small choices. Which means AI ethics, in practice, is product design.
The agency test
We use one question to sort every feature: does this give a person more agency, or less. Not more convenience, not more engagement, not more time in the app. Agency: the capacity to understand your situation and act on your own intentions.
The test is easier to see in pairs.
An AI that writes the email for you, in a voice that is not yours, saying things you would not quite say, takes something from you even as it saves you time. An AI that helps you say what you actually meant, faster and more clearly, gives you something. Same technology, opposite direction.
A feed that decides what you see, tuned to maximize the time you spend looking at it, is optimizing you. An assistant you ask, that answers and then stops, is serving you. The difference is not the model. The difference is who initiates and who benefits from the loop continuing.
A system that scores citizens in secret, where an algorithm somewhere holds a number about you that you cannot see, contest, or carry, is the low point of the genre. A credential that a citizen holds, and presents by consent, when they choose, to whom they choose, is the same category of technology pointed in the opposite direction.
In every pair, the capability is nearly identical. The design intent is not. That gap is where ethics actually lives.
Three commitments we make
The agency test only means something if it costs you features you would otherwise ship. Here are three commitments it has produced at Cari.
The person initiates. Cari is voice-first and tap to talk. You start every conversation. Nothing runs without you, nothing listens until you ask it to, and when the conversation ends, it ends. This closes off a whole family of tempting features: proactive nudges, ambient monitoring, engagement loops that reopen themselves. We think the trade is correct. An assistant that waits to be called is a tool. A tool that activates itself is starting to become something else.
Memory belongs to the person. An assistant that remembers you can genuinely serve you: your context, your projects, your way of speaking. But memory is also the most dangerous thing an AI company can hold, because a profile you cannot see is leverage over you. So the remembering has to be inspectable and it has to be yours. You should be able to look at what the assistant knows, correct it, and delete it. Memory in service of the person is a feature. Memory in service of the platform is surveillance with better branding.
Honesty over comfort. An AI that tells you what you want to hear is not on your side, it is on the side of your continued usage. We built an AI-era readiness test that scores how exposed a person's occupation and skills are to automation, and the scores are real. Some people get numbers they do not enjoy seeing. We ship those numbers anyway, because a flattering assessment is worthless and an honest one is the starting point of a plan. Agency requires accurate information about your own situation. Comfort that comes at the price of accuracy is a subtle form of control.
Why this matters more in government
A company that fails the agency test loses your trust and, eventually, your business. A government that fails it changes what citizenship means.
When the state adopts AI, the stakes of every design choice go up an order of magnitude, because citizens cannot opt out of their government. A private app that scores you in secret is creepy. A public system that scores you in secret decides benefits, permits, and access to your own institutions. At that scale the agency test stops being a product principle and becomes a civil-rights question.
This is why we build government technology around credentials rather than databases. A credential a citizen holds, cryptographically verifiable, presented by consent, beats a central record the citizen cannot see. The verification is just as strong. The difference is custody: the citizen carries the proof, decides when to show it, and can see exactly what it says. The state gets integrity. The person keeps agency. There is no technical reason to choose otherwise, only an institutional habit of preferring control.
Policymakers evaluating AI systems could do worse than to carry the same one-line test into every procurement meeting: does this give the citizen more agency, or less. Most of the hard questions collapse into that one.
Common questions
Is agency just a nicer word for friction? No. Agency-preserving design can be extremely fast: tap, speak, get an answer, done. The test is not whether the AI does a lot for you. It is whether you initiated it, whether you can see what it holds about you, and whether it is optimizing for your goal or for its own continuation.
Does the agency test ever conflict with making a good product? It conflicts with certain metrics, mostly engagement time, which we simply do not treat as a success measure. It does not conflict with usefulness. In our experience it improves it, because a tool people trust gets used for things that matter.
What should I ask a vendor, or a government, deploying AI on my behalf? Three questions. Who initiates each action, the person or the system. Can the person see, correct, and delete what is stored about them. And when the truth is uncomfortable, does the system say it anyway.
There is an older, more optimistic strand of science fiction that got this right. The famous fictional ship computers were extraordinarily capable, and they never replaced their crews. They answered when asked, told the truth, held nothing back about what they knew, and then let the humans decide. The crew stayed the protagonists. The computer made them better at being so.
That is the whole test, dramatized. Tools worthy of people amplify the person in front of them. That is the standard we hold every feature to, and if you want to see how it feels in practice, you can talk to Cari at ai.cari.global.