I fear that AI assistants are not likely to be a very reliable protection against these many exploits. First, profitable and widely disseminated agents will have incentives to exploit the same cognitive and social attack surfaces they also partially protect against. Second, echo chambers will move to the agent layer. I may want my agent endorsed by the Catholic Church or branded by the NRA, while you may prefer an agent endorsed by the ACLU or one that claims to be written by a self-proclaimed Anonymous hacker. What my agent (and, thus, I) consider misinformation will be well-documented truths to your agent (and you)— and conversely.
Ever since the webinar last week, I have been taking my time with this article, reading and rereading it carefully, and the more I sit with it, the more two things become unmistakably clear to me.
First, we are both trying to solve the same problem, and second, in many places we are saying the exact same thing, just arriving from different directions and through different vocabularies. That kind of convergence, when it happens independently, tends to mean something.
I reached out to you directly by email a little while ago and look forward to that conversation. But I wanted to engage here as well because this work deserves public dialogue.
The framework you have built is impressive and genuinely rigorous. The five-layer topology, the honest accounting of the structural challenges, the mapping onto established security methodologies, this is the kind of foundational work the field has been missing. That said, I want to be clear that what follows is not a challenge but an invitation to think together.
You make the case compellingly that perception is the new attack surface. I agree completely. My question, and I am stress-testing my own model here as much as yours, is this: if perception is the new attack surface, where is the most effective entry point for intervention? And does the answer to that question change how we approach the ghosts in the machine?
I ask because I think the entry point question is load-bearing for everything else. The intervention layer you choose, upstream or downstream, environmental or individual, before consolidation of beliefs or after, determines whether the structural challenges you have identified are problems to be solved or constraints to be designed around.
I would genuinely love to hear where your thinking is on this, and how you are currently approaching those structural challenges in your ongoing research.
With the utmost respect toward what you're building,
I hesitate to enter this conversation as I am coming from a new-Luddite perspective and have a tremendous resistance to most of the digital 'gifts'. I have been an environmental activist most of my long life and strongly an innately believe that the further one gets from nature the more vulnerable one becomes.
Still much mulling to be done, but I wanted to express my appreciation at your union of cybersecurity, risk management & psychotherapy disciplines as a means to help protect against cognition-directed attacks.
FWIW, kernel panic happens in people too. Polite society calls them nervous breakdowns, but most of these are at root a mind’s response to the corruption that has been experienced.
> At civilizational scale, individual resilience is irrelevant. You can personally resist a world-scale narrative shift all you want, and it will still happen around you. The question of what collective remediation looks like—and who even has the resources or standing to implement it!—is one this field must sit with.
Really interesting conceit and articulation of the framework. In regards to this last point, this has likely been driving the ever-increasing degree of polarization we see!
I particularly enjoyed that you see the weaknesses and limitations in the approach and called them out, because I was wiggling in my seat about them from the first third or so. Not only all the ones you call out, but for a good fraction of people, it's likely to be fundamentally a hardware issue, and the human hardware is famously non-upgradeable and declining in not just relative, but absolute, capabilities with every year.
As to your larger question here, I think the only scalable solution is the AI assistants that are inevitably going to come to live in practically everyone's ears shortly, just as essentially everyone owns and uses smartphones now.
They can curate your media diet, and be the updateable and active hardware immune systems for people's epistemics and memetic diets. In fact, I think they're essentially the *only* feasible solution.
Obviously there will be both free and paid tiers with the assistants, and just as obviously, the vastly-higher-MAU free tiers will be at significant moral hazard in terms of auctioning their audiences off to the attention sphere (basically the "attention economy" economic model, with an additional layer on top), and the ones willing to pay the most will be those with ill intent and bad memetics.
So it behooves people to pay for their AI assistants on this front, and for us to urge our friends and loved ones and social sphere to do so too.
Perhaps we’ll be able to benefit from an EU-sponsored, open source, trust-based & user-centric AI assistant effort … it won’t solve the problem of echo chambers forming, but it could substantially level the playing field as a reasonably secure platform available to all.
Hey. I've been keeping an eye on the comments. My team and I are in the beginning stages of building a fully accessible open-source initiative similar to what you're suggesting here. Except here in the United States. Would love to connect, talk, and collaborate with you.
I fear that AI assistants are not likely to be a very reliable protection against these many exploits. First, profitable and widely disseminated agents will have incentives to exploit the same cognitive and social attack surfaces they also partially protect against. Second, echo chambers will move to the agent layer. I may want my agent endorsed by the Catholic Church or branded by the NRA, while you may prefer an agent endorsed by the ACLU or one that claims to be written by a self-proclaimed Anonymous hacker. What my agent (and, thus, I) consider misinformation will be well-documented truths to your agent (and you)— and conversely.
Hey there Ms. Melton,
Ever since the webinar last week, I have been taking my time with this article, reading and rereading it carefully, and the more I sit with it, the more two things become unmistakably clear to me.
First, we are both trying to solve the same problem, and second, in many places we are saying the exact same thing, just arriving from different directions and through different vocabularies. That kind of convergence, when it happens independently, tends to mean something.
I reached out to you directly by email a little while ago and look forward to that conversation. But I wanted to engage here as well because this work deserves public dialogue.
The framework you have built is impressive and genuinely rigorous. The five-layer topology, the honest accounting of the structural challenges, the mapping onto established security methodologies, this is the kind of foundational work the field has been missing. That said, I want to be clear that what follows is not a challenge but an invitation to think together.
You make the case compellingly that perception is the new attack surface. I agree completely. My question, and I am stress-testing my own model here as much as yours, is this: if perception is the new attack surface, where is the most effective entry point for intervention? And does the answer to that question change how we approach the ghosts in the machine?
I ask because I think the entry point question is load-bearing for everything else. The intervention layer you choose, upstream or downstream, environmental or individual, before consolidation of beliefs or after, determines whether the structural challenges you have identified are problems to be solved or constraints to be designed around.
I would genuinely love to hear where your thinking is on this, and how you are currently approaching those structural challenges in your ongoing research.
With the utmost respect toward what you're building,
Shaq
I hesitate to enter this conversation as I am coming from a new-Luddite perspective and have a tremendous resistance to most of the digital 'gifts'. I have been an environmental activist most of my long life and strongly an innately believe that the further one gets from nature the more vulnerable one becomes.
Steve Friedman, West Kelowna, BC, Canada
Tremendous, thank you!
Still much mulling to be done, but I wanted to express my appreciation at your union of cybersecurity, risk management & psychotherapy disciplines as a means to help protect against cognition-directed attacks.
FWIW, kernel panic happens in people too. Polite society calls them nervous breakdowns, but most of these are at root a mind’s response to the corruption that has been experienced.
> At civilizational scale, individual resilience is irrelevant. You can personally resist a world-scale narrative shift all you want, and it will still happen around you. The question of what collective remediation looks like—and who even has the resources or standing to implement it!—is one this field must sit with.
Really interesting conceit and articulation of the framework. In regards to this last point, this has likely been driving the ever-increasing degree of polarization we see!
I particularly enjoyed that you see the weaknesses and limitations in the approach and called them out, because I was wiggling in my seat about them from the first third or so. Not only all the ones you call out, but for a good fraction of people, it's likely to be fundamentally a hardware issue, and the human hardware is famously non-upgradeable and declining in not just relative, but absolute, capabilities with every year.
As to your larger question here, I think the only scalable solution is the AI assistants that are inevitably going to come to live in practically everyone's ears shortly, just as essentially everyone owns and uses smartphones now.
They can curate your media diet, and be the updateable and active hardware immune systems for people's epistemics and memetic diets. In fact, I think they're essentially the *only* feasible solution.
Obviously there will be both free and paid tiers with the assistants, and just as obviously, the vastly-higher-MAU free tiers will be at significant moral hazard in terms of auctioning their audiences off to the attention sphere (basically the "attention economy" economic model, with an additional layer on top), and the ones willing to pay the most will be those with ill intent and bad memetics.
So it behooves people to pay for their AI assistants on this front, and for us to urge our friends and loved ones and social sphere to do so too.
Perhaps we’ll be able to benefit from an EU-sponsored, open source, trust-based & user-centric AI assistant effort … it won’t solve the problem of echo chambers forming, but it could substantially level the playing field as a reasonably secure platform available to all.
Hey. I've been keeping an eye on the comments. My team and I are in the beginning stages of building a fully accessible open-source initiative similar to what you're suggesting here. Except here in the United States. Would love to connect, talk, and collaborate with you.
Kindly,
Shaq
Please feel free to drop me a DM, Shaqeal. DC