What is...the Role of Automation in War?

00:00:06: Welcome to Voices, the EISA Podcast.

00:00:09: The space for cutting edge research in the discipline of international relations and the audible companion to EISA – the European International Studies Association.

00:00:24: This podcast sets the stage for deeper insights into award-winning papers books and thesis as much as it provides a room for critical engagement with key concepts on political and sociological thought.

00:00:43: Voices, the EISA podcast.

00:00:46: Feeds your reading lists makes cutting-edge IR research audible.

00:01:02: Please welcome our today's host Polly Palister Wilkins political geographer and associate professor at The University of Amsterdam And board member of the EISA.

00:01:17: What is the role of automation in war?

00:01:20: To answer this question and more I am joined this episode by Elkish Fortz to discuss the role of automated technologies, including increasing use artificial intelligence in pursuit military objectives and ethics for outsourcing decision-making.

00:01:41: Her work focuses on the nexus of ethics, technology politics and warfare with a specific emphasis on new and emerging military technologies including military artificial intelligence autonomous weapon systems drones and robots.

00:01:56: She is the author of Death Machines The Ethics Of Violent Technologies published by Manchester University Press And her work on Military AI An Autonomous Weapon Systems has been widely published in a range international publications.

00:02:10: She is Vice-Chair of the International Committee on Robot Arms Control and an Associate of the Imperial War Museum, a Royal Society of Arts Fellow.

00:02:24: Welcome

00:02:25: to The Voice's podcast Elka.

00:02:26: thank you so much for joining us today!

00:02:29: Thank You for

00:02:30: having me very welcome.

00:02:32: So I'm sure our listeners are aware of the growing role of automation in warfare not least because automation's recent use in Israel's genocide in Gaza and reports about the use of automation in the U.S.-Israel war on Iran, but could you set this scene for us as to the current state-of play when it comes?

00:02:56: The

00:03:06: grandfather of cybernetics, Norbert Wiener worked on automation for military purposes in the nineteen forties and fifties.

00:03:14: And further on in the sixties until he realized that his expertise leads to some rather devastating human consequences.

00:03:22: but their desire to automate warfare really stems or has its infancy or point of origin in the nineteen forties and fifties.

00:03:32: So it is really nothing particularly new fast forward to today.

00:03:39: what we're seeing are a lot more autonomous type systems So away from this pre-programmed sequence, if this happened then this should happen and more towards self learning systems.

00:03:52: And that happened of course from the nineteen fifties onwards with a lot of government funding for military projects which supported really budding computational industry transferred into the civilian domain but remained prominent in the military environment.

00:04:08: an even autonomous weapon system go back all the way to the nineteen seventies.

00:04:14: The current iteration of AI warfare and autonomous weapons systems are then not new at all.

00:04:20: They're just a different iteration of what emerged from this particular moment when cybernetics really took hold, And then became accelerated with drone warfare and become much more prominent today When computational processing power expanded and became more ubiquitous you know, as a kind of informatic logic systems if you will.

00:04:45: I think perhaps it's worthwhile to situate autonomy versus automation before we begin so that where we know what we're talking about... So automation really is a set of prescribed actions for example a toaster which pops up your toast at a specific point, which corresponds to the level of toasting that you wish.

00:05:11: That's an automated system.

00:05:12: it does things automatically but based on specific settings as human input.

00:05:20: An autonomous system is a system that seeks and responds So it is more dynamic.

00:05:28: It looks at feedback and takes into consideration Feedback from sensor data in other kind of data, and adapts to that without human intervention.

00:05:38: And that's the shift from automation towards autonomy.

00:05:42: so keep with it.

00:05:43: a toaster analogy perhaps Although we might be stretching limits here A little bit.

00:05:48: a toster an autonomous toaster for example would be one That is let say artificial intelligence enabled and works with other systems to identify perhaps when your alarm is set and rings.

00:06:01: And it would then start toasting the bread, taking into account that time it takes for you to take a shower or calibrate—or taken in to account external temperature —to calibrate precisely the perfect level of

00:06:12: toasting.".

00:06:13: Of course, as humans we still have input on what constitutes the perfect number of toastings.

00:06:21: That has to be programmed!

00:06:22: And of course, you need to still put in the toaster into the toast.

00:06:27: But that would be a much more feedback oriented type system which is not just automatic but becomes increasingly autonomous because it can act and adapt based on external input.

00:06:39: So coming back from the Toaster to the Military Systems, a similar gradient of technological systems in new and emerging defense technologies is in place there as well.

00:06:47: What we're seeing more and more shift towards this full autonomy system that I've mentioned specifically on the application of lethal force.

00:06:56: There's an associated gradient of capabilities at work here which has developed over time and increasingly displaces from the decision loop.

00:07:07: So here we bring in the loop as a figure, if you will so that human IN THE LOOP OF DECISION MAKING becomes the human ON the loop of Decision Making removed ever more and then eventually out-of-the-loop.

00:07:21: that's the typical terminology from semi-autonomous or automated to semi autonomous to fully autonomous.

00:07:29: And so then we talk about this transition or the shift as you look towards development of drones, from drone warfare to fully autonomous type systems.

00:07:41: We consider for example systems like Maven smart systems and AI decision support systems which again is semi-autonomous where human are somewhere in loop on loop moving however toward a full autonomous system.

00:07:57: And all of that relates to the question, what degree does a human have meaningful oversight and control?

00:08:06: That with greater autonomy decreases.

00:08:09: So it's bit of trade-off there.

00:08:12: If we look at AI decision support systems for example like MavenSmart Systems but also Lavender which was used in Gaza with devastating effects.

00:08:22: What were talking about here is not an autonomous weapon as such, rather it is an AI system which suggests possible targets in a particular area for human operators to approve or disapprove and then action.

00:08:38: So here the human are still one technically making the decision to direct lethal force onto target.

00:08:47: those systems gather and assess analyze large troves of data from various sources.

00:08:56: That can be demographic data, social network data, radio data behavioral data communications data and so on And based on that the system will suggest a target that fits a pre stipulated or predetermined category of suspect or suspicious behavior Or a system increasingly might also discover what might constitute suspicious behavior normal behavior abnormal behavior Based uncertain input parameters or the contextualization and analysis of certain data.

00:09:29: And it does so at very high speeds, on a large scale.

00:09:34: The focus in both genocidal actions in Gaza became about scale of targeting, not running out of targets.

00:09:44: About finding or producing targets at a fast rate so that the operators then found themselves in an environment where they have to cognitively keep pace with all these suggestions.

00:09:56: Fifty targets per hour moving towards one hundred targets and so on.

00:10:03: we've seen this.

00:10:04: particularly big boast was that a thousand targets were struck in twenty four hours and eleven thousand targets in three weeks, so on.

00:10:14: It becomes an obsession with large numbers.

00:10:17: warfare enabled through computational systems which then leaves very little time for deliberation or double checking whether the system's information is accurate becomes prioritized and humans become the executors if you will of machine logics, and machine decisions.

00:10:37: And that in turn of course delimits the scope for overriding a computational output or from making a decision not to act on a recommendation that you have to action somehow at fast pace.

00:10:53: In some ways even though it is a semi-autonomous system.

00:10:58: You kind of have a quasi-autonomous weapon systems in which the human becomes this functional element, and a computational environment.

00:11:06: Tasked with approving targets at an ever accelerated rate... ...with less information or predigested information.

00:11:17: So it becomes about the workflow environment And so the types are automated.

00:11:25: The shift towards autonomous type of warfare happens systemically, if you will.

00:11:30: So we look at some promotional material that Palantir puts out.

00:11:35: in a recent industry event the Chief Digital AI Officer of the US Cameron Stanley explained this system and becomes very much about.

00:11:48: As he was saying, all you need to do is left click right-click left-click.

00:11:51: Magically the data becomes a detection and then once you have a detection this system Then becomes one stop shop for A full targeting workflow where your by the system can identify The best asset in the region And then potentially also action that detected target and as he says than close the kill chain like killing people Is the same as raising processing?

00:12:14: Closing an it ticket or something of this sort.

00:12:18: So targets, including human targets or targets with humans in such a workflow process then become very fully digitized ultimately so that they can be optimized for swift action.

00:12:30: Which again... raises a big question of oversight, restraint.

00:12:36: Of human interference with the system or human control over our systems but more importantly process application.

00:12:44: violence becomes depersonalized and dehumanised.

00:12:49: kind workflow fantasy frictionless lethality.

00:12:54: you know it really is A very far abstraction from the reality is what it actually means to close a kill chain, especially when such systems are working within let's say urban environments where data is scarce.

00:13:10: Where things are messy... ...where you know things are not easily foreseeable.

00:13:15: so this abstraction really takes us far away from an imagination or being able to understand the long-term consequences of social political legal implications of using such systems in human contexts, basically out-of-sight.

00:13:31: Out-of mind a dream of smooth workflow efficiency.

00:13:35: so that really is the aim where we're not there yet.

00:13:38: but That Is The Aim and those are the systems that I already used And trialed In Life Conflicts.

00:13:44: So Really?

00:13:45: The Envisioned Future at least the future envisioned by industry from what i'm hearing It takes the human increasingly, nominally because a human will remain in the decision chain.

00:13:56: In some form out of that decision and action chain.

00:13:59: And we're seeing shift towards speed and scale and targeting great autonomy all around whether That's navigating drones warms or in Targeting.

00:14:07: I really focusing on coalescing around the idea of machine speed as the primary value, if you will or parameter around which everything else has to configure.

00:14:19: And that is not just a case.

00:14:20: in the US we are seeing this and UK quite starkly developing at the moment.

00:14:26: there's a prioritization of the digital targeting web of speeding and scaling up targeting without really asking too many questions about what that means for how we envision warfare.

00:14:41: But the bottom line is, so it's no longer just automation towards autonomy and that has achieved through a combination of various systems.

00:14:50: AI-enabled systems – systems with systems always are collection data analytics platforms weapon platforms drones or otherwise And all of this then packaged together in multi domain ecology command & control which I think provides what I think is an illusion of global oversight and control at speed.

00:15:13: Brilliant, thank you!

00:15:15: That's really comprehensive.

00:15:16: so you've sort-of started to touch on my next question a little better with this notion.

00:15:26: If we think about the technology itself, so sort of the specificity of that technology itself.

00:15:35: How does this technology?

00:15:37: This automated autonomous AI technology change the way wars are now being fought with?

00:15:45: some you know can be thinks more what is actually happening on the battlefield where it's a battle field?

00:15:52: how come we reconfigure our understanding

00:15:58: Again, prefaces with the fact that – and it is a fact.

00:16:02: Wars remain about destruction of bodies in things.

00:16:05: Wars will always be about bombing, exploding things or violence ultimately.

00:16:12: That's not going to change regardless what kinds of weapons systems we throw into the mix.

00:16:20: War is political and social affairs so it'll be human.

00:16:25: to impose your will onto another human context, destruction and violence will always remain the perceived tool towards that end.

00:16:37: But what is likely?

00:16:48: likely?

00:16:48: cognitive shift and a different type of compartmentalization for justifying the use of force on other humans.

00:16:56: So really, I'm much greater distancing and dissociation mediated through screens interfaces And to the idea that you can see everything if have enough data.

00:17:09: However, what we're seeing at the moment is that you can have all the data.

00:17:12: You wish to half.

00:17:13: it will never be enough.

00:17:15: He would never really fully understand the human context because some of the Human affairs are not cannot be captured easily as data points.

00:17:23: So do you end up with systems?

00:17:25: That gives you a lot of information yet, you know nothing.

00:17:28: so that's The preface now there isn't.

00:17:31: has a long history of this trajectory as well right.

00:17:33: so automation comes out in the forties fifties and sixties.

00:17:37: But the idea or the desire to want-to know human context through computational systems also has a longer history and a longer trajectory.

00:17:44: So this desire for more precision, knowledge... ...and illusion that can be gained through computational processes certainly became much more foregrounded since Vietnam War where computational logics played a greater role.

00:18:01: And so in many ways what we're seeing today quite starkly bears heritage of the quantification obsession of The Vietnam War policy.

00:18:11: So it really was with the introduction of computational logics and their budding behavioral sciences that this desire to see human affairs like systems, right?

00:18:21: To understand humans as though they're physics or some other types of systems that you can control an interfere into... That became foregrounded during that time.

00:18:33: And there's perhaps something about the desire to mediate the application of violence or manipulation of humans through computational systems.

00:18:44: that comes out, perhaps a hugely traumatic World War II environment but also use nuclear weapons and Hiroshima and Nagasaki.

00:18:58: But there was kind predecessor, if you will of some of those data analytics companies like Palantir for example in the sixties.

00:19:06: The company was called Simulmatics.

00:19:08: I was just reading Jill LePours' If.

00:19:10: Then book about that and their system.

00:19:12: really it's a company Of course.

00:19:15: they sought to gather data information about the population In Vietnam And process this data in order To figure out how do influence?

00:19:22: The outcomes of the war ultimately They wanted to sort of figure out who's on which side in order to win over hearts and minds, or indeed the desire was to calculate when the war would finally be won.

00:19:36: And how it could best be one?

00:19:38: Of course all of that failed spectacularly and misunderstanding of what humans are all about and what it means to speak to humans, gather data from humans.

00:19:52: And we know that the Vietnam War a horrific human cost.

00:19:57: but similar data logic is at work today with more processing power perhaps an even more entrenched understanding or misunderstanding as war as engineering problem.

00:20:12: in many ways change practices of warfare is always about the humans, not technology itself.

00:20:19: Technologies do not make war better or necessarily fundamentally change what war is.

00:20:26: rather they channel possible actions in different ways and a current shift towards autonomy with artificial intelligence has a number of consequences or iterations or manifestations if you will.

00:20:40: So what characterizes the current wars is this prioritization of speed.

00:20:44: The allure, of course with AI-enabled weapon systems, lethal or otherwise... ...is that they can execute tasks much faster including lethal action but also you know, finding targets producing targets and so on.

00:21:00: That's an acceleration of previous capacities in quite significant way.

00:21:04: the second allure is that scale that's unaccounted to technological capability potential for ubiquitous surveillance, the potential for large-scale data crunching and processing.

00:21:20: And then that just packaged together with a potential of applying lethal force at greater scales much faster.

00:21:26: What is also changing really?

00:21:29: The role human comes to play in all this configuration... ...and what they can ultimately know about context within which violence is applied or what they want to know about the context within which violence is applied.

00:21:44: AI, if we think of technology itself, is ultimately designed to marginalize human cognition.

00:21:51: but you can say some might say that it enhances human cognition in some ways and replaces human cognition simply on a kind of speeding up cognitive action associated with decision-making and warfare.

00:22:06: so task of making a decision about the use of lethal force, targeting and destroying.

00:22:15: Which always includes or traditionally has included a responsibility to make it moral judgment right?

00:22:22: To render judgement whether one could morally justify the application.

00:22:29: violence in any context which is THE most fraught decision never should be taken lightly or outsource in any kind of way.

00:22:38: But the role that a human decision-maker, being able to render this judgment and make it take moral responsibility becomes all more marginalised.

00:22:49: The Human Decision Point or Judgment Point becomes in itself technified Or fully technical in its configuration And that changes what it means to action lethal targeting decisions.

00:23:05: We all know that the human in warfare is not particularly benevolent or, you know filled with restrained.

00:23:12: It's a moral endeavor warfare it has horrific enterprise to be engaged and this is no shade of military profession but the fact where human lives are always at stake and human communities will be destroyed.

00:23:32: And civilians would always bear the brunt of this action, but so taking a human out could be perceived as an improvement.

00:23:45: Dispassionate application is perhaps better than the passionate ideologically inflected racially motivated hatred that is produced through humans.

00:23:56: But the human doesn't really go away, but rather... ...the application of force is diffused and without human awareness in the engagement chain The possibility for moral restraint, not making a lethal decision or doing the unobvious, the possibility for not killing ultimately shrinks considerably and that is important.

00:24:23: And additionally what we're seeing?

00:24:25: it's really an expansion of the logic of surveillance you know?

00:24:30: The possibility to survey populations in ever more pervasive ways a shift towards dehumanization humans as data objects.

00:24:43: under this fiction human life can be data-fied, drawing everything into the wake of an instrumental logic.

00:24:50: A shift towards killing as a process routed through the logics of information technology workflow management processes and related to that real diffusion or perhaps a rerouting off the significance of what it means to take responsibility for one's actions in

00:25:08: war.".

00:25:08: And then at final point I want There's a significant new issue in how the New Defence Technology Industry and its associated financial interests might shape which kinds of practices or forefathers are prioritised on account of types of technologies that scale very well as businesses, and that scale fast as business because thats where money can be made.

00:25:34: And this is something I have been trying to keep an eye.

00:25:37: And there's been a concentrated shift really in the last decade or so to foreground artificial intelligence and computational systems, its software as the key to solving the problem of war.

00:25:50: As The way to win wars swiftly.

00:25:52: these are literally things that you know.

00:25:54: new defense technology companies have said.

00:25:58: You know wars will now be one with code, or whether we solve with code.

00:26:03: And that in itself produces some trends that are worth noting.

00:26:07: and the prioritization of course of these data analytics platforms that connect to data from all domains sea air land space cyber and so on ideally globally um To provide a command-and-control platform That then gives the illusion that everything suspicious can be detected.

00:26:24: an action at speed across the globe you know, the U.S has instituted a program called The Joint Old Domain Command and Control, JETC-II Program for which the model was Uber platform.

00:26:39: And so together with that came the prioritization of cheaper drone systems from loitering munitions to first person view drones To so-called attributable Drone Systems thousands of drones That are used in conflict or overwhelm the enemy to produce what is called a hellscape in the lingo of U.S Department of Defense and that focuses attention on two mass, so speed and scale introduced by Silicon Valley ideologies but also the ethos really The ethos of Silicon Valley startups has been brought into the military domain.

00:27:16: So this notion have moved fast and break things has then foregrounded certain capabilities that this industry is really good at.

00:27:24: But the upshot of all of this, it also has for grounded lethality as a main mode of addressing conflict because that's what these systems enable.

00:27:32: Thank you and actually moves into my next question.

00:27:35: quite nicely so.

00:27:38: weaponry military technology always evolved.

00:27:41: we can trace genealogies with different types weapons system across time.

00:27:47: At the same time, you've also been telling us that automation and the move towards autonomy has perhaps a longer history than it's people are aware of in popular culture.

00:27:57: I think this kind of technology is presented to us currently as if there was a huge break or something new.

00:28:09: Does it really represent fundamental change?

00:28:12: Or can we think more about continuity?

00:28:16: It is absolutely a continuity.

00:28:18: I mean, there are some fundamental changes.

00:28:21: when we look at The types of techniques that are used for artificial intelligence, That is different than the early days of Artificial Intelligence in the nineteen fifties and nineteen sixties.

00:28:34: The technology itself has its origins in cybernetic technologies coming out at the fiftys.

00:28:40: First workshop on artificial intelligence was instituted where there's a field of inquiry but also as a technology data processing was sort of inaugurated, it wasn't in nineteen fifty six.

00:28:54: There are different techniques now when we look at the large language models transformer models

00:29:02: that

00:29:03: process data differently.

00:29:04: but the ethos of all this really has a longer history and is continuity even though there are some interesting, if you want to use that word innovations in various technologies.

00:29:18: they come out of the civilian sector and have then transferred back into the military sector.

00:29:23: So it's a bit of loop that is going on because of course the software industry and Silicon Valley comes and was largely supported by Military Research or Military Grants or Military Funding.

00:29:35: That kind provided the backbone for what we see as Silicon Valley.

00:29:40: today The technologies having developed in a civilian space and Silicon Valley are introduced back into military, to the military domain.

00:29:49: So this is an interesting kind of like circular environment but they're not really that new And I'm trying to convey that To people who try get their heads around it because we can also think about how the technologies relate to us as humans and our cognitive ability in our human agency.

00:30:13: Some of the texts coming out at the fifties again, Norbert Wiener has written about this but a number other philosophers as well have really put their finger on the core problem that we are grappling with today And that is the relationship between these different agencies right?

00:30:28: The Human Agency which is very different than a Computational Agency because the systems, AI systems are modeled in some ways on neural pathways and our brains.

00:30:42: We as humans I'm not computers And we don't function like computers.

00:30:47: so there's a tension always in this interaction In that space between The human machine teaming but again they ethos has had A much longer trajectory can even go As recent the twenty-tens in The Drone Warfares, right?

00:31:04: So there's a desire for drones.

00:31:06: For targeted killing comes out of the eighties when terrorism is seen as big problem.

00:31:11: and wouldn't it be great if we could just like you know assassinate individual guys?

00:31:14: that was frowned upon in the eightes And this technology has an advanced drone technology over the nineties into the two thousand made It possible to target individuals In ways that I wasn't possible before and targeting became not a frowned upon practice of assassination, which the international community was certainly not in favor off but rather it became a practice that was instituted with the capability of the drone advanced capabilities.

00:31:46: And more sophisticated technological systems lots are technologies we're seeing even in AI.

00:31:52: decision support system is not new to current iteration also comes very clearly out of the two thousands, two thousand and ten algorithmic warfare practice which arose with together in tandem with a drone warfare and targeted killing.

00:32:10: So their notion that you can identify threats based on certain profiles from behavioral patterns or patterns time in the two thousand and two thousands and ten where we were talking about signature strikes but also personality strikes.

00:32:28: But signature strikes then, well again frowned upon to a certain degree has become the norm right?

00:32:34: That you that... We understand or accept that if you have data You can identify what constitutes The profile of a suspect which is potentially an illusion.

00:32:46: But the technological advancements have gone hand-in-hand with a desire of what would be a great thing to have.

00:32:56: And, with that norms have gradually shifted and it's quite problematic on.

00:33:02: specifically has shifted towards how we see the world as a system or territory.

00:33:07: data is set up objects can be manipulated optimized into for improvement.

00:33:14: ultimately what my first book, my only book to date.

00:33:18: Death Machines is all about an institution of ethos a biopolitical ethos or necropolitical ethas rather um or aspiration to improve one's context through the use of violent technologies so through using technological systems.

00:33:36: but also The fiction always is that with all this data, a wall can finally become precise and controllable.

00:33:44: Which of course... ...is not bearing out at all rather what seems to bear out as an expansion of violence As the technology gives the illusion of control.

00:33:55: But there's something in this configuration Of humans working With being co-constituted by technologies.

00:34:03: And so I think it's important To look beyond The idea that we use technologies as instruments, which is very prevalent in the military context.

00:34:12: Right?

00:34:13: So oh!

00:34:13: We have this wonderful technology... ...we make use of it and we control the technology.

00:34:17: That's really challenging in this contemporary configuration.

00:34:22: A lot of philosophers of technology Have argued that any kind of technology Is always more than a simple tool even a gun or tank Or missile.

00:34:33: It has attributes that foreground certain perspectives and practice while it just lets other things disappear, makes the other options less prominent or visible.

00:34:45: And really goes back to the old saying as simple as it is if you have a hammer everything looks like an ale.

00:34:51: So material objects are never just tools that we use at our leisure or direct with our willful capacities but always, especially today's technologies has the ability to produce new ways of thinking.

00:35:06: Has the ability normalize new practices that hitherto had not been acceptable or you know have been frowned upon.

00:35:13: and there is a certain level interest in power designed into systems with AI system status even more pronounced because humans are embedded much deeper within these systems not just physically but cognitively as well.

00:35:27: so the systems themselves take on a much more pronounced agentic capacity in shaping our ideas, practices perspectives and possibilities.

00:35:36: And so even moreso as we begin to see large language models find their way into defense products The future of all that is having to deal with autonomous AI agents In the defence domain.

00:35:50: Of course design always matters.

00:35:52: a decade ago now when I started speaking on this topic and started researching the subject, i always like to point out our growing dependency of course on smartphones in our computational environment.

00:36:03: And the phantom pain that you feel when your phone is not at hand or the prompts that you...the urges that you have When You See A Red Dot Somewhere Or Message Appear That Produces A Certain Impulse Or Prioritization To Immediately Respond To A Visual Queue.

00:36:22: we must assume that this condition, our human reaction to these kinds of interfaces the relationship that we have with technology.

00:36:31: That holds for military context as well.

00:36:34: so it is important how things are designed which very often isn't interrogated at all.

00:36:38: So technologies shape cognition or direct cognition if you will and from there practices are established they're normalized.

00:36:46: And then a question becomes especially for a technology that is particularly good at speed and scale, efficiency and optimization.

00:36:53: And if this technology then has useful lethal outcomes the question of what happens in these contexts?

00:36:59: What can we expect to become

00:37:00: normalized?".

00:37:02: At the moment really it's shift towards large-scale violence and greater lethality.

00:37:07: It raises also questions whether such wars ever end or they will simply perpetuate.

00:37:13: more war and conflict

00:37:17: already started to introduce us.

00:37:19: Your work has, importantly focused on the ethics of all this.

00:37:24: and so what are the main ethical concerns with automation?

00:37:28: With a move towards autonomy can these be disentangled from technology itself?

00:37:33: or is there something also intrinsic in warfare?

00:37:37: more generally Can you talk about it today.

00:37:44: Yeah I mean This is the core of my work in the last decade, although I've shifted a little bit more into political economy.

00:37:52: But it was motivated by being interested.

00:37:54: and what really the broadest ethical consequences are of this new AI-shaped landscape in defense for the use of semi-autonomous and autonomous kinds of lethal systems?

00:38:08: And that's a big topic so i can only scratch the surface.

00:38:13: but there a fundamental moral questions of what we are doing with introducing these kinds of violence here.

00:38:23: There are ethical issues that arise simply at the technical level, right?

00:38:27: So artificial intelligence is fallible in so many ways it's unreliable and so many way any AI system only ever as good data has been trained on or works with.

00:38:40: AI systems tend to be relatively updated frequently.

00:38:46: Data is not easy to come by for context of conflict which are dynamic, messy you know swiftly changing.

00:38:52: so there's an intrinsic brittleness and vulnerability in the use of artificial intelligence.

00:38:59: To produce greater automation autonomy and weapons systems introducing large language models.

00:39:05: we all know about hallucinations And the fact that these aren't human systems.

00:39:11: they're probabilistic systems.

00:39:16: compute the next possible, you know token or part of a word or sentence.

00:39:23: So there's no understanding.

00:39:26: There is not meaning but there are lot of vulnerabilities.

00:39:29: You can spoof large language model by just introducing two hundred and fifty different poisoned documents.

00:39:36: so this has so much capacity for something going wrong.

00:39:39: simply on account at technical level And in context of warfare comes to pass through this risk, the sheer technological risk is borne by civilians.

00:39:51: That has passed on two civilians.

00:39:53: there's a broader ethical issue on account of that dominant technological logics at.

00:39:59: I have already indicated when any context becomes a system of digitizable objects which can be and should be controlled.

00:40:06: perhaps this marginalizes Of course it complexities of human life um...that do matter for decisions in warfare.

00:40:17: It abstracts the meaning of human life from the application of violence and that's a dissociation, this most significant moral question what it means to take your life.

00:40:30: And it shifts the question if should we kill in first place?

00:40:34: Is is right course of action?

00:40:36: What for?

00:40:37: Well maybe we can kill better or more perhaps the most significant ethical issues, I think on account of human-machine interactions in that it reduces or limits the possibility and meaning of human responsibility.

00:40:56: So moral agency is always delimited when humans operate with cognitive type systems.

00:41:03: so not just physical kind of prosthesis but cognitive more responsibility and the possibility for moral restraint, which is so important in warfare.

00:41:17: if it then must happen.

00:41:19: In order to not allow mass atrocities and mass violence to come to pass I've written a paper with my colleague Neil Renwick And we focus on this human machine constellation.

00:41:31: We take issues of ways through AI enabled systems is often presented as more humane or dispassionate, somehow better.

00:41:42: But really if we look at the history of the application of systematic violence and what it means to install the possibility of systemization of violence then we must recognize that there's a link between systematisation of violence in the systematic application of violence At least, at the very least a possibility for limiting moral restraint.

00:42:13: In some ways we always as humans have defenses unless you're psychopath psychological defences against the emotionless application of violence against other humans.

00:42:24: We understand and can conceptualize what atrocity is What exceeds moral bounds in any kind way.

00:42:36: a system cannot do that, of course.

00:42:38: So even when the human is embedded in these kinds of systems did psychological defenses have to be overcome?

00:42:44: In order to scale up let's say and again here if we look at the history of mass atrocities certain enablers are always at work one as an authorization meaning you kind of haven't the ability to abdicate your responsibility because higher entity or a knowledge provider, knowledge producer to engage in a certain action.

00:43:06: The second is a routinization meaning you are embedded and a chain of actions where your part it's just one part of the workflow management process let's say Actioning your own moral restraint, perhaps.

00:43:24: And the third is always the biggest that it's a dehumanization.

00:43:27: Dehumanization as almost always present when we look at history of mass atrocities and genocide and mass violence.

00:43:36: so what we then have?

00:43:37: this humanisation off-the target on account of an AI system for example because it is an objectification in the most literal sense.

00:43:46: An AI system can only, cannot understand the meaning of what a human is but rather sees that human as an object.

00:43:52: you know data's collected disaggregated re-aggregated and people are assorted into categories that may or may not have anything to do with them as people.

00:44:00: so everything becomes.

00:44:04: Following on from that, there's also a different kind of dehumanization at work and the dehumanisation of those applying to violence.

00:44:10: That are embedded within this systems.

00:44:12: they become functional elements with in this marginalised moral agency that is present an account these various parameters And all other.

00:44:21: slowest moral restraint against violence was autonomous weapon system or AI enabled support systems too This condition as part being embedded in cognitively faster machines.

00:44:35: then the technology becomes the authority, actions become functional routines and that shrinks the horizon for moral intervention.

00:44:43: And an all around dehumanization enables potentially what Contandus called The Monstrous you know?

00:44:51: That mass extermination of other humans.

00:44:56: So really this opens up.

00:44:59: A significant ethical question is whether we're opening up the path towards greater violence, perhaps even in conflicts which are asymmetrical and which are grounded always as a foundation in racialized ideological hostilities with these kinds of systems.

00:45:17: And then finally I think there many more issues but i'm going to end on this because for me it's really important that the greatest ethical challenge The role of ethics in these configurations itself.

00:45:30: Increasingly, ethics becomes routed through the computational algorithmic pathways of utility reasoning that you can calculate right?

00:45:37: It costs benefit analysis.

00:45:39: really it's about utility and checking whether you've just met certain thresholds.

00:45:43: so ethics become a game of variables that you could compute on way and so forth.

00:45:49: And this then really replaces rigorous logical reasoning or substitutes, rigorous logical reasoning for taking responsibility and grappling with deep moral questions.

00:46:04: And really asking yourself should I act?

00:46:05: Or Should I not

00:46:06: act?".

00:46:07: Again this turns the focus on who should be killed based upon what calculus rather than question of whether we should kill at all increasingly in sidelines and ethics as a hindrance have never been asked more frequently then in the last three years.

00:46:22: What use ethicism?

00:46:23: war Right, why should we even have ethics in war when there are so many enemies that don't really adhere to ethical guidelines?

00:46:30: But without it... There's no future.

00:46:33: Without it civilization is just not possible.

00:46:35: you know It's quite practically a necessity for us to think about what ethics means and And To think about What the meaning of human life In a social context actually Is and Why We Have Ethics.

00:46:47: I am actually speechless.

00:46:49: i mean surely we need to Think About Ethics in War because war is fundamentally And first and foremost a question of ethics, no?

00:46:56: I mean it's the first

00:46:58: thing.

00:46:59: Oh my goodness!

00:47:00: Wow all right that's working up our listeners if they were falling asleep now.

00:47:09: you actually saying your work is now sort obviously focusing some more on political economy as well as ethics.

00:47:15: so maybe then this also leads into to this question.

00:47:19: So the rise of automation, autonomous systems AI is not restricted to the military and the realm of warfare.

00:47:27: I think anyone who works in a university knows all about the increasing role that an AI plays in our life for better or worse.

00:47:36: So, do we see crossovers between the use of automation and AI in the battle space?

00:47:42: And its use in other parts of society.

00:47:45: Yeah there's a really interesting ongoing enrollment of large-scale militarization with absolutely everything underway at the moment.

00:47:54: um...and this has to do with a number of things.

00:47:56: one of the aspects that is relevant here what I've already mentioned.

00:48:04: It can be digitized.

00:48:05: You know things about other humans and on account of that you can assess the risks, being able to assess them, intervene or improve your context.

00:48:16: That's not reduced to military environments.

00:48:20: We know most military technologies find their way into civilian space first and foremost as forms of policing.

00:48:28: There is an interesting and very evident crossover or meshing of military types, technologies and policing technologies.

00:48:38: And policing technologies are logics in military logics well underway... ...and we see this perhaps most starkly pronounced in the United States at that moment where you have military contractors like Palantir also provide systems for ICE, you know?

00:48:54: For immigration enforcement purposes.

00:48:59: But Palantir, of course also works with the NHS in the UK and I think it's a treasury now as well.

00:49:06: And so yeah technology providers that we all work with use have multiple government contracts usually right?

00:49:17: That includes absolutely everyone.

00:49:18: So absolutely every one is in business providing their systems to military like Google or Metta.

00:49:26: I'm not sure about Apple actually, but it's Amazon.

00:49:30: It is becoming a very totalitarian configuration almost by especially the big tech companies which hold the largest market cap and are so financially powerful that they have the ability to influence shape directions, not just of military operations but also how we should think about crowd control.

00:50:01: How we should thinking about population control?

00:50:04: I was listening to the founder of Oracle speaking his vision for the future.

00:50:16: Everybody has to wear some sort of camera and the police have to.

00:50:19: where cameras, everything can be captured at all points.

00:50:22: It's this fantasy of surveillance on account that ability to survey absolutely every thing.

00:50:28: The illusion is created so you understand everything therefore control it but manipulated an enormous amount power in there.

00:50:35: also interesting revolving door technology company representatives or C-suite executives entering the military space and then possibly, you know creating a kind of like revolving door back as well.

00:50:55: So bringing some of these insights into civilian space.

00:50:58: so US Army has created new divisions called Detachment Two or One And this division is made up by four big tech executives.

00:51:08: It was spearheaded by the CTO of Palantir Shamsankar And so these four big tech executives have now become army reservists and they were sworn in at the high rank of Lieutenant Colonel.

00:51:22: That gives them a lot of decision power, right?

00:51:24: It's not just... They're not deciding from the bottom!

00:51:26: They are coming into the higher-ranked.

00:51:29: They don't know much about military profession other than perhaps having a really swelling patriotic feeling for us.

00:51:38: I'm not entirely sure.

00:51:40: Having been sworn in into this army context, they don't need that access.

00:51:44: In order to sell contracts or work with the Army.

00:51:47: so a question is what Is this transfer now and?

00:51:51: What happens for the transfer back?

00:51:53: right because it will remain CTO CEOs chief officers of their companies in the civilian space as well.

00:52:03: There's also a question what happens to all of our data, right?

00:52:06: So as we feed the beast.

00:52:07: As we are engaging with Chachi PT and Claude in this and that on the other The question is does our engagement with these systems Help train systems then better than use for military purposes?

00:52:22: And then again could potentially be used in a domestic context To be thrown back at us since some sort of you know disadvantageous way or discriminatory way.

00:52:32: So there's many, many ways in which we are seeing a very comprehensive revolving door.

00:52:37: so things coming from the civilian context into the military context but also always need a kind of differentiation between good and bad.

00:52:52: And these differentiations will happen along ideological lines, and racialized lines almost always.

00:52:58: So you know when I started teaching my course in technology politics on war... ...and i spoke to students about what kind of suspicions they have or whether they'd be a little bit scared at all the data that they're leaving as trail in the internet or the wider web of data repositories.

00:53:17: And they've always said, no I have nothing to hide very unaware that once it's out there your data can be used in all kinds of ways and not in ways that constitute you as you but that can produce you as a category?

00:53:34: You might fall into a category of suspicion.

00:53:37: Yeah i think the naivety is really the worry here.

00:53:42: So finally, it's always difficult to talk about future trends.

00:53:47: But what do you see as some of the emerging and critical questions regarding automation?

00:53:53: And AI in war but also around international relations more broadly?

00:54:00: It is really tricky with the encroaching militarization of absolutely everything.

00:54:04: I think International Relations has serious challenges on its hands.

00:54:11: better understand the pervasiveness of militarization as a theory in itself.

00:54:19: And also, their role of technology and shaping power and concentrating power—and I think IR has maybe missed that little bit although there's been some amazing work on this last decade or so.

00:54:29: but you know our technology had not really featured into how power comes to be concentrated even though if we read origins of totalitarianism For example, the role of technology is really significant.

00:54:41: And it was so interesting to read and go back to this text at that moment... ...and see how the totalitarian fantasy in the thirties or forties was precisely to know everything about everyone… …and see who's connected with whom – in order then to eradicate all bad elements ultimately.

00:55:01: So there's literally a passage where Hannah Arendt explains his fantasy of understanding social networks and being able to wield power onto these social networks once you've made all the connections.

00:55:16: So this is a really important challenge, I think, to think about not just the technologies as we have them right now in who are big AI companies but what's the logic of technology?

00:55:27: And how has it shaped politics that progressed over last century or so?

00:55:36: They have always played a role.

00:55:37: And I think the interplay is really important to identify and grapple with in ways that we just haven't... Günther Anders has written this wonderful, so Günter Anders as a philosopher mid-century philosopher German philosopher.

00:55:54: He's written a number of interesting texts where he tries to get at the roll of technology but also our products relationship with human capabilities and limitations.

00:56:08: And he was writing against the background of World War II, but also in the advent of the atomic bomb... ...and its use!

00:56:20: The absolute horror that had emerged from it.

00:56:24: He

00:56:24: wrote an open letter to the son of Klaus Eichmann And in this letter, he tries to get his head around how it was possible... ...to produce the monstrous.

00:56:39: He calls it das Monsteröse and says what happened in Nazi Germany?

00:56:45: Was of course a political ideology But it wasn't just about politics.

00:56:50: It was about certain cultural roots.

00:56:53: and these cultural routes have to do with technology, in the way we relate to technology.

00:56:57: And they're where you think of ourselves in relation to technology.

00:57:00: So he says what has enabled the monstrous is these roots that become a technologized world?

00:57:07: These roots continue to exist.

00:57:09: They persist.

00:57:10: We haven't done away from them In this text.

00:57:14: He said unless we recognize who we are in relation to our artifacts and technologies, the cultural roots that have been produced through this interplay.

00:57:26: We're very likely to see a revisitation of the types of politics that gave rise to Nazi Germany... ...and mass violence that ensued with it.

00:57:37: because if you don't understand how it is possible for humans to become what he calls Eichmann, right?

00:57:46: Not just active Adolf Eichman but passive people who are able to justify all kinds of things through technological artifacts.

00:57:57: To mediation and a certain relationship that is created you know doomed to just repeat history as it is.

00:58:10: And I found that text, like keep coming back.

00:58:12: so and i find this text so interesting he wrote that in the nineteen sixties i think...and then he wrote it not at a time in nineteen eighty eight.

00:58:19: but really i think we have to pay attention.

00:58:22: it's perhaps tricky it's probably more very self-evident..but there's an enormous amount of concentrated power on their hands with technology elites at the moment.

00:58:29: And these technology elites at the moment have considerable vested interests in this technological perspective and making us into technological agents ourselves.

00:58:38: They have vested interest of course, scaling up military spending on defense tech companies they're invested in which has to come to fruition within the next five or ten years Which is why you always see these like oh twenty thirty, twenty eight right?

00:58:53: So there's certain timelines where investments need a bare fruit And that is terrifying if you consider the general perspectives and ideologies of some of these individuals, and the ideologies that some other companies also present.

00:59:06: It's always almost a type of an apocalyptic perspective in which everything is zero-sum ,in which everything about extinction or survival.

00:59:15: .And with this perspective The build out ever more defensive and military typespending I come back to, you know the thing that we learn early on in our careers as IR scholars and that's a security dilemma right?

00:59:30: i feel like we haven't really learned much in the current context about what this security dilemma is.

00:59:36: The more you build up your military arsenal... ...and the more you threaten to build-up your military arsenals are more likely it is.. ..that you end up in a conflict that you end up using this, and I'm terrified at the moment.

00:59:50: That we are sliding into a willful build-out of military capabilities which will then maybe produce a certain urgency to use the things that we have amassed?

01:00:02: And that's of course sidelines –the horrors–of war!

01:00:07: We see these in Ukraine—I think the horrors of war on the Ukraine is very often sidelined in favor talking about how fantastic drones.

01:00:15: But really, that if you look at the number of amputees.

01:00:18: If you look in a number of devastated communities... ...if you look into the numbers of displaced people….

01:00:23: …if we look to the number sheer human death that has been inflicted and speak to people who had to fight this war You know?

01:00:33: In Ukraine for survival they will say This is awful!

01:00:36: We want it to end And don't want us to be perpetuated.

01:00:40: So I think looking into future I mean, obviously we need to regulate the development and use of autonomous weapons.

01:00:48: And AI systems like now you know?

01:00:51: We have to regulate them!

01:00:53: We can't leave it to industry just sort-of.

01:00:55: do the right thing but... ...we need certainly think a little bit harder about what that means to maintain peace or not having conflict Or solve conflicts without violence promote diplomatic ties find ways to trust rather than mistrust one another, because if we build our future on the logic of an economy that is built on wartime principles and we bet on economic growth through defense spending.

01:01:24: And if you bet on security purely through a buildup with more AI type lethal systems then I think that heads us into potentially quite a violent future.

01:01:36: all while there's These are not really walls that can be won, right?

01:01:42: We're seeing no decisive victory anywhere.

01:01:45: In fact we've seen just a greater mess, greater geopolitical instability.

01:01:51: but the winners of all this?

01:01:53: those who capitalise on the ever-growing defence technology environment or at defense environment at large.

01:01:59: I mean if you look at stock prices even the defence primes it's going through the roof And of course, all the investors are now thinking well maybe I should invest in defense products which was something that was frowned upon only five years ago as well.

01:02:14: So looking through the future we need to think much harder about how to maintain stability without use violence.

01:02:23: I think that's a fantastic place to end this episode.

01:02:26: Thank you so much for joining us, Arka and like... Fantastic discussion!

01:02:30: And a lot of, i think our listeners from myself also uh.. To take away and to think about.

01:02:36: So thank you

01:02:38: Thanks for having me.

01:02:38: it was pleasure.

01:02:56: If you liked it, subscribe now.

01:03:00: Voices.

01:03:01: the EISA podcast feeds your reading lists makes cutting-edge IR research audible.