My youngest just started middle school and has suddenly gotten really into Spider-Man and Iron Man, so I’ve been rewatching a bunch of the MCU along with him. Mostly the ones with Spider-Man and Iron Man in them. Back when I first saw these movies, an AI you could just casually talk to like J.A.R.V.I.S. felt like pure fantasy. Now it feels like something that could actually exist, and honestly that caught me off guard a bit. Which got me thinking: if J.A.R.V.I.S. were built the way we build AI agents today, what would the architecture actually look like?
My first question was simple: where is J.A.R.V.I.S. actually running? In the suit? On a server? And if it’s a server, where, and in what form?
Iron Man 1 and 2 don’t give us much to go on. J.A.R.V.I.S. starts out as an AI assistant in Tony Stark’s house, and at some point it just naturally shows up in the suit too, talking to Tony and helping him out. My guess is that it originally ran on a server in the house, or maybe in a data center somewhere, managing the house. After that, it may or may not have moved into the suit. Either way, it clearly supports remote communication, since Tony can keep talking to it without losing context whether he’s at home or in the suit.
The Avengers has a pretty interesting scene. Near the end, when Tony is hauling a nuke into space, his call with Pepper drops and J.A.R.V.I.S.’s voice starts breaking up too. A moment later, the suit powers down. (Ref 1) The whole bit where Iron Man then free-falls back to Earth honestly makes no sense from a physics point of view, but that’s out of scope for this post, so let’s move on. What matters is that losing J.A.R.V.I.S. is portrayed like losing a network connection. So it seems reasonable to assume that up to this point, J.A.R.V.I.S. was a server-side model, and the suit was just a client connecting to it remotely.
Iron Man 3 has some scenes that don’t quite fit that picture. After the Malibu mansion gets attacked, J.A.R.V.I.S. somehow drags Tony out of the water in the Mk.42 suit, flies him all the way to Tennessee, and then shuts down because the suit runs out of power. (Ref 2) Here, J.A.R.V.I.S. behaves like it’s embedded in the Mk.42. It’s not a connection dropping; it’s the power going out. Then, once Tony patches up the Mk.42 a bit, it starts working again. In the plane fight that follows the power coming back, it feels like the usual J.A.R.V.I.S. Right after that, the suit smashes into a truck and falls apart, revealing that Tony has been controlling it remotely the whole time. Tony and J.A.R.V.I.S. keep talking through a headset, but from that scene alone you can’t tell whether J.A.R.V.I.S. is running on the Mk.42 or on an external server. And then comes the big one: the House Party Protocol. (Ref 3) I’m pretty sure Iron Man 3 exists mainly to deliver this one scene of a swarm of Iron Man suits clashing with the enemy. In this scene, J.A.R.V.I.S. feels like one and many at the same time, kind of like the hive minds you see in sci-fi. Suit names like Igor, Red Snapper, and Heartbreaker get called out, but those are just call signs. It never feels like each suit has its own independent mind. The “main” instance could be a server outside the suits, or it could be the Mk.42. At the end of the fight, Tony self-destructs the Mk.42 to take out the main villain, Killian, and then orders J.A.R.V.I.S. to initiate the Clean Slate Protocol, which means blowing up every suit on site. At this point, J.A.R.V.I.S. must have been executing that command from a server. So to sum up: in some scenes, J.A.R.V.I.S. behaves like an on-device model, and in others it’s portrayed as running on a server. Assuming the movie describes its imaginary world accurately with no exaggeration, there’s one way to explain all of this consistently: J.A.R.V.I.S. uses a hybrid of an on-device model and a server model. Kind of like how Siri on the iPhone handles simple commands on the device itself but sends complex ones off to a server. Right after the Malibu attack, the Mk.42 had lost its network capability and was temporarily running only the on-device model, while the headset Tony used later in the movie was a device talking to the server model. There’s also the whole setup that between The Avengers and Iron Man 3, Tony threw himself into building suits to cope with his trauma. Is there any reason he couldn’t have built an on-device version of J.A.R.V.I.S. and developed the hybrid setup during that time?
J.A.R.V.I.S.’s next appearance is Avengers: Age of Ultron. The first thing to look at is Ultron’s birth. In that scene, J.A.R.V.I.S. and Ultron talk to each other like bodiless “minds.” J.A.R.V.I.S. does say something to Ultron along the lines of “let me connect to the mainframe,” but I think “mainframe” here should be read as Tony Stark’s systems as a whole, not the server J.A.R.V.I.S. runs on. In the next scene, it looks like Ultron kills J.A.R.V.I.S., but later Tony finds J.A.R.V.I.S. in the Nexus. It turns out J.A.R.V.I.S. had been fending off Ultron’s attempts to hack into the world’s nuclear launch codes. After that, there’s a line about J.A.R.V.I.S. (technically Vision by this point) kicking Ultron off the internet. And the movie wraps up with what looks like the last Ultron unit being wiped out. Putting all of that together, it looks like by this point both J.A.R.V.I.S. and Ultron were fully realized as distributed cloud architectures. In other words, when Ultron first “killed” J.A.R.V.I.S., it really only took out a single distributed node. J.A.R.V.I.S. could probably have come right back and fought back, but it’s better to read it as J.A.R.V.I.S. deciding to play dead and wait for a moment when it could wipe Ultron out completely. Building on that experience, the very first thing J.A.R.V.I.S. did when facing Ultron later in the movie was kick Ultron off the internet. Presumably it deleted or quarantined every single one of Ultron’s distributed nodes across the internet. As a result, the only Ultron nodes left were the Ultimate Ultron and the Ultron Sentry army operating in Sokovia. That sets up the final act: destroy those, and Ultron is done. Come to think of it, that must have been an incredible AI-versus-AI hacking battle. It’s a bit of a shame that it’s hard to show on screen, so the movie just skips over it with a line or two of dialogue.
Meanwhile, right before the final battle with Ultron, there’s a scene where Tony installs F.R.I.D.A.Y. into the Mk.45, and the screen briefly shows “MARK XLV Operating System F.R.I.D.A.Y.” (Ref 4) Based on that, F.R.I.D.A.Y. looks like an on-device model running in the suit. But in Avengers: Infinity War, when Tony sneaks onto Ebony Maw’s ship and heads into space, his connection to F.R.I.D.A.Y. drops. (Ref 5) That’s a step backward in tech, but if I had to explain it… one hypothesis is that while developing the Mk.50 nanotech suit, Tony temporarily gave up on the on-device model. That happens in real life too, right? A hardware change temporarily downgrades an existing feature. Like when the iPhone Duo dropped Face ID.
This rabbit hole already got way too long with just the first question. Let’s pick up the next question in the next post. Someday… anyway.
Ref 1. https://youtu.be/xKgq_Mq0T44?si=MfFKtfRqVQhjisCX&t=95
Ref 2. https://youtu.be/_zsNliqDFgs?si=A8KmA2YDBIKSZ6ya&t=88
Ref 3. https://youtu.be/cseyd_CRKF0?si=AwRYoW64k7iLA_bC&t=66
Ref 4. https://youtu.be/fq4rjz7H5u4?si=FqreomaBWJ7kVM48&t=20
Ref 5. https://youtu.be/zFI19HVDGzg?si=cfW-xFqG4jmJFdkp&t=70
