Demystifying JAM, Polkadot’s game-changer

The following, adapted and abridged from my original post here, is a ground-up explanation of Polkadot 1.0, Polkadot 2.0, and the eventual evolution into JAM. It is targeted towards a technical audience, and particularly those who are not very familiar with Polkadot, but have a good high-level understanding of blockchain-based systems, and are possibly familiar with one other ecosystem at a technical level. I believe reading this article can be a helpful "prelude" before reading the JAM Gray Paper.

Background Knowledge

This article makes use and assumes familiarity of the following concepts:

• A blockchain as a state-transition function
• An understanding of what is "state"

(Both explained here)

• Economic Security and Proof of Stake

(Explained in this Polkadot Blockchain Academy lecture, and its recording)

Prelude: Polkadot 1.0

First, a recap of what I consider the top novel features of Polkadot 1.0

Social

• Polkadot is one massive DAO, with fully onchain, self-enacting governance and forkless runtime upgrades.
• DOT is considered software, not a security, by the Web3 Foundation.
• The majority of network development is carried out by the Polkadot Fellowship, not a fiscal company (e.g. Parity).

Technological

• Shared security, sharded execution.
• Use of a WASM-based meta-protocol. Storing the code of the blockchain in the state as byte-code. This enables most upgrades to be forkless, and supercharges Polkadot to be not only sharded, but heterogeneously sharded.

Let's dive further into sharded execution and what we mean by it.

Sharded Execution: All About Cores

Polkadot is often described as an L0 (Layer 0), hosting other sovereign chains that can be considered L1. For the sake of this article, however, we adjust this definition to describe Polkadot as an L1 network hosting other L2 blockchains, much like Ethereum. Therefore, the words L2 and parachain can be used interchangeably.

The core problem with blockchain scalability can be stated as follows: there exists a set of validators, whose execution of some code can be trusted through proof-of-state crypto economics. By default, these validators are expected to re-execute the entirety of each other's work. Therefore, the system as a whole is not scalable so long as we force all validators to perform so many executions.

Increasing the number of validators in this model doesn't really increase the system's throughput, so long as the absolute re-execution principle is in place.

This describes a monolithic (as opposed to sharded) blockchain. Inputs (i.e. blocks) are processed by all network validators, one by one. In such a system, if the L1 wants to host further L2s, all validators have to now re-execute the work of all L2s as well.

Obviously, this will not scale. Optimistic rollups are one way to circumvent this issue, in that re-execution (aka. fraud-proofs) only happen if someone claims a fraud to have happened.

One naive solution to sharding is to merely split the validator set into smaller subsets, and have this smaller subset re-execute L2 blocks. The disadvantage here is that sharding execution diminishes the economic security of the network. The security of the L2 is weaker than that of the L1, and reduces further and further as we carve up the validator set into more shards.

Unlike optimistic rollups that cannot afford re-execution at all times, Polkadot was designed with execution sharding in mind, and therefore can have a subset of its validators re-execute L2 blocks, whilst providing sufficient crypto-economical evidence to all network participants that the veracity of the L2 block is as secure as if the entire set of validators had re-executed it. This is possible through the novel ELVES mechanism.

In short, ELVES is a "cynical rollup" mechanism. Through a few rounds of validators proactively asking other validators if an L2 block is valid, we reach an extremely high probability that the L2 block is valid. Indeed, in the case of any disputes, the entire validator set is asked to participate. This is explained in detail in an article by Rob Habermeier, Polkadot co-founder.

ELVES is why Polkadot can have two properties previously assumed to be mutually exclusive: "Sharded Execution" with "Shared Security". This is the main technological outcome of Polkadot 1.0 when it comes to scalability.

Now, let's consider the "Core" analogy. An execution-sharded blockchain is very much like a CPU: in much the same way that a CPU can have many cores that execute instructions in parallel, Polkadot can progress L2 blocks in parallel. This is why an L2 on Polkadot is called a Parachain, and the environment in which the smaller subgroup of validators re-executes a single L2 block is called a "core*. Each core can be abstracted as "a group of validators working in coordination".

You can imagine a monolithic blockchain as one that ingests a single block at any given time slot, while Polkadot ingests one Relay Chain block, and one parachain block per core, per time slot.

Heterogenous Architecture

So far, we only talked about scalability, and that Polkadot provides sharded execution. It is important to understand that each of Polkadot's shards is an entirely different application. This is achieved with a bytecode-stored meta-protocol, in which the definition of the blockchain is stored as bytecode in state of the blockchain itself. In Polkadot 1.0, WASM was used as the bytecode of choice, and in JAM, PVM/RISC-V is being adopted.

Polkadot is a heterogeneous, sharded blockchain because each of the L2s is an entirely different application.

Polkadot 2.0

A big part of Polkadot 2.0 is about making cores more useful to the projects building in the ecosystem. In the original Polkadot model, a core could be rented for between six months and two years at a time. This is suitable for resourceful businesses, but less so for small teams. The feature that enables Polkadot cores to be used in a more flexible way is called "Agile Coretime". In this model, Polkadot cores can be rented for as little as one block at a time, up to a month at a time, with price cap guarantees for those who want to rent long-term.

Inside Core vs. On the Chain

To understand JAM, it is first useful to look into what happens in a Polkadot core when an L2 block comes in. You will recall that a core is constituted primarily from a group of validators. So, when we say data is "sent to the core", it is gossiped to this group of validators, as so:

1. An L2 Block + a subset of that L2's state is sent to the core. This is all the data that is needed to execute the L2 block.

2. A subset of validators, those that constitute the core, re-execute the L2 block and proceed with further consensus-related tasks.

3. The core validators make the data needed for re-execution available to other validators (outside the core). Further validators might decide to re-execute this L2 block based on ELVES rules, and they need the data to do so.

Note that all the operation so far is outside Polkadot's main block and state transition function. Everything so far happens inside a core, and the data availability layer.

Eventually, a small commitment of the L2's latest state becomes visible on the main Polkadot Relay Chain state. This operation, much cheaper than the actual re-execution of the L2 block, (unlike everything else so far), affects the main Polkadot state, becomes visible in a Polkadot block, and is executed by all Polkadot validators.

What operations can Polkadot now perform?

From 1, we can learn that there exists a new type of execution in Polkadot that differs from normal blockchain state transition function. This typically happens when all validators of a network execute a task, and as the outcome the main blockchain state is updated. We call this an on-chain operation. This is what happens in step 3. Yet, what happens inside a core (step 1) differs from this. We call this novel type of blockchain computation in-core execution.

Next, from point 2 we can deduce that Polkadot already provides a native data-availability (henceforth DA) layer, and L2s automatically use it to keep their execution evidence available for some period of time. But the blob that can be posted to this DA layer is fixed, and it is always the evidence needed to re-execute an L2 block. Moreover, parachains' code is never read from the DA layer.

Understanding the above is the foundation of understanding the majority of what JAM does. To recap:

• in-core execution: What happens inside a core. Abundance, scalable, as secure as on-chain execution through crypto-economics and ELVES.
• on-chain execution: What happens in all validators. Secured by default through economically secured validators. More expensive and constrained, as everyone is executing everything.
• Data Availability: The ability for Polkadot validators to commit to having some data available for some period of time, and providing it to other validators.

JAM

JAM is a new protocol, heavily inspired by Polkadot, and fully compatible with it, aiming to replace the Polkadot Relay Chain and make the usage of cores radically unopinionated.

JAM builds on top of Polkadot 2.0, in that it tries to make Polkadot cores more accessible, but in ways radically more flexible and unopinionated than Agile Coretime.

• Polkadot 2.0 makes the deployment of L2s more flexible on cores.
• JAM is all about deploying any application on Polkadot cores, even those that don't resemble a blockchain/L2.

This is primarily achieved through exposing all the main three primitives discussed in the previous section to programmers, namely: on-chain, in-core, and the DA layer.

With JAM, developers can allow any work done in-core and on-chain to be fully programmable. And they can also allow arbitrary data to be read from and written into the Polkadot DA layer.

This is a foundational description of what JAM aims to be. Needless to say, a lot has been simplified here, and the protocol might still evolve.

With this foundation in mind, we can now dive further into a few details of JAM in the coming sections.

Service and Work Items

In the context of JAM, what was once called an L2/parachain is now called a service, and what used to be called block/transaction is now called work-item or work-package. Concretely, work-items belong to a service, and work-package is a group of work-items. Both terms are deliberately chosen to be generic enough to encapsulate use-cases beyond a blockchain/L2.

A service is described by three entry points, two of which are fn refine() and fn accumulate. The former describes what the service does in-core, and the latter describes what the service does on-chain. The third entry point is on_message, which is called when a message from another service arrives.

Finally, the name of the two entry points is why the protocol is called JAM: the Join Accumulate Machine. Join is fn refine(), when all Polkadot cores do a lot of work, all in parallel, for different services. Join is when the data is distilled into a smaller subset, and is then passed onto the next stage. Accumulate is when the result of all the aforementioned are accumulated into the main JAM state. This is the on-chain execution part.

Work-items can specify exactly what code they execute in-core, on-chain, and if/how/what they read and write to/from the distributed data lake.

Semi Coherence

Recall from existing material around XCM, Polkadot's language of choice for parachain communication, that all such communication is asynchronous. That is, a message is sent, and its reply cannot be waited upon.

Asynchrony is the manifestation of an incoherent system, and is a major drawback of systems that are permanently sharded, such as Polkadot 1.0 and Polkadot 2.0 and the existing L2 landscape in Ethereum.

Yet, as described in the Gray Paper section 2.4, a fully coherent system that is always synchronous for all of its tenants can also only grow so much without compromising generality, accessibility or resilience.

Synchronous ~ Coherent || Asynchronous ~ Incoherent

This is another area where JAM stands out: through the introduction of multiple properties, JAM achieves a novel middle-ground: a semi-coherent system, one in which sub-systems that communicate often have a chance at creating a coherent environment with one another, whilst not enforcing the entire system to be coherent. Dr. Gavin Wood, the author of the Gray Paper, has given several interviews about JAM which shed further light on this aspect.

Another helpful analogy may be to see Polkadot in the JAM era as a sharded system where the boundaries of those shards are fluid, and determined on the fly.

The properties that enable this are:

1. Access to a stateless, parallel in-core execution where different services can only interact synchronously with other services that reside on the same core in that particular block, and on-chain execution where a service has access to the outcome of all services across all cores.

2. Freedom over the scheduling of services. Services that talk to each other frequently can create an economic incentive for their sequencers to create WorkPackages that contain WorkItems of services that often communicate. This enables them to reside in the same core, and in practice, talk to each other as if they were in a synchronous environment.

3. Access to the DA layer, usable as an ephemeral, yet extremely cheap data layer. Once data is placed in the DA, it is eventually propagated to all cores, but is guaranteed to be available in that same core immediately. Therefore, JAM services, through scheduling themselves into the same core in consecutive blocks, can enjoy a much higher degree of access to data.

It is important to note that while the above is possible in JAM, it is not enforced at the protocol layer. Therefore, it is expected that certain interfaces are asynchronous in theory, but can act synchronously in practice through elegant abstractions and incentives. One such example is CorePlay, discussed in the next section.

CorePlay

An experimental idea in the context of JAM, CorePlay can be described as a new model for programming smart contracts. CorePlay remains an idea without specific detail. In any case, to understand CorePlay, we need to first introduce JAM's virtual machine of choice, the Polkadot Virtual Machine. This is also known as PolkaVM or plain PVM.

PolkaVM

An important detail of JAM and CorePlay is PolkaVM. Low-level details of PolkaVM are beyond the scope of this article and are best described in the Gray Paper by the domain experts. Yet, for the sake of this article, we only have to elaborate on two of its properties:

• Efficient metering
• The ability to pause and resume execution

CorePlay is one example of using JAM's flexible primitives to create a synchronous and scalable smart contract environment with a very flexible programming interface. CorePlay suggests deployment of actor-based smart contracts directly on JAM cores, enabling them to enjoy a synchronous programming interface whereby they can be coded as a normal fn main(), in which they can communicate using let _result = other_coreplay_actor(data).await?

In the case other_coreplay_actor is in the same core in that JAM block, this call is synchronous, and in case in another core, the actor is paused and will be resumed in a later JAM block. This is possible because of JAM services and their flexible scheduling, along with PolkaVM's properties.

Core Chains Service

Finally, let's wrap up by reminding all readers of the main reason why we mentioned JAM is fully compatible with Polkadot. The main product of Polkadot is parachains in the Agile Coretime fashion, and this product persists in JAM.

The first service that will be deployed in JAM is likely to be one that is called something along the lines of "core chains" or parachains. This is the service that will allow existing Polkadot 2.0-style parachains to be executed on JAM.

Further services can be deployed on JAM, and the existing core chains service can communicate with them, but the existing product offering of Polkadot will remain strong, and only new doors will be opened to existing parachain teams.