The next releases of Cardano and the Ouroboros protocol contain changes that guide us towards decentralization and the Shelley era. This Deep Dive post explains how we are approaching this phase. With the release of the Praos algorithm for Shelley, which comes with the staking process, stake pools can be set up so ada owners can delegate their stake. The networking team is focused now on two features that will enable us to run a fully decentralized system. Let me first briefly describe how the networking is designed and engineered and give an overview of where we are at the moment. This post will start at the top of our abstractions and go down the stack. Hopefully, this will be an interesting journey through our design.

Typed protocols

At the very top of the stack is IOHK’s typed-protocols framework, which allows us to design application-level protocols. The top-level goal of protocol design for Ouroboros is to distribute chains of blocks and transactions among participants in the network and that is achieved by three mini-protocols:

• chain-sync is used to efficiently sync a chain of headers;
• block-fetch allows us to pull blocks;
• tx-submission is used to submit transactions.

All three mini-protocols were carefully designed after considering the threats that can arise running a decentralized system. This is very important, because cyber attacks are very common, especially against targets that present strong incentives. There is a range of possible attacks at this level that we need to be able to defend against and one type that we were very careful about is resource-consumption attacks. To defend against such attacks, the protocols allow the consumer side to stay in control of how much data it will receive, and ultimately keep use of its resources (eg, memory, CPU, and open file descriptors) below a certain level.

If you are interested in more details about typed-protocols, we gave talks and ran workshops at Haskell events last year and these were very well received by the engineering community. In particular, see the talk by Duncan Coutts talk at Haskell eXchange and the workshop I ran at Monadic Party.

Role of the multiplexer

TCP/IP protocols form the most ubiquitous protocol suite deployed on the internet. They are also some of the most studied protocols and are available on almost every operating system and computer architecture, so are a good first choice for our purposes. TCP/IP gives us access to a two-way communication channel between servers on the internet. The only high-level requirement of typed-protocols is an ordered delivery of network packets, which is guaranteed by the TCP protocol.

Operating systems limit the number of connections at any one time. For example, Linux, by default, can open 1,024 connections per process, but on macOS the limit is just 256. To avoid excessive use of resources we use a multiplexer. This allows us to combine communication channels into a single one, so we can run all three of our mini-protocols on a single TCP connection. Another way to save resources is to use the bi-directionality of TCP: this means that one can send and receive messages at both ends simultaneously. We haven't used that feature in the Byron Reboot era, but we do want to take advantage of it in the decentralized Shelley era.

The peer-to-peer-governor

We want to use bi-directional connections, running all three mini-protocols in both directions, so we need to have a component that is aware which connections are currently running. When a node connects to a new peer, we can first check if it already has an open connection with that peer, which would be the case if the peer had connected to it already. But this is only one part of connection management that we will need.

Another requirement comes from the peer-to-peer governor. This part of the system is responsible for finding peers, and choosing some of them to connect to. Making a connection takes some time, depending on factors such as the quality of the network connection and the physical distance. Ouroboros is a real-time system, so it is good to hide some latency here. It wouldn't be good if the system was under pressure and yet still needed to connect to new peers; it's much better if the system maintains a handful of spare connections that are ready to take on any new task. A node should be able to make an educated decision about which existing connections to promote to get the best performance. For this reason we decided to have three type of peer:

• cold peers know about their existence, but there is no established network connection.
• warm peers have a connection, but it is only used for network measurements and none of the node-to-node mini-protocols is used;
• hot peers have a connection, which is being used by all three node-to-node mini-protocols.

A node can potentially know about thousands of cold peers, maintain up to hundreds of warm peers, and have tens of hot peers (20 seems a reasonable figure at the moment). There are interesting and challenging questions around the design of policies that will drive decisions for the peer-to-peer governor. Choice of such policies will affect network topology and alter the performance characteristics of the network, including performance under load or malicious action. This will shape the timely distribution of block diffusion (parameterized by block sizes), or transactions. Since running such a system has many unknowns, we'd like to phase it into two parts. For the first phase, which will be released in a few weeks (probably shortly after Praos, also known as the Shelley release), we want to be ready with all the peer-to-peer components but still running in a federated mode. In addition, we will deliver the connection manager together with implementing a server accepting connections, and its integration with the peer-to-peer governor. In this phase, the peer-to-peer governor will be used as a subscription mechanism. Running various private and public testnets, together with our extensive testing should give us enough confidence before releasing this to mainnet.

In the second phase, we will extend the mini-protocols with a gossip protocol. This will allow exchange of information about peers, finalize network quality measures, and plug them into the block-fetch logic (which decides from whom to download a block) as well as the peer-to-peer governor. At this stage, we would like to design and run some experiments to discover how peer-to-peer policies shape the network, and check how they recover from any topologies that are suboptimal (or adversarial).

I hope this gives you a good sense of where we are with the design and implementation of decentralization for Cardano, and our roadmap towards the Shelley era. You can follow further progress in our weekly reports.

This is the third of the Developer Deep Dive technical posts from our software engineering teams.

Day One of the Cardano Virtual Summit 2020: Shelley Edition started with a splash. IOHK CEO, Charles Hoskinson, took the stage to welcome almost 10k registered attendees from around the world to the digital conference space. Each participant was met with exhibition areas, lots of virtual hangout space, and most importantly, session programming across five digital stages, covering everything from governance and community, to science and enterprise. With so much content to enjoy, we can’t hope to summarize it all here. So take a look for yourself. Register now to enjoy all of today’s sessions either live or on-demand right after the summit closes this evening.

To bring you up to speed, here are the key take-outs from Thursday.

Shelley is here

The summit comes right after a significant milestone in the Shelley rollout. The pace of delivery has been ramping up considerably over the last couple of months, leading to the June 30th deployment of the first Shelley-complete node. The stage is now set for the continued rollout, with a high degree of confidence in the dates we have laid out, with delegation and staking due on mainnet in August. However, the confirmation of Shelley’s successful deployment was just the first of many announcements at the summit.

Goguen and Voltaire are coming

Within 120 days, we’ll be rolling out the native asset standard for Cardano. Within 150 days, we’ll see the roll-out of Plutus Foundations, the first point of entry for smart contracts on Cardano. Sustaining a decentralized project means more than simply allowing users to run the protocol and build on the platform. They must also decide what the platform does and where it goes. Project Catalyst and Voltaire seek to give that power to the community through decentralized democracy. A pool of funds and a voting protocol will allow the Cardano community to make their voices heard on proposed advancements to the blockchain. Ada holders will also be able to submit improvement proposals to help shape the direction of the ecosystem. Improvement proposals might include software updates, technical development options, funding decisions, and choices about the long-term plans of the ecosystem. Once delivered, the Cardano network will become a self-governed and user-run platform.

Atala PRISM

Day One continued on a successful note with the announcement of PRISM, a decentralized identity solution that enables people to own their personal data and interact with organizations seamlessly, privately, and securely. It will encourage better practice in consumer data privacy and security by offering users ‘self-sovereign’ digital identities, without Big Tech intermediaries accessing, storing or sharing personal data.

PRISM also promises to open up access to a blockchain marketplace of financial and social services for millions of users who might not previously have had access to banking and financial services. This will enable low-income populations to store and share personal information like credentials, land deeds, and health records. Through PRISM, users can also access financial products like loans and insurance.

Atala PRISM is built for compatibility with other blockchains and legacy systems to make it accessible worldwide. A pilot currently underway in Georgia will give employers the ability to verify graduate qualifications instantly. But PRISM is only one part of a suite of enterprise-grade ‘layer 2’ software solutions designed to bring advanced, highly flexible functionality to blockchains. Other elements of the evolving Atala suite of products include traceability for supply chains and the ability to monitor the movement of goods globally.

In an impressive demo, the team showcased the platform and mobile app with a smart city walkthrough. The app will work with Android and iOS devices, while the platform also supports paper wallets and smart cards.

cFund

Last but not least, we announced a landmark partnership with Wave Financial Group. The $10 million dollars pledged to the development of Cardano’s ecosystem represents the first venture capital to support IOHK. Together, the two companies will generate a $20 million dollar incubator fund to help startups who want to build on the blockchain. This provides several years of financial runway to target seed and early-stage opportunities. Wave Financial is looking to invest in global companies created on Cardano by giving funding of up to $500k. The move aligns both firms’ interests in driving Wave to become the industry’s leading digital asset management group while ensuring IOHK becomes the vanguard technology provider in the space.

We’ll revisit all these stories over the weeks ahead to give you a deeper dive into our plans and what they mean. But, as excited as we are by these opportunities for future adoption and growth, Day One was also a day of reflection.

Because we would not be where we are without the incredible support of the Cardano community, which helped to bring us here and joined from all around the world to celebrate Shelley. Connections were made between companies, platforms... and people. Day One was full of big announcements, and boosted by some fascinating speaker sessions with world-class scientists, technologists, and thought leaders sharing their vision of the future. It was also full of optimism, of ideas, buzzing with community contribution and opportunity. Thanks to everyone who helped make it such a special day. And see you back at the summit later today!

With yesterday’s release of the first Shelley-complete node on the Cardano mainnet, a new era has begun. Between this, the launch today of a brand new Cardano website and brand identity, and tomorrow’s Cardano Virtual Summit, the dawn of Shelley is casting fresh light on IOHK's future. It has also highlighted the need for a documentation framework fit for our rapidly growing ecosystem.

We have written and gathered a substantial amount of technical material on Cardano over the years. Inevitably, this has developed organically and has been of variable quality and utility. Not all of it remains accurate or useful. The deployment of Shelley, the growth in developer interest, and the evolution of Cardano has moved us from project into product. So this has created the need to take a fresh look at how we explain Cardano.

Users and consumers of Cardano need to have clear, concise, and relevant material that matches the quality of the code on which the blockchain is built. Documentation needs to be correct and useful, and well organised and focused. In addition, we have identified the need to map categories of information to each user group, or persona. We're building a best-in-class blockchain, and this necessitates high-quality supporting documentation. The old Cardano docs site was in serious need of re-organization and updating, and our technical writers have been busy doing just that. Welcome to the new Cardano docs.

Documentation for the future

Cardano is a complex product with complex technological underpinnings. However, for the developer community and wider audience to embrace it, we need them to understand how it can provide rewarding experiences. They need to know what Cardano can do and how it can solve problems. To ensure that we have the right information for all users, we are reorganising the documentation suite and mapping what each audience needs.

Over the past few months, IOHK has been engaged in a company-wide effort to revamp all the content about the Cardano project. Our technical writing team has been immersed in transforming our previous repository, working closely with our developer teams to create a single source of technical truth, A resource of effective, informative, and up-to-date information that provides value to a broad set of audiences: exchange partners, stake pools, experienced blockchain developers, as well as people who want to understand Cardano. That also caters for skilled crypto enthusiasts and those who simply want to learn about Cardano from a technical perspective.

So today, we’re launching a new documentation site. Instead of a static repository, the site is now a living, breathing source of information on Cardano, plugged into the heart of our development methodology. Just like Cardano, this is an ever-evolving entity. Components are added regularly as we strive to improve on the content we are delivering. New functionality is implemented all the time. We also intend to publish a lot more documentation based on our plans and community feedback. We need to build this out based on what works for our users, whether they sit in the technology or commercial space.

With the virtual summit and our continuing momentum bringing a host of new faces into the ecosystem, we now have a jumping-off point for a single source of developer truth that we can build on.

Shelley has arrived and IOHK is gearing up to celebrate with the Cardano Virtual Summit 2020: Shelley Edition on July 2nd and 3rd. Every presentation, panel and guest speaker at this online event, has been chosen to represent the many faces of Cardano. From world-class foundational research to the latest advances in cryptographic development; from the passion and dedication of the wider community to some of the innovators who have created our world and are lighting the path to the future. The event is meant to honor the hard work and dedication of the Cardano community, developers and contributors, and the wider ecosystem.

The virtual summit, much like IOHK’s 2019 Miami summit will include presentations from IOHK team members, special guest appearances by thought leaders and a keynote speech by IOHK CEO, Charles Hoskinson. We were proud to announce that our guest of honor is internet co-creator and lead internet evangelist for Google, Vint Cerf. As one of the lead architects of the internet Vint is uniquely positioned to give us insight into building world changing technology.

We’ll also be joined by Caitlin Long of the Wyoming Blockchain Select Committee. The Wall Street professional turned crypto pioneer has been blazing the trail for adoption in the state of Wyoming. She will give us a look at the challenges and opportunities she has faced while pushing for wider adoption of decentralization. Our final special guest, Stephen Wolfram is the creator of the computational knowledge platform Wolfram Alpha and the CEO of Wolfram Research. Cardano shares his dedication to advancing technology through open source research and a commitment to academic excellence. Between each of these important discussions our colleagues at IOHK will take center stage.

Thought leadership

IOHK’s chief scientist, Aggelos Kiayias, and director of African operations John O’Connor will talk about research and outreach in the Cardano Blockchain. Our panel entitled ‘Haskell, then, now, and the future’ will examine the impact the functional programming language has had and where it is heading. The virtual summit also serves as the launchpad for new advancements like ‘Prism’ our decentralized identity solution. More guest speakers will be announced over the week ahead and we’ll also have a number of special announcements over the 2-day summit itself; we’re keeping those under our hats until then.

The two day agenda includes five digital stages with programs dedicated to the ideology of blockchain technology; the science of decentralization, and building distributed ledgers for business and enterprise. We will be discussing next steps for growing the Cardano community alongside oncoming blockchain regulations, governance, and opportunities. In combination, we hope the summit tracks will offer something for anyone interested in the future of Cardano, by the science and ideas that surround it, by the great minds making it all happen and by the incredible community that has brought us here and will take us forward.

Sessions will encompass the philosophical as well as the technical. Brian Behlendorf, the CEO of the Hyperledger Consortium will form part of a panel focused on the importance of open source development. IOHK recently joined the Hyperledger Consortium, to better exploit our common vision of a future made better through shared knowledge. Following on the philosophy track, artificial intelligence researcher, Ben Goertzel will speak on the intersection of AI and decentralized technology. We hope to announce more exciting sessions in the days leading up to the event. At the end of the day, building the next generation of technology means bringing the best minds of many fields into the same room, even if it’s a virtual one.

Building community

The Cardano Virtual Summit 2020: Shelley Edition, won’t be all work and no play. The Covid crisis has forced every conference online in past months. While we can’t recreate the full physical conference experience in the virtual space, we’re keen to provide some of the networking and downtime opportunities you might expect. So we’ve added a virtual ‘chill-out’ zone, with guided meditations, a DJ set, and even an online calligraphy lesson/demonstration. Our virtual platform will allow attendees to enjoy the digital expo space through avatars. We’ll even get the conversation flowing with a digital Shelley cocktail ‘happy hour’ between meetings.

Recent events have made it difficult to meet in person but we see The Cardano Virtual Summit as an opportunity to invite everyone from around the world to participate. Attendance for anyone interested in the summit is absolutely free of charge. To join in, simply reserve your spot.

Shelley is the culmination of over 5 years of research and development, the creation of a multi-disciplinary team and a remarkable community. The virtual summit is just a single point of time – a time to take stock, reflect and celebrate. But it marks just the start of a groundbreaking new era of decentralization, growth and adoption.

Setting solid parameter values – while maintaining flexibility for the future – will be key to the growth and ongoing decentralization of Cardano. After consulting with the community, and working closely with my colleagues Kevin Hammond and Alex Appledoorn, we believe we’ve identified a good place to start.

The behavior of Cardano Shelley is controlled by around 20 parameters, and values have to be set for all of those before we launch the mainnet. Most of these parameters are technical in nature, so while setting them correctly is important to guarantee the safety and optimize performance of the system, their particular values do not have a significant influence on user experience.

Some parameters are different, though. They determine the level of centralization and sustainability of the Cardano ecosystem. They also drive the economics of delegation and of operating a stake pool. Choosing good values for these is exceedingly complicated, because we have to carefully balance a number of important considerations; security, performance, stability, sustainability, decentralization, fairness and economic viability.

With all parameters on the Cardano blockchain, we have three distinct goals to keep in mind:

• We want to be truly decentralized, so that no one party can threaten the integrity of the chain
• We want the stake pool operators to be incentivized to keep supporting our chain
• We do not want these incentives to significantly change at any singular point in time in a way that might negatively affect the stability of the operators’ income

We want to give equal opportunity to everyone who wants to participate in Cardano and run a stake pool. However, parameter values that might seem fair and reasonable for smaller pools can become challenging for larger pools and vice versa. For example, large pools could find it easy to put down a higher pledge than small pools can afford. Small pools, on the other hand, might be able to operate with far lower costs than larger pools.

We also consider it imprudent to change parameters too frequently, because this might negatively affect the stability and predictability of the operators’ income. Taking all of this under consideration we came up with some recommendations for initial choices of parameter values which we will outline here.

However, we do not want to stop there. With decentralization comes democracy. Our community must have a say in how the chain is governed. For this reason, we will run with these numbers initially and issue a Cardano improvement proposal, where the community can vote on optimal chain parameters. In the end, the governance of Cardano will be in the hands of the Cardano community, who we feel confident are the best people to advise us.

Desired number of stake pools

The desired number of stake pools k is an important parameter. Cardano incentives have been designed to encourage an equilibrium with k fully saturated pools, which means that rewards will be optimal for everybody when all stake is delegated uniformly to the k most attractive pools.

The higher k chosen, the more decentralized the system becomes. But a higher k also leads to a less efficient system (higher costs, more energy consumption) and lower rewards for both delegators and stake pool owners. Based on what we have learned from both the Incentivized Testnet (ITN) and the Haskell Shelley testnet, we know that our community is highly motivated to set up pools and support the chain with hundreds within a matter of weeks.

This tells us that some measure of decentralization can – and will – happen relatively quickly. But decentralization alone is not enough. Cardano needs a long term commitment from its operators, and conversely, operators need to be sufficiently incentivized to keep supporting the system.

To strike a balance between decentralization and these incentives for stake pool operators, we are proposing an initial k=150 and then to gradually increase that value. We believe this will ensure that the system is stable and efficient in the beginning, and can gradually grow over time to become more decentralized (and even more secure) later on:

The number of 150 stake pools of roughly equal size makes Cardano an order of magnitude more decentralized than any other blockchain. And this is only the beginning. There is no reason why there could not be thousands of stake pools in the future.

Monetary expansion

Staking rewards for both delegators and stake pool operators are taken from two sources; transaction fees and monetary expansion. Specifically, every epoch, all the transaction fees from every transaction from all blocks produced during that epoch are put into a virtual 'pot'. Additionally, a fixed percentage, ρ, of the remaining ada reserves is added to that pot. Then a certain percentage, τ, of the pot is sent to the treasury, the rest is used as epoch rewards.

This mechanism ensures that in the beginning, when the number of transactions is still relatively low, because users are just starting to build their business on Cardano, the portion of rewards taken from the reserves is high. This provides a great incentive for early adopters to move quickly and benefit from the high initial rewards. Over time, as transaction volume increases, the additional fees compensate for dwindling reserves.

This mechanism also ensures that available rewards are predictable and change gradually. There will be no sudden 'jumps' comparable to bitcoin halving events every four years. Instead, the fixed percentage taken from remaining reserves every epoch guarantees a smooth exponential decline.

So what value should ρ have? And how much should go to the treasury? This is again a trade off: higher values of ρ mean higher rewards for everybody initially and a treasury that fills faster. But higher values of ρ also mean faster reserve depletion. It is certainly important, especially in the beginning, to pay high rewards and incentivize early adopters. But it is also important to provide a long term perspective for all stakeholders.

As explained above, Cardano will never run out of reserves; look instead at an exponential decay. To get a feeling for the impact of a specific value of ρ, one can calculate the 'reserve half life', the time it takes for half of the reserve to have been used up.

After much deliberation, we arrived at a suggestion of 0.22% for ρ. When you crunch the numbers, you get around four to five years as 'reserve half life' for this. In other words, every four to five years, half of the remaining reserve will be used. This is close to the 'bitcoin half life' of circa four years, so Cardano reserves will deplete at about the same rate as bitcoin reserves.

It is worth noting here that it took Bitcoin around eight years to reach its peak of maximum adoption and price. We therefore feel that it makes sense to expect Cardano transaction volume and exchange rate to increase sufficiently over the next eight years to more than make up for the decrease of monetary expansion during that time.

From reserves to treasury

We also propose an initial value of 5% for τ, the percentage of rewards automatically going to the treasury every epoch. This means that at least 380,000,000 ada will be sent from the reserves to the treasury over the next 5 years.

However, the real amount going to the treasury will be significantly higher. First of all – again taking learnings from the ITN, but also predicting the use of ada in the future – it’s unreasonable to assume that all ada will be delegated. Some of it will be locked-up in exchanges, be transacted and used in various smart contracts. The ada that is not being delegated will produce unclaimed awards. Those 'unclaimed rewards' also go to the treasury, which will bump the amount to around 1,900,000,000 ada.

Secondly, we do not expect the pledge of most pools to be particularly high, just high enough to make it unattractive to launch a Sybil attack. The difference between potential pool rewards with a very high pledge and pools with the more realistic pledge level we expect goes to the treasury as well and will add an additional 1,000,000,000 ada over the first five years. The sum of all the ada flowing to the treasury means that there will be sufficient funds to pay for new exciting features and extensions for the foreseeable future.

Pledge influence factor & minimum operational cost settings

Ada that is pledged by pool owners provides essential protection against 'Sybil' attacks by ensuring that delegated stake is not excessively attracted to pools whose owners try to attack the system by creating a large number of pools without themselves owning a lot of stake. Myself, Kevin Hammond and Duncan Coutts covered this in some detail recently on the Cardano Effect show.

The pledge influence factor directly affects the rewards that a pool earns: the higher the influence factor, the more of a difference a higher pledge makes on rewards. A higher influence factor increases the level of Sybil protection and makes the system safer and more secure, but it also gives an advantage to stake pool owners that can afford a higher pledge.

Higher pledge can be used to compensate for higher operational costs, meaning that a pool with relatively high costs can maintain suitable rewards and remain attractive to delegators by increasing its pledge. We have tested a variety of pledge influence factors under various real world conditions (about a million simulations in all). The influence factor can range between 0 and infinity. Our chosen initial setting of 0.3 is designed to balance the level of Sybil protection against the required pledge.

There is no minimal pledge, though. Pool operators can set the pledge as low or as high as they like. Rewards are influenced by their choice, but there is no 'hard' rule forcing them to pledge a specific amount. This means that ultimately, pool pledges will be as high as pool owners are willing to make them, and it will be up to our community to find a sweet spot between protection against attacks, economic considerations and the desire for fairness and equal opportunity.

The minimum operational cost setting ensures that the pledge influence factor is effective, by avoiding a 'race to the bottom' where pool owners claim excessively low operating costs in order to gain a competitive advantage. While this might benefit ada stakeholders in the short term, the long-term effect would be to risk the health of the Cardano network by disincentivizing professional pool operation.

Distribution of Typical Pool Operating Costs per pool per year, obtained from a survey of experienced pool operators in May 2020.

Genuine low cost operators can greatly benefit from the minimum operational cost, because the difference between the minimal cost and their actual cost provides them with additional income on top of their margin and their staking rewards. Our research shows that typical operating costs are expected to be in the $2,000-$15,000 range per pool per year, as shown in the diagram above. We have therefore chosen a setting of $2,000 for the minimum operational cost.

Estimated Range of Average Return on Investment (ROI) for Stake Pools assuming a Monetary Expansion Rate of 0.22% per epoch.

Finally, we calculated the expected returns for stake pools under a range of different real world scenarios (about 150,000 pools in total). We used the settings for the influence factor, monetary expansion and minimum cost that were given above and varied the targeted number of pools between 150 and 500. Our results show that given the distribution of costs that we showed in the diagram above, stake pools will achieve sustainable ROIs of between 6%-6.5% on average, using today’s ada to dollar conversion rate. The ROIs would, of course, be even better if the value of Ada were to appreciate.

Conclusion

Choosing good values for all Cardano Shelley parameters is a hard and complicated endeavor, because a lot of concerns have to be balanced – security, efficiency and stability of the system on the one hand versus economic viability for stake pool operators and delegators and long-term sustainability of the ecosystem on the other hand.

No other blockchain has ever done what we are going to do, we are charting new territory with every step and move at the cutting edge of science and technology, so we can’t rely on existing data and statistics or past experience, but have to use educated guesses and mathematical models, which can never be perfect, more often than not.

We did our best to come up with a reasonable proposal, but we know it will have to be improved upon over time. The values proposed here are just a start, and we will closely work with our Community to refine and adjust them over the coming months and years.