Joystream Handbook
Search…
πŸ“
Council Period Scoring
Reward what you want to see in the world.

Introduction

There needs to be a way to measure how well the system is operating overall, as this is a key signal to the council about whether it is doing a good job or not. Given such a signal, the council would be able to change policies and reallocate resources to increase performance as measured by this signal. On mainnet, the market capitalization of $JOY is effectively this signal, but on testnet we have to find some substitute for this, and that is the role of the network performance score.

Group Scores

Previous Scoring Rounds

Here is a live dashboard where you can look up the results for individuals per scoring period, and overall status.
​

Council Period Parameters

Valid through council scoring period number 8, for the council elected at block #748,800.
Note: Increased by 5% to account for Forum weighting and score.
Name
Value
JOY_BUDGET
2,500,000 (0.25% ~USD 150,000)
REFERRER_JOY
1000 (~USD 60)
REFERREE_JOY
1000 (~USD 60)
tJOY_BUDGET
100,000,000
USD_SUBSIDY
USD 2400 *
CAP
JOY 15,000 (~USD 9,000)
COUNCIL_tJOY_REWARD
3,024,000 (15,120,000 in total)
`*` includes $100 subsidy for SPs, and $150 subsidy for distributors This means that JSG will mint $250 worth of tJOY and distribute across the workers, but these tJOY will not earn the recipients JOY.
The Parameters for each new scoring period should be made available the following Monday 1800 CET, along with the updated metrics.
Until published, the council should assume they will stay the same until the update is made.
One change that can be accounted for already, is the fact that the Storage Providers and Distributors will receive an extra $tJOY subsidy, that will NOT count towards their $JOY earnings. This is done to account for the fact that workers (not Leads) in these roles have real operational costs, but their role doesn't require much time. The subsidy will be distributed at the time of grading, and will be shared equally across each worker in each group, adjusted by the number of blocks they were part of the group.

Summary Report and Plan Deadlines

For the system to work well, there is a need for a feedback-loop, between
For this to be the case, strict deadlines are required.
The following deadlines will apply for a Council Scoring Roundn with definitions below.
Deliverable
Responsible
Deadline
Note: Council n elected
NA
NA (B_ce)
​Working Group Summary (for n-1)
WG Leads
Max(B_ce+3600, N_cs)
​Council Period Summary (for n-1)
Council
Max(B_ce+3600, N_cs)
Period Scoring Results (for n-1)
Jsgenesis
Min[B_ce+21600,
09.05-1400CET]
​Council Period Parameters and Metrics (for n)
Jsgenesis
Min[B_ce+24000,
09.05-1800CET]
​Council Period Plan (for n)
Council
Max[B_ce+24000, PPM+3000]
WG Leads
Max[B_ce+24000, PPM+3000]
  • B_cs means the block height the council term started (B_cs+600thus means 600 blocks later)
  • N_cs means the first time the clock strikes noon CET after the council term started
  • B_ce means the block height the council term ended
  • N_ce means the first time the clock strikes noon CET after the council term ended
  • PPM means when the new council period parameters and metrics are posted.

Dashboard

TODO

Knowledge Bases

The council and each working group must maintain its own Notion board where a body of knowledge related to the operations of the group are maintained. This base will be owned by Jsgenesis, while on testnet, and the current lead of each working group and all council members will be given write access to maintain it. The integrity of the boards is the responsibility of each person granted access, and sharing of credentials is not allowed. If this constraint causes problems in efficiently maintaining the boards, contact Jsgenesis.

Council Knowledge Base

Council
Joystream on Notion
Council Knowledge Base

Total tJOY Spending

The total tJOY spending over a given council period is something Jsgenesis attempts to directly constrain through its policy parameter tJOY_BUDGET , and it the sum of the following non-overlapping categories of flows during the blocks of a council period
  • Any increase in tJOY that any validator can cashout, but currently has not.
  • Any tJOY that has actually been cashed out by validators.
  • Any tJOY actually awarded to any council member, lead or worker.
  • Any tJOY paid through a Funding Request.
  • Any $tJOY approved for bounty work.

Council Period Summary

Motivation

  • Allow Jsgenesis to update the Council Period Parameters and network-performance-score metrics for the next period based on up to date and accurate information.
  • Allow the possible next council to have good information about the most urgent matters and status of the network.

Scope

  • Accounting of how much was spent on what.
  • Actual hires made.
  • Actual firings done.
  • Spending proposals passed.
  • Changes made to the notion board.
  • A summary on how well the network as a whole performed, and what problems occurred.
  • Recommendations for what should be focused on in next council period in order to make the network more effective.
  • Feedback on the Council Period Parameters during their term.

Submission

The report itself must be written as a markdown document in English, in the forum category Governance>Council Reports, as a thread which has the title which includes the council period ID. The council must pass a text proposal (add link when proposal list is in) which references the report on the forum.
If there is a need to link to extra information, statistics, etc. these should be in the notion board.

Council Period Plan

Motivation

Give Jsgenesis visibility into the priorities of the council, and also serve as a canonical document to focus everyone in the council around a clearly defined plan.

Scope

  • Prioritization to each working group about what they should focus on solving.
  • How total $tJOY budget will be allocated across groups and council.
  • Set the schedule for all the Council Meetings until the council period is over.
  • List all known tasks to be assigned to individual council members, and add them to the council period task tracker on the Notion board.
  • The council are further encouraged to include specific workflows for items such as:
    • How to follow up each working group, and by whom
    • How budgets are managed, and by whom
    • Who is responsible for taking and publishing minutes
    • etc.

Submission

The plan itself must be written as a markdown document in English, in the forum category Governance>Council Reports, as a thread which has the title which includes the council period ID. The council must pass a text proposal (add link when proposal list is in) which references the plan on the forum.
If there is a need to link to extra information, statistics, etc. these should be in the notion board.

Council Meetings

As a minimum, all council members are expected to be available for
  • two handover calls, after getting elected and after their term has ended
    • organized and always attended by Jsgenesis
  • daily standups for coordinating between them
    • organized by the council themselves

Motivation

Handover

A handover meeting allows the new council to prepare for their Council Period Plan with assistance from both the previous council and Jsgenesis. The previous council will get immidiate feedback on their Council Period Summary and Jsgenesis will learn what the current pressure points are.

Daily Standup

As for any organization, internal communication is key for performance. Every day between the handovers, the Council shall have a daily sync call, held in public on Discord. The scheduling will be set in the Council Period Plan, where the inaugural shall be after Jsgenesis publishes the new Council Period Parameters and the scoring metrics for the period, but before the deadline to release the Council Period Plan.
See Summary Report and Plan Deadlines for more information.

Scope

Handover

  • Discuss the performance of the previous council
  • Review the temporary summary, presented by the previous council, for the benefit of all parties, with an emphasis on lessons learned
  • Questions and comments
A member of the new council takes notes, and quickly prepares a brief minutes of meeting to post on the forum or discord for feedback by the others. To avoid conflict and reduce the barrier to speak, the minutes must obey the "Chatham House Rules".

Daily Standup

  • Follow up the tasks in the councils own task tracker

Submission

Handover

Once approved (informally, eg. no comments in discord unaddressed), the minutes are submitted in the forum category Governance>Council Reports as a thread which has the title which includes the two council period IDs.

Daily Standup

Minutes are added to the Governance>Council Reports in a thread that contains all standups for the week.

Council Daily Sync

​

Lead Opportunities

There must be made space for people to try to participate as leads for working group, even to the extent that other more experienced - but proficient leads, have to pause their participation. It is up to the council to determine the cheapest way to accommodate newcomers without undermining the operations of the working group. Not only must the space be allocated and reserved for newcomers, but there must be effective collaboration with the Human Resources working group to actually identify potential applicants and encourage them to apply to these opportunities.

Score

Overview

At the beginning of each council Jsgenesis staff will state a set of explicit metrics that will apply to the upcoming council period. At the end of each council period, Jsgenesis staff will determine a final network performance score for the council which will be in the range [0,1]. This score is used to determine the $tJOY reward and founding-member-points for the council members, as described above.
Specifically, the score is a weighted linear combination of scores working group scores, which themselves are in the range [0,1], which is normalized by the weights, and lastly discounted by exponentially by the number of catastrophic errors detected, that is
1
NETWORK_PERFORMANCE_SCORE = [
2
BUILDER_SCORE*B_W +
3
CONTENT_SCORE*C_W +
4
DISTRIBUTOR_SCORE*D_W +
5
FORUM_SCORE*F_W +
6
HR_SCORE*HR_W +
7
MARKETER_SCORE*M_W +
8
STORAGE_SCORE*S_W +
9
SUMMARY_SCORE*SU_W +
10
PLAN_SCORE*P_W +
11
MEETING_SCORE*ME_W +
12
LEAD_OPPORTUNITIES_SCORE + LO_W
13
]/((B_W + C_W + D_W + F_W + HR_W + S_W + M_W + SU_W + ME_W + P_W + LO_W)*2^N)
Copied!
where
  • BUILDER_SCORE (B_W): computed with metrics defined in Builders Score.
  • CONTENT_SCORE(C_W): computed with metrics defined in Content Directory Score.
  • DISTRIBUTOR_SCORE (D_W): computed with metrics defined in Forum Score .
  • FORUM_SCORE (F_W): computed with metrics defined in Distributors Score.
  • HR_SCORE (HR_W): computed with metrics defined in Human Resources Score.
  • MARKETER_SCORE (M_W): computed with metrics defined in Marketers Score.
  • STORAGE_SCORE (S_W): computed with metrics defined in Storage Providers Score.
  • SUMMARY_SCORE (SU_W): is a score computed for the quality of the council summary, which will be in the range [0, 1], and will emphasize things like
    • Clarity of communication and organization.
    • Appropriate scope.
    • Accuracy of facts and information.
    • Quality of changes to the knowledge base.
  • PLAN_SCORE (P_W): is a score computed by Jsgenesis staff for the quality of the council plan, which will be in the range [0, 1], and will emphasize things like
    • Clarity of communication and organization.
    • Appropriate scope.
    • Accuracy of facts and information.
  • MEETING_SCORE (ME_W): is a score in the range [0, 1], computed based on two factors:
    • Turnout of council members (n/m), for all handover and standups, where 80% is required to achieve a full score
    • The quality and promptness of publishing the minutes of meeting, where the score falls linearly from 1 to 0 if minutes are published between 2 hour and 8 hours after the meeting started
  • LEAD_OPPORTUNITIES_SCORE is 1/min(x_1,..., x_k) , which will be in the range [0, 1], where x_i is the total number of council period in which the ith lead has worked in this group.
  • *_W : are the weights from the table below.
  • N : The number of catastrophic error instances which occurred, as defined below.

Weights

The current weights are:
Weight
Value
B_W
8
C_W
4
D_W
6
F_W
1
HR_W
8
M_W
S_W
6
SUM_W
2
P_W
2
CM_W
2
LO_W
2
Jsgenesis reserves the right to add 1 point to the M_W assuming a scope is agreed.
Which Means:
1
NETWORK_PERFORMANCE_SCORE = [
2
BUILDER_SCORE*B_W +
3
DISTRIBUTOR_SCORE*D_W +
4
CONTENT_SCORE*C_W +
5
HR_SCORE*HR_W +
6
MARKETER_SCORE*M_W +
7
STORAGE_SCORE*S_W +
8
SUMMARY_SCORE*SUM_W +
9
PLAN_SCORE*P_W +
10
COUNCIL_MEETING_SCORE*CM_W +
11
LEAD_OPPORTUNITIES_SCORE*LO_W
12
]/((B_W + C_W + D_W + HR_W + M_W + S_W + SUM_W + P_W + CM_W + LO_W)*2^N)
Copied!

General Working Group Grading Notes

For Jsgenesis to be able to complete the grading in "due time", all plans summaries needs to be
  • easy to find (eg. in the applicable forum category)
  • punctual
  • formatted the same, easy to digest, way
To address this problem, the council will be tasked to create a template for the general working group summary and plan. Any report that is not made from this template, and posted on the forum in time, will score 0.
​

Resources

It has become clear that some extra resources are needed for all parties to:
  • See grading in flight
  • Learn how Jsgenesis is grading each metric
  • Further tips and ideas, not formally part of the overall scoring
https://www.notion.so/joystream/Tools-and-Resources-8969641b5b284d1c85c658106a79792d
www.notion.so
Tools and resources for grading

Catastrophic Errors

Missed runtime upgrade

A runtime upgrade was proposed by Official Jsgenesis Membership, but it was not approved.

Upgrading non-Jsgenesis runtime

A runtime upgrade, not proposed by Official Jsgenesis Membership, was approved.

Total $tJOY exceeds tJOY_BUDGET

Total spending, as defined in total-usdtjoy-spending, exceeded tJOY_BUDGET

Block time too low

The average time between blocks was greater than 10s for more than a 2 hour interval.

Member faucet empty

The membership faucet requires both that the Membership working group budget has sufficient tJOY to pay the membershipPrice and that the Lead has sufficient invitations. If at any point during the council term, the faucet capacity is less than 5 new members, that counts as effectively empty.

On chain Budget and Opening proposals without JOY equivalent

Any approved proposals that deals with a working group budget (denominated in tJOY), or the opening for a Lead role (reward denominated in tJOY), must also contain a reference to the JOY value. This is a key factor in increasing awareness of the value of incentives, and must be respected as such.