1. fabric@lists.hyperledger.org
  2. Messages
   Date   
COULD NOT ASSEMBLE TRANSACTION, ERR PROPOSAL RESPONSE WAS NOT SUCCESSFUL, ERROR CODE 500, MSG INSTANTIATION POLICY VIOLATION: SIGNATURE SET DIDNOT SATIFY POLICY #fabric-chaincode

Faisal
 

Failure Scenario

1- Started the network with privatechannel_cc v1 chaincode installed
2- Did multiple transactions 
3- Ran a script to upgrade the privatechannel_cc v2 ( GOT THE ABOVE ERROR)
`COULD NOT ASSEMBLE TRANSACTION, ERR PROPOSAL RESPONSE WAS NOT SUCCESSFUL, ERROR CODE 500, MSG INSTANTIATION POLICY VIOLATION: SIGNATURE SET DIDNOT SATIFY POLICY`
4- The condition for step 3 is that the peer went down just before the chaincode upgrade and were out-of-sync. They were in the process of fetching all the previous blocks and the chaincode containers of V1 were down on both machines.


Checked the IDs (Hashes) of chaincodes installed on Peers of both Orgs and they are same.
Tried to instantiate instead of upgrade and it gives error as the chaincode is already instantiated.
Tried to query/invoke the chaincode and it gives the following error

"[channel org1channel] failed to get chaincode container info for privatechannel_cc:v1: could not get chaincode code: chaincode fingerprint mismatch: data mismatch" 

LOGS of the peer are attached in the file below.

 

peer chaincode upgrade -C org1channel -n privatechannel_cc -l golang -v v2 -c '{"Args":["init"]}' -P "OutOf(2,'org1MSP.member','org2MSP.member')" -o 192.168.171.32:7050 --tls --cafile /data/tls/ca.pem --clientauth --keyfile /data/tls/client-key.pem --certfile /data/tls/client.pem

?
cc_logs_org1

Meet the biggest technologists in blockchain at Genesis DevCon

Suzana Joel <suzana.joel@...>
 



Just 14 days to go!

Know more about the Speaker @ Genesis DevCon
The conference editorial is a mix of accomplished experts in the blockchain industry who are researchers, entrepreneurs, and innovators from India and across the globe.
Be sure to get a first-hand experience in learning about recent developments in blockchain technology and its real-world blockchain applications.

We're sure you will have a great time meeting them.

There's a special discount for the members of Hyperladger Fabric.
Coupon Code: GENESIS50
Book Your Seat Now
We hope to see you at Genesis DevCon.

Regards,
The IBC Media Team

ᐧ

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This message and any files or text attached to it are intended only for the recipients named above, contain information that is confidential or privileged. If you are not an intended recipient, you must not read, copy, use or disclose this communication. Please also notify the sender by replying to this message, and then delete all copies of it from your system.
Re: Wrong world state #fabric #fabric-questions

Joao Antunes
 

Thank you for reaching out again Dave,

In my case, for some unknown reason, I had 1 good world state and 3 bad world state. After updating fabric image on peers and resetting all peers, the world state got coherent with the ledger. So from my analysis, world state was wrong, but the ledger was correct.
I had 3 wrong peers and 1 correct, but even then I was not able to change anything due to my queries readset. One of the readsets was different so, no actions were possible.
How this happened I don't know. I know that these entities originated from a single huge transaction (10K new entities created) but the ownership of some of them (3 or 4) were not coherent in the peers. Even after deleting the CouchDB database from the peer, it still recovered but with wrong data. Only resetting the peer solved the issue.

I'll investigate more and if possible I'll provide more data to this thread and if that's the case a Jira issue
Re: Hyperledger Fabric Scalability

Christopher Ferris
 

Indeed, I think there's often confusion about network topology when it comes to "organizations". To be clear, there are 3 types of users,
orgs that operate ordering nodes, orgs that operate peer nodes and end users of the system/application who likely do not operate more than
just an application leveraging one of the SDKs. As for orgs that operate peer nodes, these could be distinguished between those who
are operating an endorsing node, and those merely operating a validating/committing node. The former need to be able to operate the chaincode
the latter do not, they just maintain a copy of the ledger.

In a system with thousands or more users, there might only be tens or hundreds of organization nodes operating peers. Think of it
using the banking example. The banks may create a consortium network and Bank America, TD Bank, Wells Fargo, Sovereign Bank each
operate peer nodes that are both endorsing and validating. Maybe they also operate ordering nodes. Then, all the TD bank account
holders would have identities affiliated with TD Bank, and all the Bank America account holders similarly have identities affiliated with
Bank America, etc. The account holders don't operate nodes, they simply interact with the system from their mobile or web application.

The channels might be bilateral between the banks, and transactions that exchanged funds from one bank to another would be
processed over those bilateral channels. Otherwise, there would be a channel per bank against which account holders would process
their individual transactions. The chaincode would leverage access controls that ensured that the account holder, or an authorized
entity could process transactions against their account.

As for future blog posts, I plan to publish updates once we have 2.0 images published. I'll share the blog links here.

Chris


On Sat, Nov 9, 2019 at 1:00 AM Brian Behlendorf <bbehlendorf@...> wrote:
The blog posts are the right place to start your exploration of the issues, but you can't go much further without testing it yourselves.

You should be apprehensive about depending upon any blockchain for high TPS and high numbers of nodes at the same time. Proper blockchain engineering at this point will be about finding ways to store the least amount of data on chain, with the least number of required TPS, as possible. Do not confuse Fabric or any other blockchain to be a high performance big data management tool. If anything, it's about "small data" that happens to be trust-critical.

As I understand your use case, in my opinion, you should not be storing all sensor data and transaction history directly on ledger; store that in large files chunked by time and gas station network, share those offledger (encrypted S3 buckets ate cheap and global, or IPFS), and use Fabric for storing hashes that can attest to the timing, provenance, metadata and integrity of that off-ledger data.

You also shouldn't try to have every gas station be a node in the network - I presume they are already part of a chain of gas stations, and within a chain there is already the trust of a single corporation, so the corp office itself should run nodes on behalf of its stations. It doesn't sound like you need prevention of double spend; I'm not even sure why you need smart contracts (you mean chaincode?) to "monitor fuel levels".

If scale is your number one goal, constantly look for ways to reduce the frequency and amount to write to the blockchain. (Reads, of course, are cheap and easy to scale).

Brian

On 9 November 2019 10:47:52 AM GMT+08:00, alok gupta <metech11@...> wrote:
Hello Mark & Christopher,

Thanks for your reply. I have gone through the blog but I am still apprehensive whether we would be able to support hundreds or thousands of organisations? We did TPS optimisation after struggling a bit with MVCC error, our solution is running fine for a single fuel station. Right now there are max 50 tps.  But not sure how to take this forward. What should be our approach?  Please advise.



On Sat, 9 Nov 2019, 07:59 Mark Rakhmilevich, <mark.rakhmilevich@...> wrote:

Alok,

    We have seen throughput in high hundreds of TPS to low thousands of TPS in Oracle projects.  The scalability and performance depend on many factors well explained in Chris’ blog posts.  Certainly transaction payload size, batching parameters in the ordering service, number of channels an ordering cluster has to support given certain network bandwidth play a significant role. There are also techniques to batch transaction data, which are useful, for example,  to capture an audit log from many IOT sensor readings. Feel free to reach out directly to discuss specifics.

 

Best Regards,

    Mark

 

From: Christopher Ferris <chris.ferris@...>
Sent: Wednesday, November 06, 2019 3:33 AM
To: alok gupta <metech11@...>
Cc: fabric@...
Subject: Re: [Hyperledger Fabric] Hyperledger Fabric Scalability

 

Alok,

 

I wrote a couple of blog posts earlier this year on Hyperledger Fabric performance and scale.

 

There have been some more recent improvements that should improve performance even more when 2.0 ships.

I'll have another post up with those results. There are some additional efforts in the community where performance

has been pushed even further.

 

Hope this helps.

 

Chris

 

On Wed, Nov 6, 2019 at 3:36 AM alok gupta <metech11@...> wrote:

Hello There,

We are conducting a POC for Oil & GAS Retail automation. In which, we are recording all digital/ cash sales onto the Fabric ledger. We monitor the stock levels in the fuel tanks through smart contract. The idea is to replace the current automation system in India which requires massive investment in installation and maintenance. Our app is running successfully on a fuel station at a fuel station in Chandigarh, India. 

My query is about the scalability of fabric over no. of channel, organizations, and peers. Can we scale up our solution to connect the fuel companies ( IOCL. HPCL etc.) with India wide fuel stations? I have seen other use cases like Wallmart food safety where there are running a huge network on blockchain. To move forward in our idea, we need a clarity on scalability Please advise.

Thank you
Alok


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Hyperledger Fabric Scalability

alok gupta <metech11@...>
 

Hello Brian,

Thank you so much for your guidance. It helps, we will develop the architecture according to your inputs.

On Sat, Nov 9, 2019 at 11:30 AM Brian Behlendorf <bbehlendorf@...> wrote:
The blog posts are the right place to start your exploration of the issues, but you can't go much further without testing it yourselves.

You should be apprehensive about depending upon any blockchain for high TPS and high numbers of nodes at the same time. Proper blockchain engineering at this point will be about finding ways to store the least amount of data on chain, with the least number of required TPS, as possible. Do not confuse Fabric or any other blockchain to be a high performance big data management tool. If anything, it's about "small data" that happens to be trust-critical.

As I understand your use case, in my opinion, you should not be storing all sensor data and transaction history directly on ledger; store that in large files chunked by time and gas station network, share those offledger (encrypted S3 buckets ate cheap and global, or IPFS), and use Fabric for storing hashes that can attest to the timing, provenance, metadata and integrity of that off-ledger data.

You also shouldn't try to have every gas station be a node in the network - I presume they are already part of a chain of gas stations, and within a chain there is already the trust of a single corporation, so the corp office itself should run nodes on behalf of its stations. It doesn't sound like you need prevention of double spend; I'm not even sure why you need smart contracts (you mean chaincode?) to "monitor fuel levels".

If scale is your number one goal, constantly look for ways to reduce the frequency and amount to write to the blockchain. (Reads, of course, are cheap and easy to scale).

Brian

On 9 November 2019 10:47:52 AM GMT+08:00, alok gupta <metech11@...> wrote:
Hello Mark & Christopher,

Thanks for your reply. I have gone through the blog but I am still apprehensive whether we would be able to support hundreds or thousands of organisations? We did TPS optimisation after struggling a bit with MVCC error, our solution is running fine for a single fuel station. Right now there are max 50 tps.  But not sure how to take this forward. What should be our approach?  Please advise.



On Sat, 9 Nov 2019, 07:59 Mark Rakhmilevich, <mark.rakhmilevich@...> wrote:

Alok,

    We have seen throughput in high hundreds of TPS to low thousands of TPS in Oracle projects.  The scalability and performance depend on many factors well explained in Chris’ blog posts.  Certainly transaction payload size, batching parameters in the ordering service, number of channels an ordering cluster has to support given certain network bandwidth play a significant role. There are also techniques to batch transaction data, which are useful, for example,  to capture an audit log from many IOT sensor readings. Feel free to reach out directly to discuss specifics.

 

Best Regards,

    Mark

 

From: Christopher Ferris <chris.ferris@...>
Sent: Wednesday, November 06, 2019 3:33 AM
To: alok gupta <metech11@...>
Cc: fabric@...
Subject: Re: [Hyperledger Fabric] Hyperledger Fabric Scalability

 

Alok,

 

I wrote a couple of blog posts earlier this year on Hyperledger Fabric performance and scale.

 

There have been some more recent improvements that should improve performance even more when 2.0 ships.

I'll have another post up with those results. There are some additional efforts in the community where performance

has been pushed even further.

 

Hope this helps.

 

Chris

 

On Wed, Nov 6, 2019 at 3:36 AM alok gupta <metech11@...> wrote:

Hello There,

We are conducting a POC for Oil & GAS Retail automation. In which, we are recording all digital/ cash sales onto the Fabric ledger. We monitor the stock levels in the fuel tanks through smart contract. The idea is to replace the current automation system in India which requires massive investment in installation and maintenance. Our app is running successfully on a fuel station at a fuel station in Chandigarh, India. 

My query is about the scalability of fabric over no. of channel, organizations, and peers. Can we scale up our solution to connect the fuel companies ( IOCL. HPCL etc.) with India wide fuel stations? I have seen other use cases like Wallmart food safety where there are running a huge network on blockchain. To move forward in our idea, we need a clarity on scalability Please advise.

Thank you
Alok


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.


--
Thank you
Regards
Alok Gupta
+91-9646579412
Re: Hyperledger Fabric Scalability

Brian Behlendorf <bbehlendorf@...>
 

The blog posts are the right place to start your exploration of the issues, but you can't go much further without testing it yourselves.

You should be apprehensive about depending upon any blockchain for high TPS and high numbers of nodes at the same time. Proper blockchain engineering at this point will be about finding ways to store the least amount of data on chain, with the least number of required TPS, as possible. Do not confuse Fabric or any other blockchain to be a high performance big data management tool. If anything, it's about "small data" that happens to be trust-critical.

As I understand your use case, in my opinion, you should not be storing all sensor data and transaction history directly on ledger; store that in large files chunked by time and gas station network, share those offledger (encrypted S3 buckets ate cheap and global, or IPFS), and use Fabric for storing hashes that can attest to the timing, provenance, metadata and integrity of that off-ledger data.

You also shouldn't try to have every gas station be a node in the network - I presume they are already part of a chain of gas stations, and within a chain there is already the trust of a single corporation, so the corp office itself should run nodes on behalf of its stations. It doesn't sound like you need prevention of double spend; I'm not even sure why you need smart contracts (you mean chaincode?) to "monitor fuel levels".

If scale is your number one goal, constantly look for ways to reduce the frequency and amount to write to the blockchain. (Reads, of course, are cheap and easy to scale).

Brian


On 9 November 2019 10:47:52 AM GMT+08:00, alok gupta <metech11@...> wrote:
Hello Mark & Christopher,

Thanks for your reply. I have gone through the blog but I am still apprehensive whether we would be able to support hundreds or thousands of organisations? We did TPS optimisation after struggling a bit with MVCC error, our solution is running fine for a single fuel station. Right now there are max 50 tps.  But not sure how to take this forward. What should be our approach?  Please advise.



On Sat, 9 Nov 2019, 07:59 Mark Rakhmilevich, <mark.rakhmilevich@...> wrote:

Alok,

    We have seen throughput in high hundreds of TPS to low thousands of TPS in Oracle projects.  The scalability and performance depend on many factors well explained in Chris’ blog posts.  Certainly transaction payload size, batching parameters in the ordering service, number of channels an ordering cluster has to support given certain network bandwidth play a significant role. There are also techniques to batch transaction data, which are useful, for example,  to capture an audit log from many IOT sensor readings. Feel free to reach out directly to discuss specifics.

 

Best Regards,

    Mark

 

From: Christopher Ferris <chris.ferris@...>
Sent: Wednesday, November 06, 2019 3:33 AM
To: alok gupta <metech11@...>
Cc: fabric@...
Subject: Re: [Hyperledger Fabric] Hyperledger Fabric Scalability

 

Alok,

 

I wrote a couple of blog posts earlier this year on Hyperledger Fabric performance and scale.

 

There have been some more recent improvements that should improve performance even more when 2.0 ships.

I'll have another post up with those results. There are some additional efforts in the community where performance

has been pushed even further.

 

Hope this helps.

 

Chris

 

On Wed, Nov 6, 2019 at 3:36 AM alok gupta <metech11@...> wrote:

Hello There,

We are conducting a POC for Oil & GAS Retail automation. In which, we are recording all digital/ cash sales onto the Fabric ledger. We monitor the stock levels in the fuel tanks through smart contract. The idea is to replace the current automation system in India which requires massive investment in installation and maintenance. Our app is running successfully on a fuel station at a fuel station in Chandigarh, India. 

My query is about the scalability of fabric over no. of channel, organizations, and peers. Can we scale up our solution to connect the fuel companies ( IOCL. HPCL etc.) with India wide fuel stations? I have seen other use cases like Wallmart food safety where there are running a huge network on blockchain. To move forward in our idea, we need a clarity on scalability Please advise.

Thank you
Alok


--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: Wrong world state #fabric #fabric-questions

David Enyeart
 

We haven't seen such a problem in Fabric system tests (e.g. there are tests that repeatedly crash peers and ensures that peer restart recovers in-flight data with correct data integrity). The occurrences we've seen have been due to things like mounting incorrect database volumes or having data left over from a prior trial. Not that it would be impossible for there to be a bug in the state database code, but nobody has opened such an issue. So if you can reproduce such a problem, please open a bug in Jira with reproduction steps.

If you do get into a bad state, the 'bad' peer won't be able to impact the network... transaction endorsements from a 'bad' peer won't match endorsements from 'good' peers for the afflicted keys, therefore your client application should detect such a problem, and if such a transaction were submitted to the channel anyways, it would be invalidated by all peers due to the mismatched endorsements.


Dave Enyeart

"Joao Antunes" ---11/08/2019 04:43:06 PM---Hi, Recently I got a strange behaviour in my network regarding ledger height, that was not consisten

From: "Joao Antunes" <joao.antunes@...>
To: fabric@...
Date: 11/08/2019 04:43 PM
Subject: [EXTERNAL] [Hyperledger Fabric] Wrong world state #fabric #fabric-questions
Sent by: fabric@...




Hi,

Recently I got a strange behaviour in my network regarding ledger height, that was not consistent.

I solved that issue but, after a bit more investigation, I found that some values in world state were not correct. I used "peer node reset" across all my peers and the value was corrected. So is it correct of me to assume that the ledger is correct but the world state is not? How can this happen?

My setup, is 4 peers, 2 orderers, 2 ca and 2 orgs (2 peers, 1 orderer and 1 ca in each). I'm also using kafka setup. To communicate with the Fabric, I'm using Java SDK.




Re: Hyperledger Fabric Scalability

alok gupta <metech11@...>
 

Hello Mark & Christopher,

Thanks for your reply. I have gone through the blog but I am still apprehensive whether we would be able to support hundreds or thousands of organisations? We did TPS optimisation after struggling a bit with MVCC error, our solution is running fine for a single fuel station. Right now there are max 50 tps.  But not sure how to take this forward. What should be our approach?  Please advise.



On Sat, 9 Nov 2019, 07:59 Mark Rakhmilevich, <mark.rakhmilevich@...> wrote:

Alok,

    We have seen throughput in high hundreds of TPS to low thousands of TPS in Oracle projects.  The scalability and performance depend on many factors well explained in Chris’ blog posts.  Certainly transaction payload size, batching parameters in the ordering service, number of channels an ordering cluster has to support given certain network bandwidth play a significant role. There are also techniques to batch transaction data, which are useful, for example,  to capture an audit log from many IOT sensor readings. Feel free to reach out directly to discuss specifics.

 

Best Regards,

    Mark

 

From: Christopher Ferris <chris.ferris@...>
Sent: Wednesday, November 06, 2019 3:33 AM
To: alok gupta <metech11@...>
Cc: fabric@...
Subject: Re: [Hyperledger Fabric] Hyperledger Fabric Scalability

 

Alok,

 

I wrote a couple of blog posts earlier this year on Hyperledger Fabric performance and scale.

 

There have been some more recent improvements that should improve performance even more when 2.0 ships.

I'll have another post up with those results. There are some additional efforts in the community where performance

has been pushed even further.

 

Hope this helps.

 

Chris

 

On Wed, Nov 6, 2019 at 3:36 AM alok gupta <metech11@...> wrote:

Hello There,

We are conducting a POC for Oil & GAS Retail automation. In which, we are recording all digital/ cash sales onto the Fabric ledger. We monitor the stock levels in the fuel tanks through smart contract. The idea is to replace the current automation system in India which requires massive investment in installation and maintenance. Our app is running successfully on a fuel station at a fuel station in Chandigarh, India. 

My query is about the scalability of fabric over no. of channel, organizations, and peers. Can we scale up our solution to connect the fuel companies ( IOCL. HPCL etc.) with India wide fuel stations? I have seen other use cases like Wallmart food safety where there are running a huge network on blockchain. To move forward in our idea, we need a clarity on scalability Please advise.

Thank you
Alok

Re: Hyperledger Fabric Scalability

Mark Rakhmilevich
 

Alok,

    We have seen throughput in high hundreds of TPS to low thousands of TPS in Oracle projects.  The scalability and performance depend on many factors well explained in Chris’ blog posts.  Certainly transaction payload size, batching parameters in the ordering service, number of channels an ordering cluster has to support given certain network bandwidth play a significant role. There are also techniques to batch transaction data, which are useful, for example,  to capture an audit log from many IOT sensor readings. Feel free to reach out directly to discuss specifics.

 

Best Regards,

    Mark

 

From: Christopher Ferris <chris.ferris@...>
Sent: Wednesday, November 06, 2019 3:33 AM
To: alok gupta <metech11@...>
Cc: fabric@...
Subject: Re: [Hyperledger Fabric] Hyperledger Fabric Scalability

 

Alok,

 

I wrote a couple of blog posts earlier this year on Hyperledger Fabric performance and scale.

 

There have been some more recent improvements that should improve performance even more when 2.0 ships.

I'll have another post up with those results. There are some additional efforts in the community where performance

has been pushed even further.

 

Hope this helps.

 

Chris

 

On Wed, Nov 6, 2019 at 3:36 AM alok gupta <metech11@...> wrote:

Hello There,

We are conducting a POC for Oil & GAS Retail automation. In which, we are recording all digital/ cash sales onto the Fabric ledger. We monitor the stock levels in the fuel tanks through smart contract. The idea is to replace the current automation system in India which requires massive investment in installation and maintenance. Our app is running successfully on a fuel station at a fuel station in Chandigarh, India. 

My query is about the scalability of fabric over no. of channel, organizations, and peers. Can we scale up our solution to connect the fuel companies ( IOCL. HPCL etc.) with India wide fuel stations? I have seen other use cases like Wallmart food safety where there are running a huge network on blockchain. To move forward in our idea, we need a clarity on scalability Please advise.

Thank you
Alok

Wrong world state #fabric #fabric-questions

Joao Antunes
 

Hi,

Recently I got a strange behaviour in my network regarding ledger height, that was not consistent.

I solved that issue but, after a bit more investigation, I found that some values in world state were not correct. I used "peer node reset" across all my peers and the value was corrected. So is it correct of me to assume that the ledger is correct but the world state is not? How can this happen?

My setup, is 4 peers, 2 orderers, 2 ca and 2 orgs (2 peers, 1 orderer and 1 ca in each). I'm also using kafka setup. To communicate with the Fabric, I'm using Java SDK.
Re: Which cert should be copied for TLS( ca-cert.pem or tls-cert.pem) #fabric-ca

Nye Liu <nye@...>
 

From memory (I could be wrong):

If you are using a single instance CA, ca-cert.pem is the root cert for both TLS and non-TLS certs issues by ca-server, including the tls-cert.pem issued to itself, so the clients should only need ca-cert.pem.

tls-cert.pem is the ca-server tls cert for its own endpoint, not a CA cert.

On 11/6/2019 5:43 PM, Jeehoon Lim wrote:

Hi all.

I' m studying the use of fabric CA.

If Fabric CA Server without 'TLS Enabled' option, it generates ca-cert.pem file.
If Fabric CA Server with 'TLS Enabled' option, it generates both ca-cert.pem and tls-cert.pem files.

I thought the tls-cert.pem file should be copied to the fabric clients for use TLS communication with CA Server .

But in the 'Setup TLS Server' section of  operation guide , it says like below :

   you would need to acquire the file located at /tmp/hyperledger/tls/ca/crypto/ca-cert.pem on the machine running the TLS CA server and copy this file over to the host where you will be running the CA client binary.

 
Which cert should be copied for TLS -  ca-cert.pem or tls-cert.pem ?


Regards,
Jeehoon Lim

Re: #raft #fabric #raft #fabric

Prehoda Balazs
 

Hi,

I attached the orderer logs.

Thanks again,
Balazs

On Fri, Nov 8, 2019 at 4:53 AM Jay G <guojiannan1101@...> wrote:
hi Balazs,

could you attach full orderer log so we could diagnose? or pls observe orderer log to see how long it took to elect a leader, some thing similar to "Raft leader changed: 0 -> x"

thx
- J


On Thu, Nov 7, 2019 at 11:03 PM Prehoda Balazs <prehoda.balazs@...> wrote:

Hi,

I tried raft recently, and ran into this strange behavior:

When I invoke peer channel create ..., I always get &{NOT_FOUND} first, then &{SERVICE_UNAVAILABLE} 4 or 5 times. After that, everything is OK, I can join the peers and update the anchor peers successfully. I get the same even if I wait 5-10 minutes before trying to create the channel, or if I reduce the number of both the peers and orderers from 3 to 1. In the orderer logs, I can see 4 or 5 warnings about rejecting deliver request for my CLI's IP because of consenter error.

This behavior does not block me and my project, but I am sure it is not expected, and I would like to know why it works this way, and how I could fix it.

Any help or suggestions would be much appreciated.

Thanks,
Balazs

P.S.:

The script used to create the channel, join peers and update the anchor peers is attached.
The output of the create-join-channel.sh script:


The orderer logs:

?
orderer_logs.txt

Re: #raft #fabric #raft #fabric

Jay Guo
 

hi Balazs,

could you attach full orderer log so we could diagnose? or pls observe orderer log to see how long it took to elect a leader, some thing similar to "Raft leader changed: 0 -> x"

thx
- J


On Thu, Nov 7, 2019 at 11:03 PM Prehoda Balazs <prehoda.balazs@...> wrote:

Hi,

I tried raft recently, and ran into this strange behavior:

When I invoke peer channel create ..., I always get &{NOT_FOUND} first, then &{SERVICE_UNAVAILABLE} 4 or 5 times. After that, everything is OK, I can join the peers and update the anchor peers successfully. I get the same even if I wait 5-10 minutes before trying to create the channel, or if I reduce the number of both the peers and orderers from 3 to 1. In the orderer logs, I can see 4 or 5 warnings about rejecting deliver request for my CLI's IP because of consenter error.

This behavior does not block me and my project, but I am sure it is not expected, and I would like to know why it works this way, and how I could fix it.

Any help or suggestions would be much appreciated.

Thanks,
Balazs

P.S.:

The script used to create the channel, join peers and update the anchor peers is attached.
The output of the create-join-channel.sh script:


The orderer logs:
Unable to debug smart contract from VS Code debugger

Siddharth Jain
 

I am unable to debug my smart contract from vs code debugger. Here is my setup:
  • peer node is started with CORE_CHAINCODE_MODE=dev
  • fabric-chaincode-node is run with --peer-chaincodedev=true
    $ npm start -- --peer.address localhost:17052 --chaincode-id-name asset-contract --peer-chaincodedev=true
  • a vs code debug session is started with Node.js Attach by process ID configuration 
  • debugger is attached to the fabric-chaincode-node process and I am able to set breakpoints which also show up as activated in vs code


  • I can invoke the chaincode from command line but the debugger does not break
anyone knows what is the problem and how to fix it? Using 1.4 of Fabric
Re: Peers with different heights #fabric #database #consensus

David Enyeart
 

In addition to CORE_PEER_GOSSIP_EXTERNALENDPOINT for exposing the peer endpoints to other orgs, make sure you understand how anchor peers are configured to bootstap the cross-org communication:
https://hyperledger-fabric.readthedocs.io/en/release-1.4/gossip.html#anchor-peers
https://hyperledger-fabric.readthedocs.io/en/release-1.4/build_network.html#create-a-channel-configuration-transaction
https://hyperledger-fabric.readthedocs.io/en/release-1.4/build_network.html#update-the-anchor-peers

That being said, cross-org configuration is not strictly needed for block sync... Blocks should be synched correctly from org leader to org followers without the cross-org communication configured. Or like I said, you may want to force each peer to get blocks from orderer:
CORE_PEER_GOSSIP_USELEADERELECTION = false
CORE_PEER_GOSSIP_ORGLEADER = true

The cross-org communication becomes necessary if you are using private data or service discovery. Good luck!


Dave Enyeart

"Joao Antunes" ---11/07/2019 11:57:46 AM---Hi Dave, You are right and thank you for sending the lines of the default behaviour.

From: "Joao Antunes" <joao.antunes@...>
To: fabric@...
Date: 11/07/2019 11:57 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Peers with different heights #fabric #database #consensus
Sent by: fabric@...




Hi Dave,

You are right and thank you for sending the lines of the default behaviour.


After some investigations, I think I'm missing CORE_PEER_GOSSIP_EXTERNALENDPOINT.

I have CORE_PEER_GOSSIP_BOOTSTRAP configured on both peer2 from org2 and org1 and they both communicate with peer1-org2 and peer1-org1 respectively. But there is no gossip between orgs. Currently checking the effect of both variables in this setup.




Re: CA keys and storing/sending them

Gari Singh <garis@...>
 

Private keys are never sent anywhere.  Only public keys are included with transactions.

If you are using the fabric-ca-client or any of the SDKs, by default privates keys are created on the local file system of the host in which enroll.  You can also choose to use the PKCS11 provider to have the private generated and stored in an HSM.

If you do generate it on the local file system, then you should set the permissions to 0400 on *nix based OS’s.  You should also encrypt the file system ( especially when running in a public cloud)

Gari Singh
garis@...
978-846-7499



On Nov 7, 2019, at 11:32 AM, Trevor Lee Oakley <trevor@...> wrote:


 
Hi 
 
I understand that HLF  uses a client server CA and each member has its own CA. But txn approvals I have a question about securely storing and sending keys. Are there any guidelines for this? 
 
Trevor
 

Documentation Workgroup: Agenda for Friday, 08 November

Anthony O'Dowd <a_o-dowd@...>
 

Hello All,

We hold our regular documentation workgroup call this week, both Eastern and Western hemispheres. All US and European clocks have now moved back to standard time from daylight savings, so call times have returned to normal!

As is now usual, you can read the summary minutes for last week's call :https://wiki.hyperledger.org/display/fabric/2019+11+01+DWG+Agenda
Note that we've added a recordings page where you can catch up: https://wiki.hyperledger.org/display/fabric/Recordings  
A particular highlight was Chris Gabriel's demo: https://wiki.hyperledger.org/download/attachments/24773827/2109.11.01.DWG.part2.mp4?api=v2

Please feel free to contribute to the agenda for this week: https://wiki.hyperledger.org/display/fabric/2019+11+08+DWG+Agenda

We've also added an outline agenda for next week's meeting : https://wiki.hyperledger.org/display/fabric/2019+11+15+DWG+Agenda Again, feel free to add items.

Best regards,

Anthony, Pam, Joe, Nik
Meeting Details
-------------
Please use the following link to attend the meeting:  https://zoom.us/j/6223336701

The meeting times are as follows: https://wiki.hyperledger.org/display/fabric/Documentation+Working+Group

Meeting 104A: Friday 01 Nov
                   1130 India Standard Time
                   1400 China Standard Time
                   1500 Japan Standard Time
                   1700 Australia Eastern Time
                   1400 Singapore Time
                   1000 Gulf Standard Time
                   0900 Moscow Standard Time
                   0600 Greenwich Mean Time
                   0700 Central European Time    

Meeting 104B: Friday 01 Nov
              1000 Central Daylight Time
                   1100 Eastern Daylight Time
                   0800 Pacific Daylight Time
                   1200 Brasil Standard Time
                   1600 Greenwich Mean Time
                   1700 Central European Time
                   1800 Moscow Standard Time


Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: Peers with different heights #fabric #database #consensus

Joao Antunes
 

Hi Dave,

You are right and thank you for sending the lines of the default behaviour.


After some investigations, I think I'm missing CORE_PEER_GOSSIP_EXTERNALENDPOINT.

I have CORE_PEER_GOSSIP_BOOTSTRAP configured on both peer2 from org2 and org1 and they both communicate with peer1-org2 and peer1-org1 respectively. But there is no gossip between orgs. Currently checking the effect of both variables in this setup.
Re: Peers with different heights #fabric #database #consensus

David Enyeart
 

'peer node reset' should only be used if you suspect your peer's data is corrupted - it resets all channels to genesis block so that peer can re-pull/re-process blocks, but wouldn't change block dissemination behavior on your network.

If CORE_PEER_GOSSIP_USELEADERELECTION and CORE_PEER_GOSSIP_ORGLEADER environment variable overrides are not set, then it will fall back to the core.yaml configuration that is baked into the peer image, defaults can be found here:
https://github.com/hyperledger/fabric/blob/release-1.4/sampleconfig/core.yaml#L90-L100

Note that the private data reconciliation message is different... that is a background daemon thread that always runs to check whether there is any missing private data. It does not indicate a problem with block height or that there is actual missing private data, it's just checking, and in your case it found no problems.


Dave Enyeart

"Joao Antunes" ---11/07/2019 07:35:10 AM---Hi, Just making a small update.

From: "Joao Antunes" <joao.antunes@...>
To: fabric@...
Date: 11/07/2019 07:35 AM
Subject: [EXTERNAL] Re: [Hyperledger Fabric] Peers with different heights #fabric #database #consensus
Sent by: fabric@...




Hi,

Just making a small update.
I received another answer that suggested to do a peer node reset:
      1. Took a backup of peer docker container
      2. Took a backup of respective couchdb's data
      3. Stopped the chaincode container associated with this peer, if any
      4. Stopped the couchdb container of the peer
      5. Stopped the peer
      6. Since I was starting the peer using docker-compose file, I updated the peer startup command from 'peer node start' to 'peer node reset'. This will reset the peer's channel data to the genesis block.
      7. Next, again update the peer startup command from 'peer node reset' to 'peer node start'. This time since the peer does not have ledger data, it will pull the blocks from the other peers and rebuild its couchdb data.
(Thank you Mrudav Shukla)

Unfortunately, I don't have the 1.4.3 fabric version. So I scraped this solution.
I restart the peer and it's CouchDB. After the startup, the peer started to get the blocks that were missing and getting synchronized.

At the end of this, all the peers were in sync.


On another test that we did on the same setup, now it's peer1-org1 that is out os sync.

I checked docker-compose.yml and I have no CORE_PEER_GOSSIP_USELEADERELECTION and CORE_PEER_GOSSIP_ORGLEADER variables defined. So it's acting by default. Was is the default behaviour? (And thank you, David Enyeart, for the explanation).
I can see some gossip in logs, but peer1-org1 is still out of sync.

For example:

2019-11-07 12:29:08.498 UTC [gossip.privdata] reconcile -> DEBU 65a92e Reconciliation cycle finished successfully. no items to reconcile

(at this stage I know that is still out of sync).


One question that came up; If one peer is out of sync, and we have an OR policy for endorsement, where we require one member from Org1 or Org2 to endorse the transaction, why is there no block created and sent to orderer?

Thank you all.




CA keys and storing/sending them

Trevor Lee Oakley <trevor@...>
 

 
Hi 
 
I understand that HLF  uses a client server CA and each member has its own CA. But txn approvals I have a question about securely storing and sending keys. Are there any guidelines for this? 
 
Trevor
 
4441 - 4460 of 11576