Re: Commit Phase Re-Validation - Phantom Reads
Bharg Pvr
Dear Tomás,
Have you seen this? Please review the following - perhaps you need to craft the solution to the problem slightly differently and side step the implementation challenge. https://github.com/hyperledger/fabric-samples/tree/master/high-throughput Best wishes.
|
|||||||
|
|||||||
Re: Commit Phase Re-Validation - Phantom Reads
Tomás Peixinho
I don't think it's possible in my project (this is my master's thesis, the idea was defined like this by my thesis advisors, who don't know much about blockchain technology).
But I can be more specific. The keys that I am storing are IP prefixes. And the value is some information that I have to store regarding each prefix. However, whenever I receive a new prefix, I don't want to simply insert it. I need to check it against the
ones that are already stored, in order to see if it is a subnet of one that is already in the blockchain (if it is a subnet, instead of a new insertion, I need to update the "supernet" that is already stored, with the new information from this new transaction).
For this reason, I need to query a lot of prefixes, so that I can compare them with the one that I just received.
For example, if I have stored in the blockchain a prefix like "100.0.0.0/24" and I receive two transactions with the prefixes "100.0.0.128/25" and "100.0.0.172/31", both of these prefixes are subnets of the one already stored, so they both need to update it.
But to know this, I need to check octet by octect (in the prefix), in order to find one that matches (if it even exists in the blockchain), hence the multiple queries.
And thank you for the input, by the way!
De: Prasanth Sundaravelu <prasanths96@...>
Enviado: segunda-feira, 8 de fevereiro de 2021 17:37 Para: Tomás Peixinho <tom.peixinho@...> Cc: fabric@... <fabric@...> Assunto: Re: [Hyperledger Fabric] Commit Phase Re-Validation - Phantom Reads Is it possible for you to diversify the range query you would be using?
For example: Let's say there are 3 clients, each client can have a namespace in ledger and are only able to update keys in their relevant spaces.
When performing range query, client namespace can also be added, thereby letting only keys in the particular space be queried.
This way, query results of one client's request will not clash with the other, thereby preventing phantom-reads across clients.
The diversification could be based on any logical categorization you could come up with in your use case. Range query that might return all the keys in the db doesn't sound ideal (hope thats not the case with your app)
On Mon, 8 Feb 2021, 10:50 pm Tomás Peixinho, <tom.peixinho@...> wrote:
|
|||||||
|
|||||||
Re: Commit Phase Re-Validation - Phantom Reads
David Enyeart
There is no ability to turn off the phantom validation. The idea behind range queries in update transactions is that you would scope them to a set of keys that you expect won't be updated concurrently. A classic example is updating all the assets you own with some information. You would do a range query over your assets to get them all, and then update them all in the same transaction. Since nobody else is authorized to update your assets, there typically would be no conflicts and the transaction would be validated and committed. However imagine if somebody transferred an asset to you while the transaction was in flight. You would miss the update on this transaction. The phantom check is designed to catch these types of conflicts. Good afternoon, I'm developing an application using Hyperledger...
I'm developing an application using Hyperledger Fabric that requires me to, in order to do an insertion on the blockchain, check a lot of the stored keys, so that it can know which one it needs to update. For this, I'm using the getStateByPartialCompositeKey function from the Java SDK, and then I'm iterating over the results to find the specific key that needs to be updated (because due to the rules of the application, I can't insert the key as a new transaction, some keys are supposed to be sub-keys of others). The problem with this is that, since there are multiple transactions being sent all the time, the world state is always being updated (between endorsement time and commit validation time), which means that for every transaction that is accepted, there are a ton that are invalidated due to phantom reads. My question is if there is any way around this. Is there any way to disable the commit phase validation? And even if there were, what would happen if two transactions altered the same key at the same time? What would actually be stored in the world state of the blockchain? And is it possible to achieve this using an external CouchDB database? Or would I have the same problem when I performed the queries to find the keys? If possible, would it be ok to disable commit time validation, if I handled the "double-spend" on the client side? And if ultimately there is no way around this, what is the purpose of these ranged type queries? Is it just for printing/showing what is stored? Seems counter-intuitive. This is very problematic for my project (and for the scope of a blockchain in general), because what happens is it almost only inserts transactions sequentially. There is no point in handling transactions concurrently (I'm doing it using threads), because most of them will fail due to phantom reads. I'm really stumped with this, any ideas would be helpful! Thank you very much Tomás
|
|||||||
|
|||||||
Re: Commit Phase Re-Validation - Phantom Reads
Is it possible for you to diversify the range query you would be using? For example: Let's say there are 3 clients, each client can have a namespace in ledger and are only able to update keys in their relevant spaces. When performing range query, client namespace can also be added, thereby letting only keys in the particular space be queried. This way, query results of one client's request will not clash with the other, thereby preventing phantom-reads across clients. The diversification could be based on any logical categorization you could come up with in your use case. Range query that might return all the keys in the db doesn't sound ideal (hope thats not the case with your app)
On Mon, 8 Feb 2021, 10:50 pm Tomás Peixinho, <tom.peixinho@...> wrote:
|
|||||||
|
|||||||
Commit Phase Re-Validation - Phantom Reads
Tomás Peixinho
Good afternoon,
I'm developing an application using Hyperledger Fabric that requires me to, in order to do an insertion on the blockchain, check a lot of the stored keys, so that it can know which one it needs to update. For this, I'm using the getStateByPartialCompositeKey
function from the Java SDK, and then I'm iterating over the results to find the specific key that needs to be updated (because due to the rules of the application, I can't insert the key as a new transaction, some keys are supposed to be sub-keys of others).
The problem with this is that, since there are multiple transactions being sent all the time, the world state is always being updated (between endorsement time and commit validation time), which means that for every transaction that is accepted, there are a
ton that are invalidated due to phantom reads.
My question is if there is any way around this. Is there any way to disable the commit phase validation? And even if there were, what would happen if two transactions altered the same key at the same time? What would actually be stored in the world state of
the blockchain? And is it possible to achieve this using an external CouchDB database? Or would I have the same problem when I performed the queries to find the keys? If possible, would it be ok to disable commit time validation, if I handled the "double-spend"
on the client side? And if ultimately there is no way around this, what is the purpose of these ranged type queries? Is it just for printing/showing what is stored? Seems counter-intuitive.
This is very problematic for my project (and for the scope of a blockchain in general), because what happens is it almost only inserts transactions sequentially. There is no point in handling transactions concurrently (I'm doing it using threads), because most
of them will fail due to phantom reads.
I'm really stumped with this, any ideas would be helpful!
Thank you very much
Tomás
|
|||||||
|
|||||||
Re: Fabric chaincode error when trying to query the database through a partial Composite Key
#fabric-chaincode
#fabric-questions
Matthew White
Thanks - Andy Hurt has replied to the comment on stack overflow.
|
|||||||
|
|||||||
Re: Peer & Orderer Domain Names Visibility
#hyperledger-fabric
#fabric
Samyak Jain | TraceX
Hi Tsvetan,
Thanks for that clarification. In fact, that is our ideology also where we're saying participating organisations joining the network can decide whether or not they want to run their own infra... we're expecting the network will eventually become decentralised
over a period of time gradually as more organizations get ability to manage their own nodes and infra.
Samyak Jain
From: Tsvetan Georgiev <tsvetan@...>
Sent: Saturday, February 6, 2021 10:39:50 PM To: Samyak Jain | TraceX <samyakj@...> Cc: fabric@... <fabric@...> Subject: Re: [Hyperledger Fabric] Peer & Orderer Domain Names Visibility #fabric #hyperledger-fabric
Hi Samyak Jain,
My last point is regarding the technical network layer (not the fabric network). Let's assume there are multiple organizations that each has fabric nodes. From technical networking perspective you may use for example a k8s cluster
to orchestrate all those fabric nodes of all organizations. However that means your k8s cluster becomes the centralized component in your overall network design. That also means whoever controls the k8s cluster has control over all fabric nodes regardless
what organization they belong to. Furthermore if your k8s goes down the whole fabric network (all nodes) goes down - k8s is the single point of failure.
For example the opposite is that every organization has dedicated k8s cluster(s) running its own fabric nodes on their own cloud provider where the fabric nodes connected on a fabric channel communicate on internet over tls. That
case you have no single point of failure.
Regards,
---- On Sat, 06 Feb 2021 00:40:09 -0500 samyakj@... wrote ----
|
|||||||
|
|||||||
Re: Peer & Orderer Domain Names Visibility
#hyperledger-fabric
#fabric
Tsvetan Georgiev
Hi Samyak Jain, My last point is regarding the technical network layer (not the fabric network). Let's assume there are multiple organizations that each has fabric nodes. From technical networking perspective you may use for example a k8s cluster to orchestrate all those fabric nodes of all organizations. However that means your k8s cluster becomes the centralized component in your overall network design. That also means whoever controls the k8s cluster has control over all fabric nodes regardless what organization they belong to. Furthermore if your k8s goes down the whole fabric network (all nodes) goes down - k8s is the single point of failure. For example the opposite is that every organization has dedicated k8s cluster(s) running its own fabric nodes on their own cloud provider where the fabric nodes connected on a fabric channel communicate on internet over tls. That case you have no single point of failure. Regards,
---- On Sat, 06 Feb 2021 00:40:09 -0500 samyakj@... wrote ----
|
|||||||
|
|||||||
Re: Peer & Orderer Domain Names Visibility
#hyperledger-fabric
#fabric
Samyak Jain | TraceX
On Sat, Feb 6, 2021 at 06:06 AM, Tsvetan Georgiev wrote:
Thanks, Samyak Jain
|
|||||||
|
|||||||
Re: Peer & Orderer Domain Names Visibility
#hyperledger-fabric
#fabric
Tsvetan Georgiev
Hi Samyak Jain, I expect many of the networks to be hybrid deployments running on very heterogeneous infrastructure and communication over internet is essential (just make sure TLS is always used!). The nodes may be exposed with their domain names (DNS) so they can resolve each other and communicate over internet. Part of the exercise of course is to design and organize properly your networking (i.e. port mapping, DNS setup, k8s, docker etc) so that your nodes can successfully establish connection regardless where they run (in a k8s pod, docker container, etc). The orderers names are encoded in the channel config block so that when a peer joins a channel it can discover the host names of the orderers and connect to them. For example lets take 2 orgs that host orderers: acme and joy. Assume they each own their own domains acme.com and joy.com. They have as well configured 1 orderer each using subdomains - orderer1.acme.com:7878 , orderer1.joy.com:443. One may run on docker and the other may run on k8s ... essentially that shouldn't matter as their internal networking is abstracted. The peers follow the same principle. The difference is that if a peer needs to be discoverable it has to expose its host name. Furthermore there should be an anchor peer for an org to make that corss org discover between peers possible.. The anchor peers and cross org discovery mechanics are well documented on the community website (check: https://hyperledger-fabric.readthedocs.io/en/latest/gossip.html?#anchor-peers). I honestly don't think it makes much sense to run all peers of all orgs in a consortium on a single network ... That way you lose the decentralization notion of the DLT by introducing a central control on network level.
---- On Fri, 05 Feb 2021 04:27:28 -0500 Samyak Jain | TraceX <samyakj@...> wrote ----
|
|||||||
|
|||||||
Fabric chaincode error when trying to query the database through a partial Composite Key
#fabric-chaincode
#fabric-questions
jonashiltl2003@...
Hi Community,
this is my first time deploying chaincode to a fabric network and I'm fairely new to golang. Right now I'm trying to create a system where a patient can create a permission for a doctor to access the patients files. Just a basic EHR system. This is the struct of the permission asset:
In the process of deploying the chaincode I run a query at the end to test if the chaincode is working but I get this error:
Here is my chaincode file and I use the function "ListDoctorPermissions" to display all permissions given to a doctor by all patients. When creating the permission a composite key is being created, consisting of the patientId the doctorId and the dataCategory. For retrieving all the permissions given to a doctor I use the "GetStateByPartialCompositeKey" function and iterate over the response. At the end I want to append all the permissions to an array and return that array. Here is the addressed function (you can also visit the github link):
I laso published this question on Stackoverflow for clearer code samples: Stackoverflow I hope that my explanation isn't too complicated and thanks for your efforts!
|
|||||||
|
|||||||
Hyperledger Fabric Documentation Workgroup call - Western hemisphere - Fri, 02/05/2021
#cal-notice
fabric@lists.hyperledger.org Calendar <noreply@...>
Hyperledger Fabric Documentation Workgroup call - Western hemisphere When: Where: Organizer: Description: Join Zoom Meeting
https://zoom.us/j/6223336701?pwd=dkJKdHRlc3dNZEdKR1JYdW40R2pDUT09
Meeting ID: 622 333 6701
Passcode: 475869
|
|||||||
|
|||||||
Peer & Orderer Domain Names Visibility
#hyperledger-fabric
#fabric
Samyak Jain | TraceX
Hi Community, Want to ask how the peer and orderer hostnames in Hyperledger Fabric should be mapped across a cluster of nodes on the network? If some organization wants to join a running network, does it mean the hostnames used have to be accessible on the internet in order for the new organization to communicate with the existing network? By default, I presume they are private to the network like orderer0.org1.example.com, so if this needs to be accessed by other org peers not in the same subnet/VPC how does this work? Thanks, Samyak Jain
|
|||||||
|
|||||||
Re: Fw: Re: [Hyperledger Fabric] Kubernetes for Hyperledger Fabric
Hi
Tong Li, Yes, everything is optionally exposed to the outer world. CA's and CouchDB's are exposed via Ingress. Peers and orderers are exposed either via Ingress or Kubernetes service type of LoadBalancer. Please have a look at the diagrams at the "Network Architecture" section and also the "Cross-cluster Raft network" section. Helm chart itself doesn't create the certificates, but depends on certificates being present on the local file system. The init script uses cryptogen to create the certificates, but this is not mandatory. As mentioned in the FAQ, you can create certificates in other means and still use our Helm charts as long as certificates are placed in a folder structure compatible with the cryptogen tool. Best, Hakan
On Thu, Feb 4, 2021 at 5:06 PM Tong Li <litong01@...> wrote:
|
|||||||
|
|||||||
Hyperledger Fabric Documentation Workgroup call - Western hemisphere - Fri, 02/05/2021 11:00am-12:00pm
#cal-reminder
fabric@lists.hyperledger.org Calendar <fabric@...>
Reminder: Hyperledger Fabric Documentation Workgroup call - Western hemisphere When: Friday, 5 February 2021, 11:00am to 12:00pm, (GMT-05:00) America/New York Where:https://zoom.us/my/hyperledger.community.backup?pwd=dkJKdHRlc3dNZEdKR1JYdW40R2pDUT09 Organizer: Pam Andrejko pama@... Description: Documentation workgroup call. Join Zoom Meeting
https://zoom.us/j/6223336701?pwd=dkJKdHRlc3dNZEdKR1JYdW40R2pDUT09
Meeting ID: 622 333 6701
Passcode: 475869
|
|||||||
|
|||||||
Re: Kubernetes for Hyperledger Fabric
Samyak Jain | TraceX
Hi,
Thanks for sharing, will go through this
Samyak Jain
From: hakan eryargi <hakan.eryargi@...>
Sent: Thursday, February 4, 2021 8:13:46 PM To: Samyak Jain | TraceX <samyakj@...> Cc: fabric@... <fabric@...> Subject: Re: [Hyperledger Fabric] Kubernetes for Hyperledger Fabric Hi Samyak,
Have a look at our repository:
Which contains a couple of Helm charts to:
Cheers,
Hakan
On Thu, Feb 4, 2021 at 10:01 AM Samyak Jain | TraceX <samyakj@...> wrote:
|
|||||||
|
|||||||
Re: Kubernetes for Hyperledger Fabric
Hi Samyak, Have a look at our repository: Which contains a couple of Helm charts to:
Cheers, Hakan
On Thu, Feb 4, 2021 at 10:01 AM Samyak Jain | TraceX <samyakj@...> wrote:
|
|||||||
|
|||||||
Re: Kubernetes for Hyperledger Fabric
Hi Samyak,
toggle quoted messageShow quoted text
Kubernetes based automation can be end to end starting from automatically creating a cluster on chosen cloud to deploy services with a version choice 1.4.x or 2.2.x then configuring consensus, creating multiple orgs having any combination of orders, CA and peers under them, furthermore Peers can individually be chosen for couchdb or leveldb. A channel can be also created on the fly for peers to automatically join. a complete Git integrated approach to deploy your dApps on the created network, again automated. Beyond K8 deployments, automated monitoring of resources and services over kubernetes is also available. All this can be achieved using an intuitive platform interface. Do checkout the details at: https://www.zeeve.io/deploy-protocol-hyperledger-fabric/ Regards Ghan
On 04-Feb-2021, at 2:31 PM, Samyak Jain | TraceX <samyakj@...> wrote:
|
|||||||
|
|||||||
Kubernetes for Hyperledger Fabric
Samyak Jain | TraceX
Hi Community, We are trying to explore using Kubernetes for deploying our Fabric network. We want to understand what level of automation will be possible using Kubernetes? Are things like deploying chaincodes, adding a new peer organization, joining existing peer to a new channel, onboarding a new orderer, etc. able to be automated on Kubernetes? Also are there any limitations we should keep in mind for this kind of architecture? Thanks, Samyak Jain
|
|||||||
|
|||||||
ANNOUNCEMENT: Hyperledger Fabric v2.3.1 now available!
David Enyeart
Hyperledger Fabric v2.3.1 is now available.
|
|||||||
|