Can chaincode make an external api call to an external system to enrich the data source before endorsing a transaction?


Nathan Aw
 

Hi all,

Two quick questions. Thank you.

1. Can chaincode make an external api call to an external system to enrich the transaction before endorsing a transaction thereby committing to the ledger?

2. Can chaincode make an external api call to the internet or another system to notify that the transaction has been endorsed?

Thank you!

Nathan Aw


Olivier Bernin
 

Hi,

It is technically possible, but there may be unintended consequences.

All endorsing peers will be making the same call - will it be possible for them ? ie will the external system be available to them - credentials, firewalls, etc ...
If one of them cannot make it or receive a different result, your transaction will likely be rejected.

A more blockchain-y approach may be to push the data you want to use to enrich your transaction on the Blockchain first, and have your chaincode pick it up from there.

Olivier Bernin

Nathan Aw ---09/04/2018 05:09:03---Hi all, Two quick questions. Thank you.

From: Nathan Aw <nathan.mk.aw@...>
To: hyperledger-technical-discuss@..., hyperledger-fabric@...
Date: 09/04/2018 05:09
Subject: [Hyperledger-fabric] Can chaincode make an external api call to an external system to enrich the data source before endorsing a transaction?
Sent by: hyperledger-fabric-bounces@...





Hi all,

Two quick questions. Thank you.

1. Can chaincode make an external api call to an external system to enrich the transaction before endorsing a transaction thereby committing to the ledger?

2. Can chaincode make an external api call to the internet or another system to notify that the transaction has been endorsed?

Thank you!

Nathan Aw_______________________________________________
Hyperledger-fabric mailing list
Hyperledger-fabric@...
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.hyperledger.org_mailman_listinfo_hyperledger-2Dfabric&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=1CAYP3S27sl8wDeYjKLWBhSE8XAf5xzCUxr7v7cCLso&m=teBffiYvvVrC6tHQWT2XZdvA7-rXC9wDYZHpDZP-JPk&s=1CPcPScyErrqTCqtbjP4iEXkVFngzs5vM09H143c5jQ&e=




Kim Letkeman <kletkema@...>
 

My opinions ...

While this is technically possible, it is a risky idea for several reasons:

1) chaincode must be deterministic, and the problem with 'enriching' a transaction using an external API is that it must return the same result running anywhere in a business network, which may very well be running globally, so you need to trust that the answers will all be the same within a time windows that is quite a bit wider than a few ms
2) running just one endorser in development *and* production gets you around that problem, but weakens consensus a bit, and makes it essentially impossible to prove determinism for any given transaction
3) designing to such a weakened system is not a good idea, since inevitably someone will realize that the endorsement policy should be stronger and you go right back to the issues in point 1

One way around this issue is to use a distributed external API with versioned data (and you might need to write an oracle to provide this facility on top of an API that is not versioning its data) such that all endorsers store the external data's current version in the asset repository in world state as well. This makes certain that the data read is identical and accounts for delays in propagation in the oracle network. The presence of the API data version in the final asset data in world state (more accurately in the read/write set for the transaction) ensures that different versions of data in different regions in the oracle (e.g. propagation delays) will fail any multi-endorsement policy. Of course, a client designed in such an environment is free to resubmit a transaction for endorsement to get consensus.

4) external notification should never be done from the endorsement code unless using the approved mechanism of emitting an event from the transaction code in the smart contract. This because a successfully endorsed transaction does not necessarily get committed if it happened to execute in parallel with another transaction that read the same keys (e.g. parallel updates to the same asset). In that case, a direct notification says all is well and the update never actually happens in world state.

So you just need to listen for these events, perhaps in some form of oracle if you like, and then propagate to whoever needs to know that the transaction executed successfully *and was committed to the blockchain*.

Apps can listen directly of course, but the event hub is not necessarily guaranteed deliver and it certainly does not deliver if the connection is interrupted, so you need a reliable message service listening for the events and propagating them to apps that are not always connected (e.g. mobile).

And it is even possible to crawl the blocks and store all emitted events into a service that provides the propagation, thereby taking the event hub out of the loop. (Slightly slower, but theoretically more reliable.) I would probably listen to the event hub for block creation and poll in the background every so often to make sure an event was not missed / lost.

My preference for talking to external systems is that it is best done through applications and integrations that push info in through transactions and take info out through events.

Kim


Kim Letkeman
Senior Technical Staff Member, IBM Watson IoT

IoT Blockchain


Phone: +1 (613) 762-0352
E-mail:
kletkema@...


Nathan Aw ---04/09/2018 12:09:02 AM---Hi all, Two quick questions. Thank you.

From: Nathan Aw <nathan.mk.aw@...>
To: hyperledger-technical-discuss@..., hyperledger-fabric@...
Date: 04/09/2018 12:09 AM
Subject: [Hyperledger-fabric] Can chaincode make an external api call to an external system to enrich the data source before endorsing a transaction?
Sent by: hyperledger-fabric-bounces@...





Hi all,

Two quick questions. Thank you.

1. Can chaincode make an external api call to an external system to enrich the transaction before endorsing a transaction thereby committing to the ledger?

2. Can chaincode make an external api call to the internet or another system to notify that the transaction has been endorsed?

Thank you!

Nathan Aw
_______________________________________________
Hyperledger-fabric mailing list
Hyperledger-fabric@...
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.hyperledger.org_mailman_listinfo_hyperledger-2Dfabric&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=u4kPaQW3J1JDN4jq-BW7ucxX9uJfnfTE0MgE87ReAOY&m=r7d9QIo22MPneWNbWFg6jif8eOh3rYu9_lKvKyqVS5o&s=ucWtaEUpHnjkFZTDAHT9jFCF-ZnYeUvSzXIN0fS5iCw&e=




Nathan Aw
 

Very amazing and insightful answers. Thank you kim and olivia!

Nathan Aw

On Mon, 9 Apr 2018, 22:25 Kim Letkeman, <kletkema@...> wrote:

My opinions ...

While this is technically possible, it is a risky idea for several reasons:

1) chaincode must be deterministic, and the problem with 'enriching' a transaction using an external API is that it must return the same result running anywhere in a business network, which may very well be running globally, so you need to trust that the answers will all be the same within a time windows that is quite a bit wider than a few ms
2) running just one endorser in development *and* production gets you around that problem, but weakens consensus a bit, and makes it essentially impossible to prove determinism for any given transaction
3) designing to such a weakened system is not a good idea, since inevitably someone will realize that the endorsement policy should be stronger and you go right back to the issues in point 1

One way around this issue is to use a distributed external API with versioned data (and you might need to write an oracle to provide this facility on top of an API that is not versioning its data) such that all endorsers store the external data's current version in the asset repository in world state as well. This makes certain that the data read is identical and accounts for delays in propagation in the oracle network. The presence of the API data version in the final asset data in world state (more accurately in the read/write set for the transaction) ensures that different versions of data in different regions in the oracle (e.g. propagation delays) will fail any multi-endorsement policy. Of course, a client designed in such an environment is free to resubmit a transaction for endorsement to get consensus.

4) external notification should never be done from the endorsement code unless using the approved mechanism of emitting an event from the transaction code in the smart contract. This because a successfully endorsed transaction does not necessarily get committed if it happened to execute in parallel with another transaction that read the same keys (e.g. parallel updates to the same asset). In that case, a direct notification says all is well and the update never actually happens in world state.

So you just need to listen for these events, perhaps in some form of oracle if you like, and then propagate to whoever needs to know that the transaction executed successfully *and was committed to the blockchain*.

Apps can listen directly of course, but the event hub is not necessarily guaranteed deliver and it certainly does not deliver if the connection is interrupted, so you need a reliable message service listening for the events and propagating them to apps that are not always connected (e.g. mobile).

And it is even possible to crawl the blocks and store all emitted events into a service that provides the propagation, thereby taking the event hub out of the loop. (Slightly slower, but theoretically more reliable.) I would probably listen to the event hub for block creation and poll in the background every so often to make sure an event was not missed / lost.

My preference for talking to external systems is that it is best done through applications and integrations that push info in through transactions and take info out through events.

Kim


Kim Letkeman
Senior Technical Staff Member, IBM Watson IoT

IoT Blockchain


Phone: +1 (613) 762-0352
E-mail:
kletkema@...


Nathan Aw ---04/09/2018 12:09:02 AM---Hi all, Two quick questions. Thank you.

From: Nathan Aw <nathan.mk.aw@...>
To: hyperledger-technical-discuss@..., hyperledger-fabric@...
Date: 04/09/2018 12:09 AM
Subject: [Hyperledger-fabric] Can chaincode make an external api call to an external system to enrich the data source before endorsing a transaction?
Sent by: hyperledger-fabric-bounces@...




Hi all,

Two quick questions. Thank you.

1. Can chaincode make an external api call to an external system to enrich the transaction before endorsing a transaction thereby committing to the ledger?

2. Can chaincode make an external api call to the internet or another system to notify that the transaction has been endorsed?

Thank you!

Nathan Aw
_______________________________________________
Hyperledger-fabric mailing list
Hyperledger-fabric@...
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.hyperledger.org_mailman_listinfo_hyperledger-2Dfabric&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=u4kPaQW3J1JDN4jq-BW7ucxX9uJfnfTE0MgE87ReAOY&m=r7d9QIo22MPneWNbWFg6jif8eOh3rYu9_lKvKyqVS5o&s=ucWtaEUpHnjkFZTDAHT9jFCF-ZnYeUvSzXIN0fS5iCw&e=