Feature Request: Harness NG Feature Request: harness webhooks for pipeline execution events
this fiscal quarter
S
Sole Chimpanzee
To integrate efficiently, we would like to be able to register endpoints to be called with Harness events (like how we register webhooks with GitHub to receive GitHub events).
WebHook endpoints will be internal and should be called via delegate (or have the option to be called from the delegate).
Minimum events needed:
Pipeline start/end (include all details in payload)
Stage start/end (include all stage details in payload)
Pipeline/stage interrupt/resume events (pause, approval needed, timeout, abort, reject, approval resume, etc.)
Pipeline created/updated
Solution:
Solution is describe above: webhooks for pipeline events.
Additionally, we would be fine with consuming events delivered to any type of pub/sub bus as well (SNS Topic, Redis pub/sub, SQS queue, etc.).
Having more data on the events would be useful, so we can avoid having to query the API for the key data that make user-friendly notifications and automated actions feasible.
- the name and tags of the pipeline, and the variable inputs of the pipeline execution
- the name of the stage, and the variable inputs of the stage execution
- for failed or rejected pipelines and stages, the name and id of the failed stage and step, and the error or rejection message
If these add on requests can't be accommodated, I would look to using the APIs to shim the feature.
- for #1, calling the audit API at short periodic intervals in order to simulate the events.
- for #2, calling the pipeline execution details API to get the needed values (once for each event. yikes)
Log In
Autopilot
Merged in a post:
NG: Envent Notifications Feedbacks and Features
R
Representative Llama
Hi,
we have identified 3 things we need to have as part of custom notifications.
- We need to have notify event when the pipeline pauses and resumes - parity miss
- We need to have the notification for the step start and complete - new feature
- we need to have the inputs provided to the pipeline and input set name if used in the event data. - new feature
Regards,
Raj Dhamsaniya
Autopilot
Merged in a post:
Reliable delivery of Harness pipeline events
T
Teal Alpaca
In the same vein as Webhook Notifications (https://developer.harness.io/docs/continuous-delivery/x-platform-cd-features/cd-steps/notify-users-of-pipeline-events/#webhook-notifications), we'd like to see a means of reliably delivering pipeline events.
By "reliably delivering", we mean in a way that is insulated from ephemeral errors or issues so that we can rely on and trust the events emitted as a single source of truth -- a "log" of events emitted by each pipeline and execution.
Suggested solution: emit events to a GCP pubsub topic which we can then consume and use for our own purposes (a pull model vs the existing push). In order to ensure viable usage as a single source of truth it must also either: a) maintain reliability constraints within Harness itself (retrying internally as necessary to emit events), and/or b) fail pipelines when the appropriate event can't be emitted.
The idea is for the minimum contract for such events to be, essentially, the event corresponding to a given stage/step/pipeline emits or the pipeline fails and no further steps/stages/etc will be run. (Context: this emulates the current behavior we manually inject into our pipelines -- manual effort which would no longer be required with such a solution).
The goal is to avoid tracking the state of pipeline executions ourselves (by all accounts, this should be reasonably characterized as Harness' responsibility) and hitting the Harness API for every pipeline event -- or at set intervals when event emissions are presumed to be missed/failed -- while insulating us from trying to manage issues concerning webhooks, such as 1) endpoint availability/uptime, 2) failed message parsing due to schema evolution (one example), 3) upstream issues within Harness related to sending these (retries + backoffs, e.g.).
In summary: a pull-based model for these events would suit our needs far better than any push-and-check-based model which is what currently manifests.
Autopilot
Merged in a post:
Harness Event publishing model - JPM AWM Ask
S
Scientific Silverfish
We should look for a decoupled / pub-sub (data product) type design for sending Notification (Kafka Topic, Messaging broker)
The event payload schema need to be enhanced to ensure that there is the ability to publish the details from stages/processes needed to fully evidence and troubleshoot the processes. All pipeline meta data including things like policy version, evaluation & results , task and error details; etc.)
Autopilot
Merged in a post:
Integration with Chronosphere Change Events
G
Graceful Marmoset
For Observability, we are using the Chronosphere platform.
They provide Change Events for tracking various events (deployment, configuration, etc.) and correlating it with metric dashboards and alerts. https://docs.chronosphere.io/investigate/change-events
They provide several native integrations for vendors already (e.g. Github) but do not have a Harness integration:
They also provide a Change Events API.
Ideally, there would be some native integration available so that we could easily instrument our CD pipelines for deployment events. Either a Drone plugin that can be included into pipelines, or some webhook-based integration that maps webhook payloads to the event payload accepted by Chronosphere.
Thank you!
Autopilot
Merged in a post:
Pipeline execute events
M
Moonstone Frog
- Currently Harness supports configuration where use can configure notification for each pipeline execution and send the events to different methods (that include webhook(url), email etc)
- We need similar functionality for all pipeline execution in harness, so that pipeline execution events should be pass to centralized (JET) service , where we will process the events and use for evidencing propose.
- Similar kind of functionality is there in Spinnaker that you can use as reference.
- Spinnaker has a micro service named echo that can configure with external url to send CD events.
- https://spinnaker.io/docs/setup/other_config/features/notifications/ please check
C
Crimson Swordtail
Looking for this functionality to be able to integrate Harness pipelines with JET instead of making api calls at the end of every stage.
N
Nickel Reindeer
I use these webhook notifications, and one piece of data that would be very useful to get added for stage events is the the nodeId or nodeExecutionId of the stage. What's is sent now is just a stage identifier, so I am able to fetch the execution from the API and either get the ENTIRE execution graph, or get no execution graph but find the stage in the layout map, and then call again to get the graph for that stage node. If the stage node Id were on the event, I could make one single call to get just the relevant data. https://apidocs.harness.io/tag/Pipeline-Execution-Details#operation/getExecutionDetailV2
G
Graceful Marmoset
Turns out the Drone Plugin approach didn't work due to other limitations of Harness:
- Support Drone plugins in CD Stages: https://ideas.harness.io/continuous-delivery/p/support-drone-plugins-in-cd-stages
- Support distroless images in Containerized Steps : https://ideas.harness.io/feature-request/p/support-distroless-images-within-containerized-steps
I managed to use HTTP Step template to add steps within my pipeline to publish change events to the API.
However, it would be MUCH cleaner to have a Notification provider instead that could be configured on the stages of my pipeline rather than having to add inline steps within the pipeline for publishing the events (as it clutters up the main flow).
Ideally, something like the HTTP Step could be configured as a custom Notification provider where the notification event would have runtime inputs available and you could customize the webhook request (URL and payload) accordingly. That would give the flexibility we need without requiring Chronosphere to implement support for Harness's webhook event payloads.
Rohan Gupta
Hi Graceful Marmoset someone from our product team will reach out to you on this. Thank you for your efforts and contribution.
G
Graceful Marmoset
I implemented a simple golang module and Drone plugin:
We could collaborate on development to make it more feature complete (and tested!).
Load More
→