ben/webhook-matrix-notifier
1
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Update to Flask 2 and matrix-nio

Browse source 
Use asynchronously running code.
Rewrite error handling and all api interactions.
Add config.yml.example and more configuration options.
Improve README.md.
Update licenses in all files.
This commit is contained in:
Ben 2021-06-28 09:56:48 +02:00
parent 9927a67a86
commit 3b50125d54
Signed by: ben
GPG key ID: 0F54A7ED232D3319
9 changed files with 382 additions and 218 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.idea/misc.xml
View file

@ -3,7 +3,7 @@
<component name="JavaScriptSettings">
<option name="languageLevel" value="ES6" />
</component>
<component name="ProjectRootManager" version="2" project-jdk-name="Python 3.8 (webhook-matrix-notifier)" project-jdk-type="Python SDK" />
<component name="ProjectRootManager" version="2" project-jdk-name="Python 3.9 (webhook-matrix-notifier)" project-jdk-type="Python SDK" />
<component name="PythonCompatibilityInspectionAdvertiser">
<option name="version" value="3" />
</component>

2
.idea/webhook-matrix-notifier.iml
View file

@ -4,7 +4,7 @@
<content url="file://$MODULE_DIR$">
<excludeFolder url="file://$MODULE_DIR$/venv" />
</content>
<orderEntry type="inheritedJdk" />
<orderEntry type="jdk" jdkName="Python 3.9 (webhook-matrix-notifier)" jdkType="Python SDK" />
<orderEntry type="sourceFolder" forTests="false" />
</component>
</module>

2
Dockerfile
View file

@ -1,6 +1,6 @@
FROM alpine
RUN apk add --no-cache uwsgi-python3 python3
RUN apk add --no-cache uwsgi-python3 python3 py3-yaml py3-flask py3-matrix-nio
# partly from https://hub.docker.com/_/python?tab=description#create-a-dockerfile-in-your-python-app-project
WORKDIR /usr/src/wmn

2
LICENSE
View file

@ -1,4 +1,4 @@
Copyright 2019 Benedikt Ziemons
Copyright 2019-2021 Benedikt Ziemons
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in

39
README.md
View file

@ -1,14 +1,31 @@
# Webhook Matrix Notifier
Takes notifications via webhook, checks a secret and notifies a
[Matrix](https://matrix.org) channel. Listens to HTTP only. Should be used
behind a reverse-proxy with HTTPS.
Takes notifications via webhook, checks a secret and notifies a [Matrix](https://matrix.org) channel.
Listens to HTTP only. Should be used behind a reverse-proxy with HTTPS.
# Testing the Hook locally
- Start the webserver locally by `env FLASK_APP=wmn.py flask run`
- Or have your IDE do it for you
- Send a POST request using curl `curl -i -X POST "localhost:5000/matrix?channel=%21yhEUnvhAZZFKRStdXb%3Amatrix.org" -H "X-Gitlab-Event: Push Hook" -H "X-Gitlab-Token: ..." -H "Content-Type: application/json" --data-binary @./testrequest_gitlab.json`
- The part after `channel=` is the room ID which can retrieved from Matrix channels you are part of
- `%21` escapes ! in URI
- `%3A` escapes : in URI
- The `X-Gitlab-Token` must correspond to the one provided in `config.yaml`
An example configuration is at `config.yml.example` and the program always reads the configuration file `config.yml`.
## Testing the Hook locally
First, start the webserver locally by `env FLASK_APP=wmn.py flask run` or have your IDE start it for you. \
Then, send a POST request using curl.
### GitLab
```
export CHANNEL_ENC=`python3 -c 'from urllib.parse import quote_plus; print(quote_plus("#channel:matrix.org"))'`
curl -i -X POST "http://localhost:5000/matrix?channel=${CHANNEL_ENC}" -H "X-Gitlab-Event: Push Hook" -H "X-Gitlab-Token: 123" -H "Content-Type: application/json" --data-binary @./testrequest_gitlab.json
```
The `X-Gitlab-Token` must correspond to the secret provided in `config.yml`
### Prometheus
```
export CHANNEL_ENC=`python3 -c 'from urllib.parse import quote_plus; print(quote_plus("#channel:matrix.org"))'`
export WMN_SECRET=`python3 -c 'from urllib.parse import quote_plus; print(quote_plus("123"))'`
curl -i -X POST "http://localhost:5000/matrix?type=prometheus&secret=${WMN_SECRET}&channel=${CHANNEL_ENC}" -H "Content-Type: application/json" --data-binary @./testrequest_prometheus.json
```
The secret must be passed as a URI parameter here.

15
config.yml.example Normal file
View file

@ -0,0 +1,15 @@
# secret for other services to interact with this webhook
secret: "…"
matrix:
server: https://matrix.org
username: …
# optional
store_path: …
device_name: …
device_id: … # will be filled automatically otherwise
# use password or token
password: "…"
token: …

59
notify.py
View file

@ -1,32 +1,39 @@
#!/usr/bin/env python3
# Copyright 2019-2021 Benedikt Ziemons
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the "Software"), to deal in
# the Software without restriction, including without limitation the rights to
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
# the Software, and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
# IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
import argparse
import asyncio
import re
import sys
import nio
import yaml
from matrix_client.client import MatrixClient
from wmn import client_login, resolve_room, send_message
# Not going to care for specifics like the underscore.
# Generally match !anything:example.com with unicode support.
room_pattern = re.compile(r"^!\w+:[\w\-.]+$")
def send_message(cfg, args):
client = MatrixClient(cfg["matrix"]["server"])
client.login(username=cfg["matrix"]["username"], password=cfg["matrix"]["password"])
room = client.join_room(room_id_or_alias=args.channel)
if "html" in args:
body = None if len(args.text) == 0 else str(args.text)
room.send_html(html=args.html, body=body, msgtype=args.type)
else:
room.client.api.send_message(
room_id=room.room_id, text_content=args.text, msgtype=args.type
)
def main():
async def main():
"""
config.yml Example:
@ -60,9 +67,23 @@ def main():
print("ERROR: Couldn't parse channel as a matrix channel", file=sys.stderr)
sys.exit(1)
send_message(cfg, args)
print("Message sent.", file=sys.stderr)
client = await client_login(cfg)
try:
room_id = await resolve_room(client=client, room=args.channel)
response = await client.join(room_id=room_id)
if isinstance(response, nio.ErrorResponse):
raise response
if "html" in args:
response = await send_message(client=client, room_id=room_id, text=(args.text or ""), msgtype=args.type, html=args.html)
else:
response = await send_message(client=client, room_id=room_id, text=args.text, msgtype=args.type)
print("Message sent.", file=sys.stderr, flush=True)
finally:
await client.close()
print(response.event_id)
if __name__ == "__main__":
main()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

4
requirements.txt
View file

@ -1,3 +1,3 @@
PyYAML>=5.1,<6
Flask>=1.1.1,<2
matrix-client>=0.3.2,<0.4
flask[async]>=2.0.0,<3
matrix-nio[e2e]>=0.18.0,<1

351
wmn.py
View file

@ -1,14 +1,37 @@
# Copyright 2019-2021 Benedikt Ziemons
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the "Software"), to deal in
# the Software without restriction, including without limitation the rights to
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
# the Software, and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
# IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
import json
import re
import sys
import traceback
import typing
from datetime import datetime
from typing import Tuple, Optional, Dict, Any
import nio
import yaml
from flask import Flask, request, abort
from matrix_client.client import MatrixClient
from matrix_client.errors import MatrixRequestError
from werkzeug.datastructures import MultiDict
Cfg = Dict[str, Any]
ErrorResponse = Tuple[str, int]
RequestArgs = MultiDict[str, str]
app = Flask(__name__)
application = app
@ -17,28 +40,25 @@ application = app
# Generally match room alias or id [!#]anything:example.com with unicode support.
room_pattern = re.compile(r"^[!#]\w+:[\w\-.]+$")
# prometheus has to many sub-second digits in their timestamp,
# older prometheus/alertmanager versions send too many sub-second digits in their timestamp,
# so we get rid of nanoseconds here
promtime_to_isotime_pattern = re.compile(
r"([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2})(\.[0-9]{6})?(?:[0-9]{3})?(Z|[+-][0-9]{2}:[0-9]{2})"
)
"""
config.yml Example:
secret: "..."
matrix:
server: https://matrix.org
username: ...
password: "..."
"""
def load_configuration() -> Cfg:
with open("config.yml", "r") as ymlfile:
cfg = yaml.safe_load(ymlfile)
return yaml.safe_load(ymlfile)
def check_token(header_field: str):
token = request.headers.get(header_field)
if token != cfg["secret"]:
def save_configuration(configuration: Cfg):
with open("config.yml", "w") as ymlfile:
yaml.safe_dump(configuration, ymlfile)
def check_token(configuration: Cfg, token: str):
if token != configuration["secret"]:
print(
"check_token failed, because token did not match",
file=sys.stderr,
@ -47,15 +67,39 @@ def check_token(header_field: str):
abort(401)
def get_a_room():
if "channel" not in request.args:
async def resolve_room(client: nio.AsyncClient, room: str) -> str:
"""Takes a room alias or room id and always returns a resolved room id."""
if room.startswith("#"):
response = await client.room_resolve_alias(room_alias=room)
if isinstance(response, nio.ErrorResponse):
abort(app.make_response(matrix_error(response)))
return response.room_id
elif room.startswith("!"):
return room
else:
raise RuntimeError(f"Room {room} could not be resolved")
async def get_a_room(client: nio.AsyncClient, request_args: RequestArgs) -> str:
"""Takes a nio.AsyncClient and the request args to return a room id."""
if "channel" not in request_args:
print(
"get_a_room failed, because channel was not in request args",
file=sys.stderr,
flush=True,
)
abort(400)
room = request.args.get("channel")
room = request_args.get("channel")
if not room:
print(
"get_a_room failed, because channel was empty",
file=sys.stderr,
flush=True,
)
abort(400)
# sanitize input
if room_pattern.fullmatch(room) is None:
print(
@ -67,13 +111,14 @@ def get_a_room():
flush=True,
)
abort(400)
return room
return await resolve_room(client=client, room=room)
def get_msg_type():
if "msgtype" not in request.args:
def get_msg_type(request_args: RequestArgs):
if "msgtype" not in request_args:
return "m.notice"
msgtype = request.args.get("msgtype")
msgtype = request_args.get("msgtype")
if msgtype in ["m.text", "m.notice"]:
return msgtype
else:
@ -102,33 +147,83 @@ def shorten(string: str, max_len: int = 80, appendix: str = "..."):
return string
def matrix_error(error: MatrixRequestError):
print("matrix_error was called with", error, file=sys.stderr)
traceback.print_exception(MatrixRequestError, error, error.__traceback__)
print(file=sys.stderr, flush=True)
async def client_login(configuration: Cfg) -> nio.AsyncClient:
client = nio.AsyncClient(
homeserver=configuration["matrix"].get("server"),
user=configuration["matrix"].get("username", ""),
device_id=configuration["matrix"].get("device_id", ""),
store_path=configuration["matrix"].get("store_path", ""),
)
response = await client.login(
password=configuration["matrix"].get("password", None),
device_name=configuration["matrix"].get("device_name", ""),
token=configuration["matrix"].get("token", None),
)
if isinstance(response, nio.ErrorResponse):
raise response
if "device_id" not in configuration["matrix"]:
configuration["matrix"]["device_id"] = response.device_id
save_configuration(configuration)
return client
async def send_message(
client: nio.AsyncClient,
room_id: str,
text: str,
msgtype: str = "m.text",
html: Optional[str] = None
) -> nio.RoomSendResponse:
content = {
"body": text,
"msgtype": msgtype,
}
if html is not None:
content["format"] = "org.matrix.custom.html"
content["formatted_body"] = html
response = await client.room_send(
room_id=room_id,
message_type="m.room.message",
content=content,
ignore_unverified_devices=True,
)
if isinstance(response, nio.ErrorResponse):
raise response
return response
def matrix_error(error: nio.ErrorResponse) -> ErrorResponse:
print("matrix_error was called with", error, file=sys.stderr, flush=True)
# see Flask.make_response, this will be interpreted as (body, status)
return f"Error from Matrix: {error.content}", error.code
if error.status_code:
status = int(error.status_code)
else:
status = 500
return f"Error from Matrix: {error.message}", status
def process_gitlab_request():
check_token("X-Gitlab-Token")
msgtype = get_msg_type()
room = get_a_room()
async def process_gitlab_request():
cfg = load_configuration()
check_token(configuration=cfg, token=request.headers.get("X-Gitlab-Token"))
gitlab_event = request.headers.get("X-Gitlab-Event")
try:
client = await client_login(cfg)
except nio.ErrorResponse as response:
return matrix_error(response)
try:
room_id = await get_a_room(client, request.args)
msgtype = get_msg_type(request_args=request.args)
if gitlab_event == "Push Hook":
if request.json["total_commits_count"] < 1:
return "", 204
try:
client = MatrixClient(cfg["matrix"]["server"])
client.login(
username=cfg["matrix"]["username"], password=cfg["matrix"]["password"]
)
room = client.join_room(room_id_or_alias=room)
except MatrixRequestError as e:
return matrix_error(e)
response = await client.join(room_id=room_id)
if isinstance(response, nio.ErrorResponse):
return matrix_error(response)
def sort_commits_by_time(commits):
return sorted(commits, key=lambda commit: commit["timestamp"])
@ -159,37 +254,46 @@ def process_gitlab_request():
text_commits = "\n".join(
(f"- [{msg}]({url})" for (msg, url) in commit_messages)
)
try:
room.send_html(
f"<strong>{username} pushed {len(commit_messages)} commits{to_str}</strong><br>\n"
response = await client.room_send(
room_id=room_id,
message_type="m.room.message",
content={
"msgtype": msgtype,
"format": "org.matrix.custom.html",
"formatted_body": f"<strong>{username} pushed {len(commit_messages)} commits{to_str}</strong><br>\n"
f"<ul>\n{html_commits}\n</ul>\n",
body=f"{username} pushed {len(commit_messages)} commits{to_str}\n{text_commits}\n",
msgtype=msgtype,
"body": f"{username} pushed {len(commit_messages)} commits{to_str}\n{text_commits}\n",
},
ignore_unverified_devices=True,
)
except MatrixRequestError as e:
return matrix_error(e)
if isinstance(response, nio.ErrorResponse):
return matrix_error(response)
except nio.ErrorResponse as response:
abort(app.make_response(matrix_error(response)))
finally:
await client.close()
# see Flask.make_response, this is interpreted as (body, status)
return "", 204
def process_jenkins_request():
check_token("X-Jenkins-Token")
msgtype = get_msg_type()
room = get_a_room()
async def process_jenkins_request():
cfg = load_configuration()
check_token(configuration=cfg, token=request.headers.get("X-Jenkins-Token"))
msgtype = get_msg_type(request_args=request.args)
try:
client = await client_login(cfg)
except nio.ErrorResponse as response:
return matrix_error(response)
try:
room_id = await get_a_room(client, request.args)
jenkins_event = request.headers.get("X-Jenkins-Event")
if jenkins_event == "Post Build Hook":
try:
client = MatrixClient(cfg["matrix"]["server"])
client.login(
username=cfg["matrix"]["username"], password=cfg["matrix"]["password"]
)
room = client.join_room(room_id_or_alias=room)
except MatrixRequestError as e:
return matrix_error(e)
project_url = request.json["githubProjectUrl"]
def extract_change_message(change):
@ -223,67 +327,60 @@ def process_jenkins_request():
else:
text_change_messages, html_change_messages = (), () # it's an owl!
newline = "\n"
try:
room.send_html(
f"<p><strong>Build {build_name} on project {project_name} complete: "
f'<font color="{result_color}">{result_type}</font></strong>, '
f"{len(changes)} commits</p>\n"
""
+ (
newline = "\n" # expressions inside f-strings cannot contain backslashes...
html_changes = (
f"<ul>\n{newline.join(html_change_messages)}\n</ul>\n"
if len(html_change_messages) > 0
else ""
),
body=f"**Build {build_name} on project {project_name} complete: {result_type}**, "
f"{len(changes)} commits\n"
""
+ (
)
text_changes = (
f"{newline.join(text_change_messages)}\n"
if len(text_change_messages) > 0
else ""
)
await send_message(
client=client,
room_id=room_id,
text=(
f"**Build {build_name} on project {project_name} complete: {result_type}**, "
f"{len(changes)} commits\n"
f"{text_changes}"
),
msgtype=msgtype,
html=(
f"<p><strong>Build {build_name} on project {project_name} complete: "
f'<font color="{result_color}">{result_type}</font></strong>, '
f"{len(changes)} commits</p>\n"
f"{html_changes}"
),
)
except MatrixRequestError as e:
return matrix_error(e)
except nio.ErrorResponse as response:
abort(app.make_response(matrix_error(response)))
finally:
await client.close()
# see Flask.make_response, this is interpreted as (body, status)
return "", 204
def process_prometheus_request():
secret = request.args.get("secret")
if secret != cfg["secret"]:
print(
"check_token failed, because token did not match",
file=sys.stderr,
flush=True,
)
abort(401)
msgtype = get_msg_type()
room = get_a_room()
if not request.json:
abort(400)
async def process_prometheus_request():
# written for version 4 of the alertmanager webhook JSON
# https://prometheus.io/docs/alerting/configuration/#webhook_config
def color_status_html(status: str, text: typing.Optional[str] = None):
def color_status_html(status: str, text: Optional[str] = None):
_status_colors = {"resolved": "34A91D", "firing": "EF2929"}
if text is None:
text = status
return color_format_html(_status_colors.get(status, "FFFFFF"), text)
def color_severity_html(severity: str, text: typing.Optional[str] = None):
def color_severity_html(severity: str, text: Optional[str] = None):
_severity_colors = {"warning": "EFAC29", "critical": "EF2929"}
if text is None:
text = severity
return color_format_html(_severity_colors.get(severity, "FFFFFF"), text)
def parse_promtime(date_string):
def parse_promtime(date_string) -> datetime:
match = promtime_to_isotime_pattern.match(date_string)
if match is None:
# weirdly enough, they switched to ISO primetime
@ -320,9 +417,7 @@ def process_prometheus_request():
return title, html_title
def extract_alert_message(
alert: typing.Dict[str, typing.Any]
) -> typing.Tuple[str, str]:
def extract_alert_message(alert: Dict[str, Any]) -> Tuple[str, str]:
"""Takes the alert object and returns (text, html) as a string tuple."""
labels = alert.get("labels", {})
@ -367,40 +462,56 @@ def process_prometheus_request():
html_message,
)
try:
client = MatrixClient(cfg["matrix"]["server"])
client.login(
username=cfg["matrix"]["username"], password=cfg["matrix"]["password"]
cfg = load_configuration()
secret = request.args.get("secret")
if secret != cfg["secret"]:
print(
"check_token failed, because token did not match",
file=sys.stderr,
flush=True,
)
room = client.join_room(room_id_or_alias=room)
abort(401)
try:
for body, html in map(
extract_alert_message, request.json.get("alerts", [])
):
if html and body:
room.send_html(html=html, body=body, msgtype=msgtype)
elif body:
room.send_text(body)
client = await client_login(cfg)
except nio.ErrorResponse as response:
return matrix_error(response)
try:
msgtype = get_msg_type(request_args=request.args)
room_id = await get_a_room(client=client, request_args=request.args)
if not request.json:
abort(400)
try:
for text, html in map(extract_alert_message, request.json.get("alerts", [])):
if html and text:
await send_message(client=client, room_id=room_id, text=text, msgtype=msgtype, html=html)
elif text:
await send_message(client=client, room_id=room_id, text=text, msgtype=msgtype)
except (LookupError, ValueError, TypeError):
room.send_text("Error parsing data in prometheus request")
await send_message(client=client, room_id=room_id, text="Error parsing data in prometheus request")
print("Error parsing JSON and forming message:", file=sys.stderr)
traceback.print_exc()
print(file=sys.stderr, flush=True)
return "Error parsing JSON and forming message", 500
except MatrixRequestError as e:
return matrix_error(e)
except nio.ErrorResponse as response:
abort(app.make_response(matrix_error(response)))
finally:
await client.close()
# see Flask.make_response, this is interpreted as (body, status)
# see Flask.make_response, this is interpreted as (text, status)
return "", 204
@app.route("/matrix", methods=("POST",))
def notify():
@app.post("/matrix")
async def notify():
if "X-Gitlab-Token" in request.headers:
return process_gitlab_request()
return await process_gitlab_request()
elif "X-Jenkins-Token" in request.headers:
return process_jenkins_request()
return await process_jenkins_request()
elif "type" in request.args and request.args.get("type") == "prometheus":
return process_prometheus_request()
return await process_prometheus_request()
else:
return "Cannot determine the request's webhook cause", 400