Install a self-hosted matrix server (part 1)

How to install a self-hosted matrix with docker and traefik.

Install a self-hosted matrix server (part 1)

Why selfhosting this ?

One of the things I hated the most on a daily basis was to switch apps, environments, to follow multiple channels I'm interested in:

  1. Slack for a client who likes it.
  2. Discord for some interesting programming rooms, mechanical keyboards geek stuff, selfhosted insane channels, etc...
  3. Gitter for some opensource libraries conversations I'm participating in.
  4. HexChat client for some big rooms where you always get accurate and precise answers, rtfm sometimes when you deserve it :)
  5. WhatsApp with some friends.
  6. Signal with other friends (better friends).
  7. Feedly on android to catch on good old blog posts,
  8. Skype when I'm gagged into Microsoft world.
  9. Instagram, well because

So with Matrix you end up having everything in one client, which I just find simply amazing, now the reality is that having it self-hosted was not that simple, this is my journey trying to explain where the difficult spots were: if it can help someone else down the road, then it's all good, if this yields comments, then I'll probably learn something also, so it's a win-win.
Oh and since I'm almost exclusively using containers nowadays, this is done all in docker containers, with traefik handling the certificates.

Enter the matrix

The layout

I tend to keep all my conpose files in seperate directories, this one is no exception and is located in ~/mydocker/synapse

Within that directory the layout will be the following at the end of part 2:

❯ cd ~/mydocker~synapse
❯ tree -I media_store -a
.
├── coturn
│   └── turnserver.conf
├── docker-compose.yml
├── matrix-appservice-irc
│   ├── appservice-registration-irc.yaml
│   ├── config.yaml
│   └── passkey.pem
├── maubot
│   ├── config.yaml
│   ├── crypto
│   ├── dbs
│   ├── maubot-cli.json
│   ├── plugins
│   │   ├── rss.db
│   │   └── xyz.maubot.rss-v0.2.5.mbp
│   └── trash
├── mautrix-signal
│   ├── config.yaml
│   └── registration.yaml
├── mautrix-whatsapp
│   ├── config.yaml
│   ├── logs
│   └── registration.yaml
├── nginx
│   ├── matrix.conf
│   └── www
│       └── .well-known
│           └── matrix
│               ├── client
│               └── server
├── .psqlrc
├── riot
│   └── config.json
├── signald
│   ├── avatars
│   ├── data
│   ├── signald.db
│   └── signald.sock
└── synapse
    ├── homeserver.yaml
    ├── homeserver.yaml.bak
    ├── matrix.YOURDOMAIN.COM.log.config
    └── matrix.YOURDOMAIN.COM.signing.key

You can refer to it when looking at subparts of the compose yaml, mostly for volumes, now if you want to share things with your hose elsewhere you'll have to adapt.

First things first, the matrix database

Since we're going to use postgres, let's spin up a database, in a container.
This was a TIL experience for me, as matrix seems to be quite strict on locale, encoding.
If you get errors related to that just look at the below:

services:
  matrix-postgres:
    image: postgres:13
    restart: unless-stopped
    hostname: matrix-postgres
    environment:
      - POSTGRES_PASSWORD=CHANGEME
      - POSTGRES_USER=synapse_user
      - POSTGRES_DB=synapse
      - POSTGRES_INITDB_ARGS=--encoding='UTF8' --lc-collate='C' --lc-ctype='C'
    volumes:
      - ./.psqlrc:/root/.psqlrc:ro

The only interesting bit here is the POSTGRES_INITDB_ARGS env variable, I got the idea reading this issue.
You could also have a sql file like:

CREATE ROLE synapse_user;
ALTER ROLE synapse_user WITH PASSWORD 'CHANGEME';
ALTER ROLE synapse_user WITH LOGIN;
CREATE DATABASE synapse ENCODING 'UTF8' LC_COLLATE='C' LC_CTYPE='C' template=template0 OWNER synapse_user;
GRANT ALL PRIVILEGES ON DATABASE synapse TO synapse_user;

in the /docker-entrypoint-initdb.d/ since this is the "official" solution as rightly described here, but I found the env variable usage way nicer.

The other interesting part (you'll have to be a psql user to like it) is to bind your .psqlrc config, so that if you need to go directly into the container db (spoiler: you'll need it at least once to become admin), you'll have your usual config file.

Every little bit counts, the large number of columns in synapse tables makes the output way more comfortable in a terminal with this config taken shamelessly from this amazing blog, in this post in particular.

Setup synapse and connect your brain to the matrix

If you are a "neo" user like me, this part may be confusing, matrix-neo-kugeln-thumb-960-retina-1490x838-1-1024x576-2 I have to say it took me a while to understand it, not knowing the inner workings of the protocol.

You'll need 2 subdomains:

  1. synapse.YOURDOMAIN.COM
  2. matrix.YOURDOMAIN.COM

Traefik can't handle serving static files, so it will be used only to reverse proxy the synapse container, and nginx will serve static files that are needed to run the federation (see the docs here)
To get things clearer the setup would look like this if I got it right:

mermaid-diagram-20210414210233

The only reason there is nginx in the stack is because traefik is just a reverse proxy and can't serve static files, which we need for federation on the .well-know/ directory.

So now the docs tell you to run synapse container once to get a config, edit it, then run it again, that's a process you'll see for pretty much all containers that is quite neat I have to say.

The command to run in the synapse folder (see above) is:

docker run -it --rm \
    -v $(pwd):/data \
    -e SYNAPSE_SERVER_NAME=matrix.YOURDOMAIN.COM \
    -e SYNAPSE_REPORT_STATS=yes \
    -e UID=1000 \
    -e GID=1000 \
    matrixdotorg/synapse:latest generate

It gives you a homeserver.yaml file that needs to be edited, here's one without all the comments:

server_name: "matrix.YOURDOMAIN.COM"
pid_file: /data/homeserver.pid
web_client_location: https://riot.YOURDOMAIN.COM/
public_baseurl: https://synapse.YOURDOMAIN.COM/
listeners:
  - port: 8008
    tls: false
    type: http
    x_forwarded: true
    resources:
      - names: [client, federation]
        compress: false
admin_contact: 'mailto:matrix@YOURDOMAIN.COM'
limit_remote_rooms:
retention:
  enabled: true
acme:
    enabled: false
    port: 80
    bind_addresses: ['::', '0.0.0.0']
    reprovision_threshold: 30
    domain: matrix.example.com
    account_key_file: /data/acme_account.key
caches:
   per_cache_factors:
database:
  name: psycopg2
  args:
    user: synapse_user
    password: CHANGEME
    database: synapse
    host: matrix-postgres
    cp_min: 5
    cp_max: 10
log_config: "/data/matrix.YOURDOMAIN.COM.log.config"
media_store_path: "/data/media_store"
url_preview_accept_language:
turn_uris:
  - "turn:matrix.YOURDOMAIN.COM:3478?transport=udp"
  - "turn:matrix.YOURDOMAIN.COM:3478?transport=tcp"
  - "turns:matrix.YOURDOMAIN.COM:3478?transport=udp"
  - "turns:matrix.YOURDOMAIN.COM:3478?transport=tcp"
turn_shared_secret: "TURN_SHARED_SECRET"
turn_user_lifetime: 1h
turn_allow_guests: true
enable_registration: true
account_validity:
registration_shared_secret: "SHOULD_BE_GENERATED"
account_threepid_delegates:
metrics_flags:
report_stats: true
app_service_config_files:
  - /mautrix-whatsapp/registration.yaml
  - /mautrix-signal/registration.yaml
  - /matrix-appservice-irc/appservice-registration-irc.yaml
macaroon_secret_key: "SHOULD_BE_GENERATED"
form_secret: "SHOULD_BE_GENERATED"
signing_key_path: "/data/matrix.YOURDOMAIN.COM.signing.key"
old_signing_keys:
trusted_key_servers:
  - server_name: "matrix.org"
saml2_config:
  sp_config:
  user_mapping_provider:
    config:
oidc_providers:
cas_config:
sso:
password_config:
   policy:
ui_auth:
email:
  smtp_host: mail.YOURDOMAIN.COM
  smtp_port: 587
  smtp_user: "matrix@YOURDOMAIN.COM"
  smtp_pass: "CHANGEME"
  require_transport_security: true
  notif_from: "Your Friendly %(app)s homeserver <matrix@YOURDOMAIN.COM>"
  app_name: CHANGEME
  enable_notifs: true
password_providers:
push:
spam_checker:
user_directory:
opentracing:
redis:
  enabled: true
  host: redis
  port: 6379

Basically everything in capitals should be changed to your liking.

Now run synapse from the compose file, dcu -d synapse, with a redis container as well:

  synapse:
    image: matrixdotorg/synapse:v1.31.0
    restart: unless-stopped
    volumes:
      - ./synapse:/data
      - ./mautrix-whatsapp:/mautrix-whatsapp
      - ./mautrix-signal:/mautrix-signal
      - ./matrix-appservice-irc:/matrix-appservice-irc
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.synapse.rule=Host(`synapse.YOURDOMAIN.COM`)"
      - "traefik.http.routers.synapse.entrypoints=websecure"
      - "traefik.http.routers.synapse.tls.certresolver=myhttpchallenge"
      - "traefik.http.services.synapse.loadbalancer.server.port=8008"
    environment:
      - SYNAPSE_CONFIG_DIR=/data
      - SYNAPSE_CONFIG_PATH=/data/homeserver.yaml
      - UID=1000
      - GID=1000
      - TZ=Europe/Paris
    networks:
      - default
      - traefik-local
  redis:
    image: redis:6.2.1
    restart: unless-stopped

The acute reader would notice the volumes, here the configuration from homeserver.yaml will be placed inside the container in the /data directory.
And since in that config we use the bridges registration (see that in part 2), we need to also use the bridges directories as volumes.

Now you should be able to see on https://synapse.YOURDOMAIN.COM after being redirected:

20210418_1616_688x565_1618755373

Congrats, that's one step closer to having Matrix running fully with federation !

Federation setup

This is the part where nginx enters the stack. The compose is as follows:

  nginx:
    image: nginx:1.19.10
    restart: unless-stopped
    volumes:
      - ./nginx/matrix.conf:/etc/nginx/conf.d/matrix.conf
      - ./nginx/www:/var/www/
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.matrix.rule=Host(`matrix.YOURDOMAIN.COM`)"
      - "traefik.http.routers.matrix.entrypoints=websecure"
      - "traefik.http.routers.matrix.tls.certresolver=myhttpchallenge"
      - "traefik.http.services.matrix.loadbalancer.server.port=80"
    networks:
      - default
      - traefik-local

Nothing really fancy, we'll need to setup the static files which will be served inside the /var/www/ inside the container.
The configuration just has 2 locations, one that reverse-proxy to the synapse container, the other that serves the static files.

server {
  listen         80 default_server;
  server_name    matrix.YOURDOMAIN.COM;

 location /_matrix {
    proxy_pass http://synapse:8008;
    proxy_set_header X-Forwarded-For $remote_addr;
    client_max_body_size 128m;
  }

  location /.well-known/matrix/ {
    root /var/www/;
    default_type application/json;
    add_header Access-Control-Allow-Origin  *;
  }
}

So create the federation files inside the nginx container with

  • mkdir -p www/.well-known/matrix/
  • touch www/.well-known/matrix/client
  • touch www/.well-known/matrix/server

the client file contains the following json:

{
  "m.homeserver": {
    "base_url": "https://matrix.YOURDOMAIN.COM"
  }
}

while the server contains:

{
  "m.server": "synapse.example.com:443"
}

Now run the nginx container and go test your setup to https://federationtester.matrix.org/ , if things went well all checks should be green.

Let's riot !

So now the server is installed, you need a client. There are many choices we'll try riot aka Element. I've not searched a lot this, but this looked a good solution, one that works in the browser but has also an android client.
I don't know the exact details but all this s React based which I use on a daily basis and like.
The process is simple, create a riot folder mkdir riot && cd riot then get a sample config with wget https://raw.githubusercontent.com/vector-im/element-web/develop/config.sample.json && mv config.sample.json config.json then edit it, it should be straightforward.
The compose section is also very straightforward and does not require too uch thinking:

  riot:
    image: vectorim/element-web:v1.7.25
    restart: unless-stopped
    volumes:
      - ./riot/config.json:/app/config.json:ro
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.riot.rule=Host(`riot.YOURDOMAIN.COM`)"
      - "traefik.http.routers.riot.entrypoints=websecure"
      - "traefik.http.routers.riot.tls.certresolver=myhttpchallenge"
      - "traefik.http.services.riot.loadbalancer.server.port=80"
    networks:
      - default
      - traefik-local

Now you're set, just connect to you client and register a new user.

Conclusion

Congratulations, you're now a matrix user, part 2 will show you some other stuff you can add, bridges mainly, video and audio with a turn server, and also bots, but for now you can begin chattting, joining servers on other platforms.