My Profile Photo

kubuszok.com


Personally just a "developer" without X in front of it, currently working with Scala.

I enjoy learning new things, especially more abstract like mathematics or algorithmics.

If you want to know about new posts follow me on Twitter or RSS feed!


Setting up Mattermost

When you live far away from some of your best friends, you want to find some reasonable ways to stay in touch. After a while, we found out that social media and instant messaging apps are not really an answer to our needs, so we decided to give a chance to something that easily engages everyone. Most, if not all, of us, know about Slack and how it (usually) improves communication within teams that use it. However, if you want to fully own your data and e.g. have access to all archives without high pricing, you might start with some cheaper alternative. It appears there is one - Mattermost Team Edition. And that’s what we decided to try out.

In this post, I want to describe how I configured my Mattermost server step by step. I made some assumptions about used hosts, distros, and setup that simplified the whole process. With those assumptions in mind, I compiled knowledge about basic server security guidelines from Mattermost’s documentation and several tutorials. I’ll show that, as long as you are not afraid of command line, you can quickly setup your own secured chat server from scratch. After that, I will describe how its UX compares to Slack’s and whether or not it’s something right for you.

Originally this post was published on ScalaC blog at 2016-10-27.

Prerequisites

The setup we decided to try out is a single Debian server with TLS secured connection and daily backups of all configs and user data in the cloud. We decided to go with the cheapest Linode server, obtain a TLS certificated from Let’s Encrypt and back up our data using both Linode backup functionality and AWS S3 storage. Altogether, it should be under $13 per month. Since Let’s Encrypt won’t allow you to register a certificate to a domain like li[number]].members.linode.com, you should also obtain some nice domain if you don’t have one already.

Easy to setup

I was honestly surprised, how easy it was to set up a Mattermost server to the point where it just works. Merely following the tutorial let me set things up in about 2 hours, from the moment I registered a Linode account to the moment when I registered as the first user on working site. Most of that time I spend reading comprehensive tutorials to learn how to tweak things in the future (links to useful materials at the end of the post).

Preparing a server

First, I registered at the Linode site and configured my planning - 1 node with 2 GB of RAM, 1 CPU Core, 24 GB of space and 2TB of transfer. With that, I created a node with a Debian Jessie distribution, set up the root password and booted it.

From Remote Access tab, I read the server’s IP and used it to log in via ssh (for the sake of this post, let’s assume that the server’s IP is 1.1.1.1):

ssh root@1.1.1.1

I chose a password that is easy to remember, but long enough that no one should hack into my server during the next hour or so - I wanted to block logging in with root and password as soon as possible.

Next, I created a new admin account on the server and added it to the sudoers group. Then I logged out of the root account:

adduser admin
usermod -a -G sudo admin
exit

Then I generated an RSA key and set up ssh to use it when connecting to my server. I connected to the server using the admin user (ssh gave me a warning about not being able to connect using RSA key - that is expected as I was just about to configure it):

ssh admin@1.1.1.1

Then I added my public key on the server:

mkdir .ssh
echo "[copy pasted id_rsa.pub]" > .ssh/authorized_keys
chown -R admin:admin .ssh
chmod 700 .ssh
chmod 600 .ssh/authorized_keys

I logged out and logged in to check if logging with the RSA key works as expected. Once I confirmed that I can do that, I could apply first security measures: blocking login by password and logging into the root account.

sudo nano /etc/ssh/sshd_config

I changed the default port, let’s say to 222 (any port < 1024 and != 22 should do) and checked that some options are set like below:

Port 222
...
PermitRootLogin
...
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile    %h/.ssh/authorized_keys
...
PermitEmptyPasswords no
...
PasswordAuthentication no

To make sure that everything worked as expected I checked that:

ssh root@1.1.1.1 # would fail
ssh admin@1.1.1.1 # would fail as well
ssh admin@1.1.1.1 -p 222 # would log in using RSA key

Finally, I made sure that the server is up to date:

sudo apt-get update
sudo apt-get upgrade

That is a nice moment to take care of configuring the domain. Usually, DNS propagation can take up to 72 hours and Let’s Encrypt would require us to have it already configured. Considering that I wanted my connection to be secure, outdated DNS would be a blocker. I assume that it is chat.mypage.com.

And now Mattermost!

Setup database

Official guide for Debian suggests using 3 machines:

  • database and files storage,
  • Mattermost node,
  • nginx load balancer.

However, if your team is really small (as mine), you can get away with a simpler single-server setup. Here nginx will be used to handle SSL termination, HTTP to HTTPS redirection and port mapping.

I decided to go with a PostgreSQL database. I installed it and logged into postgres:

sudo apt-get install postgresql postgresql-contrib -y
sudo -i -u postgres
pql

and created a database:

CREATE DATABASE mattermost;
CREATE USER mmuser WITH PASSWORD 'mmuser_password';
GRANT ALL PRIVILEGES ON DATABASE mattermost to mmuser;
\q;
view raw

then quit the postgres account (and a postgres REPL) with exit.

You don’t have to name your database mattermost, your user mmuser and make password mmuser_password. And I strongly suggest you change them.

Official guide suggests you to make some changes to the PostgreSQL configuration, but as in this example I’m not setting up the DB as a separate server, it is not needed.

Installing Mattermost

I downloaded and extracted the Mattermost installation inside /opt with the data directory under /opt/mattermost/data:

wget https://releases.mattermost.com/3.4.0/mattermost-team-3.4.0-linux-amd64.tar.gz
tar -xvzf mattermost-team-3.4.0-linux-amd64.tar.gz
rm mattermost-team-3.4.0-linux-amd64.tar.gz
sudo mkdir -p /opt
sudo mv mattermost /opt
sudo mkdir -p /opt/mattermost/data

According to good practices, I created a dedicated account for the Mattermost service (since I run it as a service) and made my admin part of the group allowed to mess with the installation:

sudo useradd -r mattermost -U
sudo chown -R mattermost:mattermost /opt/mattermost
sudo chmod -R g+w /opt/mattermost
sudo usermod -aG mattermost admin

Halfway through the setup! I opened the settings with en editor:

nano /opt/mattermost/config/config.json

to change "DataSource"’s value into "postgres://mmuser:mmuser_password@127.0.0.1:5432/mattermost?sslmode=disable&connect_timeout=10". At this point I could check if the server could be actually be run:

cd /opt/mattermost/bin
./platform

When I connected to 1.1.1.1:8065 I could see that the server was working as expected. I used that opportunity to quickly register the first user (which automatically becomes server administrator).

I got back to ssh and terminated the server with ctrl+c. The only thing left was turning it into service. I created the file /etc/systemd/system/mattermost.service and filled it with:

[Unit]
Description=Mattermost is an open source, self-hosted Slack-alternative
After=syslog.target network.target

[Service]
Type=simple
User=mattermost
Group=mattermost
ExecStart=/opt/mattermost/bin/platform
PrivateTmp=yes
WorkingDirectory=/opt/mattermost
Restart=always
RestartSec=30
LimitNOFILE=49152

[Install]
WantedBy=multi-user.target

and then loaded my newly created service with:

sudo systemctl daemon-reload
sudo systemctl enable mattermost
sudo systemctl start mattermost

With that, I had Mattermost running at (default) port 8065. Personally, I preferred to change that port (because why not) by editing /etc/systemd/system/mattermost.service and restarting the service.

Configure nginx

Well, actually at this point I already had some functioning chat service, right? But I didn’t intend to stop on that - it still required passing the port number and the connection was not secure. To address these issues, first I had to install nginx.

sudo apt-get install nginx -y

After accessing http://1.1.1.1 I could confirm that nginx was working. So now I had to configure it to redirect :80 calls to Mattermost. I’ve created file /etc/nginx/sites-available/mattermost:

server {
  server_name chat.mypage.com;

  location / {
    client_max_body_size 50M;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Frame-Options SAMEORIGIN;
    proxy_pass http://127.0.0.1:8065;
  }
}

Now I had to link it as an enabled site, remove default config and restart nginx:

sudo rm /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/mattermost /etc/nginx/sites-enabled/mattermost
sudo service nginx restart

With that, I had a nicely working server… which still needed some security tweaks.

Securing server

At that moment I’d only adjusted ssh logging in a little bit. I had to add a few more things to introduce minimal security of the server. First I started by making sure that one won’t try to attack ssh by brute force. Such an attacker could be banned with fail2ban.

fail2ban

sudo apt-get install fail2ban -y
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Now we would like to make sure that ssh configuration point to the right port:

...
[ssh]
...
port   = 222
filter = sshd
...
[sshd]
...
port   = 222
filter = sshd-ddos
...
sudo service fail2ban restart

AFAIR the rest of the default setup on Debian Jessie was good enough. Then I needed to adjust iptables to filter out any other port scanning attempts.

iptables

I created a file with iptables rules with:

sudo mkdir /etc/iptables
sudo nano /etc/iptables/rules

and filled it with the following content:

*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]

# Accept any related or established connections
-I INPUT  1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-I OUTPUT 1 -m state --state RELATED,ESTABLISHED -j ACCEPT

# Allow all traffic on the loopback interface
-A INPUT  -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT

# Allow outbound DHCP request - Some hosts (Linode) automatically assign the primary IP
-A OUTPUT -p udp --dport 67:68 --sport 67:68 -j ACCEPT

# Outbound DNS lookups
-A OUTPUT -o eth0 -p udp -m udp --dport 53 -j ACCEPT

# Outbound PING requests
-A OUTPUT -p icmp -j ACCEPT

# Outbound Network Time Protocol (NTP) request
-A OUTPUT -p udp --dport 123 --sport 123 -j ACCEPT

# SSH
-A INPUT  -i eth0 -p tcp -m tcp --dport 222 -m state --state NEW -j ACCEPT

# Outbound HTTP
-A OUTPUT -o eth0 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A OUTPUT -o eth0 -p tcp -m tcp --dport 443 -m state --state NEW -j ACCEPT

# Inbound HTTP
-A INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport 443 -j ACCEPT

COMMIT

It is slightly different from the original file from the tutorial I based my setup on - I had to uncomment outbound DHCP request as I used Linode host and it was essential for the server to work. Another change was allowing inbound HTTP - without it, some Mattermost requests were misbehaving: no message send confirmation, etc.

I had to test these rules before accepting them permanently:

sudo iptables-apply --timeout 60 /etc/iptables/rules

With command above I had 60 seconds to check if I could create new connections to the server. I recommend checking:

  • whether a new ssh connection can be made (in another terminal/console),
  • whether one can establish a connection to Mattermost server, log in and send a message with confirmation.

If the answer to both questions was true, then I could confirm that iptables were set up correctly and apply these settings permanently. I needed a few tries before it worked as intended, but once I introduced changes described above anything worked as expected. I only needed to add a hook to make sure settings would be applied on network startup:

sudo nano /etc/network/if-pre-up.d/iptables
sudo chmod +x /etc/network/if-pre-up.d/iptables

where /etc/network/if-pre-up.d/iptables was filled with:

#!/bin/sh
iptables-restore < /etc/iptables/rules

TSL and nginx again

Now the only thing I had to take care of was creating a SSL (TLS) certificate and using it to secure the connection. Let’s Encrypt allows us to get a certificate for free so I gave it a shot. First, I added jessie-backports to /etc/apt/sources.list and updated apt cache. With that I was able to install certbot from repositories:

#!/bin/sh
iptables-restore < /etc/iptables/rules
view raw

At this point I already had my domain - I really recommend setting it up early on as I REALLY had to wait about 72 hours before my ISP’s DNS cache had it. And with the settings I want to describe here, I could no longer access the site through IP or Linode’s domain. (A workaround was to move away from my ISP’s DNS cache as it surprisingly worked more reliably and faster, but that’s a tale for another time). I’d like to remind that I assume here that the domain is chat.mypage.com.

Since certbot’s support for nginx is experimental and not turned on by default I used standalone routine - it required me to turn off nginx for a moment, so that certbot could set up a temporary server, and use it for serving content to Let’s Encrypt servers to prove that the domain truly belongs to me (or specifically that the domain points to the server that cerbot is currently run on).

sudo apt-get install certbot -t jessie-backports -y

Sometimes running certbot requires several attempts - to avoid spamming with unnecessary requests, their server has quotas for certificate requests. If your request was denied, just try again later on. At some point, you will succeed. Once I obtained the certificate, I was able to access it in /etc/letsencrypt/live/chat.mypage.com. With that, I could finally make nginx use it to secure HTTP(S) connections. I changed the content of /etc/nginx/sites-available/mattermost to:

sudo service nginx stop
sudo certbot certonly --standalone -d chat.mypage.com

Finally, I started nginx again:

server {
  listen         80;
  server_name    chat.mypage.com;
  return         301 https://$server_name$request_uri;
}

server {
  listen 443 ssl;
  server_name chat.mypage.com;

  ssl on;
  ssl_certificate /etc/letsencrypt/live/chat.mypage.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/chat.mypage.com/privkey.pem;
  ssl_session_timeout 5m;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;

  location / {
    gzip off;
    proxy_set_header X-Forwarded-Ssl on;
    client_max_body_size 50M;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Frame-Options SAMEORIGIN;
    proxy_pass http://127.0.0.1:6324;
  }
}

to confirm that http://chat.mypage.com redirects to https://chat.mypage.com which in turn uses secure TLS connections.

To make sure that certificate would be renewed every 30 days (Let’s Encrypt gives certificates valid for only 90 days) I also created a cron job:

sudo nano /etc/cron.monthly/certbot_renewal
sudo chmod +x /etc/cron.monthly/certbot_renewal

with content:

#!/bin/sh
certbot renew --standalone --pre-hook "service nginx stop" \
              --post-hook "service nginx start" \
              --force-renew >> /var/log/certificate_renewal.log 2>&1

Basic security was done.

Back up your data

While the application was ready to use and I could actually start allowing users on the server (after I played around with Mattermost’s settings), I needed to make sure that whatever data would be generated - user’s archives, uploaded files and so on - would be preserved for certain. First thing I could do was enabling backup on Linode - it costs me $2.5 a month and creates a snapshot of my whole node once a week.

But once a week is not enough. What if I wanted to restore content 6 days after the last backup? What if I wanted to move it to another node/server? For that, I needed to find some additional solution.

Upload to AWS S3

I decided to go with AWS S3 storage. Uploading all the config files and user data once a day should give me enough confidence and with AWS’ prices, it is even cheaper than Linode’s backup.

Installation of AWS command line tools is rather simple:

cd /opt
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
rm awscli-bundle.zip

For AWS S3 I needed an Amazon account paired with my credit card (for billings). There, in S3 services settings, I created a new bucket for my backups. Finally, I used Security Credentials for creating keys just for my backups. I used them for creating an aws_upload bash script:

#!/bin/bash

aws_upload () {
  AWS_ACCESS_KEY_ID=[my access key] \
  AWS_SECRET_ACCESS_KEY=[secret access key] \
  AWS_DEFAULT_REGION=[bucket region] \
  aws s3 cp "$backup_enc" "s3://my.backup.bucket/$s3_new_backup_name"
}

After sourcing it into my backup script I would be able to upload backup file as backup-from-${current-date}.tar.gz with command like:

backup_enc='backup.tgz.enc'
s3_new_backup_name="backup-from-$(date '+%Y-%m-%d').tgz.enc"
aws_upload

Since it keeps credentials in a separate file I could check backup script into git ignoring aws_upload and avoiding credentials exposure.

Encryption

Because I have limited trust to third-party storages I would still encrypt the backup with something like mcrypt before the upload.

Installation of mcrypt was even faster:

sudo apt-get install mcrypt -y

Since it requires some key for encryption I decided to generate a random one:

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 512 | head -n 1 > ./keyfile

Then I could encrypt/decrypt backups with something like:

cat backup | mcrypt --keyfile ./keyfile > backup_enc
cat backup_enc | mcrypt --decrypt --keyfile ./keyfile > backup

Similarly like with credentials I added the keyfile to ignored so that the key would not be published in a repository.

My backup scripts

Long story short - backup scripts would just copy all config/data files into one place, archive them and publish on AWS S3. The only exception would be Mattermost DB which would be backed up with pqdump to create a dump file which can just be read into the DB like any other PostgreSQL script. To avoid typos in hardcoded paths, I extracted all paths into variables and wrote a few helpers to make the scripts more readable (to me :D).

.common.sh:

#!/bin/bash

# Task runners

run_task () {
    echo "  - $1"
    eval "$2" > /dev/null
    if [ $? -eq 0 ]; then
        echo "    [done]"
    else
        echo "    [failed]"
        exit -1
    fi
}

run_grouped () {
    echo "  - $1"
    eval "$2"
    echo "    [done]"
}

run_subtask () {
    echo "    - $1"
    eval "$2" > /dev/null
    if [ $? -eq 0 ]; then
        echo "      [done]"
    else
        echo "      [failed]"
        exit -1
    fi
}

# Common directories and files

encryption_key="$(dirname $0)/keyfile"

backup_dir=/var/backup
backup_rel=`python -c "import os.path; print os.path.relpath('$backup_dir', '/')"`
backup_tar=/var/backup.tgz
backup_enc=/var/backup.tgz.enc

postgresql_dir=/etc/postgresql/9.4

postgresql_config_original="$postgresql_dir/main/postgresql.conf"
postgresql_config_backup="$backup_dir/postgresql.conf"

postgresql_hba_original="$postgresql_dir/main/pg_hba.conf"
postgresql_hba_backup="$backup_dir/pg_hba.conf"

mm_dir=/opt/mattermost

mm_config_original="$mm_dir/config/config.json"
mm_config_backup="$backup_dir/config.json"

nginx_config_original=/etc/nginx/sites-available/mattermost
nginx_config_backup="$backup_dir/nginx.conf"

ssh_config_original=/etc/ssh/sshd_config
ssh_config_backup="$backup_dir/sshd.config"

iptables_config_original=/etc/iptables/rules
iptables_config_backup="$backup_dir/iptables.rules"

cron_backup_original=/etc/cron.daily/complete_backup
cron_backup_backup="$backup_dir/complete_backup"

cron_certbot_reneval_original=/etc/cron.monthly/certbot_renewal
cron_certbot_reneval_backup="$backup_dir/certbot_renewal"

network_if_pre_up_iptables_original=/etc/network/if-pre-up.d/iptables
network_if_pre_up_iptables_backup="$backup_dir/iptables.if-pre-up"

admins_original=/home
admins_backup="$backup_dir/admins"

database_backup="$backup_dir/db.dump"

local_data_original="$mm_dir/data"
local_data_backup="$backup_dir/files"

certificates_original=/etc/letsencrypt/live/chat.mypage.com
certificates_backup="$backup_dir/certificates"

s3_new_backup_name="backup-from-$(date '+%Y-%m-%d').tgz.enc"

# S3 Upload functions

# Should follow this convention:
#
# aws_upload () {
#   AWS_ACCESS_KEY_ID=[my access key] \
#   AWS_SECRET_ACCESS_KEY=[secret access key] \
#   AWS_DEFAULT_REGION=[bucket region] \
#   aws s3 cp "$backup_enc" "s3://chat.mypage.com/$s3_new_backup_name"
# }

source "$(dirname $0)/aws_upload" # should define aws_upload

Then backup is just a matter of defining tasks:

#!/bin/bash

source "$(dirname $0)/.commons.sh"

echo "Backing up Server"
if [ ! -d $backup_dir ]; then
    mkdir $backup_dir -p > /dev/null
fi

run_grouped "backing up configs" "$(cat <<-GROUPED
    run_subtask "backing up postgres config" "$(cat <<-TASK
        cp -fp "$postgresql_config_original" "$postgresql_config_backup" && \
        cp -fp "$postgresql_hba_original" "$postgresql_hba_backup"
    TASK
    )"
    run_subtask "backing up MatterMost config" "$(cat <<-TASK
        cp -fp "$mm_config_original" "$mm_config_backup"
    TASK
    )"
    run_subtask "backing up Nginx config" "$(cat <<-TASK
        cp -fp "$nginx_config_original" "$nginx_config_backup"
    TASK
    )"
    run_subtask "backing up sshd settings" "$(cat <<-TASK
        cp -fp "$ssh_config_original" "$ssh_config_backup"
    TASK
    )"
    run_subtask "backing up iptables" "$(cat <<-TASK
        cp -fp "$iptables_config_original" "$iptables_config_backup"
    TASK
    )"
    run_subtask "backing up cron jobs" "$(cat <<-TASK
        cp -fp "$cron_backup_original" "$cron_backup_backup" && \
        cp -fp "$cron_certbot_reneval_original" "$cron_certbot_reneval_backup"
    TASK
    )"
    run_subtask "backing up network jobs" "$(cat <<-TASK
        cp -fp "$network_if_pre_up_iptables_original" \
               "$network_if_pre_up_iptables_backup"
    TASK
    )"
GROUPED
)"

run_task "backing up admins" "$(cat <<-TASK
    if [ ! -d "$backup_dir/admins" ]; then
        mkdir "$backup_dir/admins" -p > /dev/null
    fi
    cp -rfp "$admins_original"/* "$admins_backup"
TASK
)"

run_grouped "backing up data" "$(cat <<-GROUPED
    run_subtask "backing up database" "$(cat <<-TASK
        sudo -u postgres pg_dump mattermost > "$database_backup"
    TASK
    )"
    run_subtask "backing up local data" "$(cat <<-TASK
        if [ ! -d "$local_data_backup" ]; then
            mkdir "$local_data_backup" -p > /dev/null
        fi
        cp -rfp "$local_data_original"/* "$local_data_backup"
    TASK
    )"
    run_subtask "backing up certificates" "$(cat <<-TASK
        if [ ! -d "$certificates_backup" ]; then
            mkdir "$certificates_backup" -p > /dev/null
        fi
        cp -rfp "$certificates_original"/* "$certificates_backup"
    TASK
    )"
GROUPED
)"

run_grouped "upload backup to S3" "$(cat <<-GROUPED
    run_subtask "compressing backup" "$(cat <<-TASK
        tar -pcvzf "$backup_tar" -C / "$backup_rel"
    TASK
    )"
    run_subtask "encrypting backup" "$(cat <<-TASK
        cat "$backup_tar" | \
          mcrypt --keyfile "$encryption_key" > "$backup_enc" 2> /dev/null
    TASK
    )"
    run_subtask "uploading backup" "$(cat <<-TASK
        mjk_upload && jelz_upload
    TASK
    )"
GROUPED
)"

echo "[done]"

Similarly restoring:

#!/bin/bash

source "$(dirname $0)/.commons.sh"

echo "Restoring Server setup"

run_grouped "restoring backup files" "$(cat <<-GROUPED
    run_subtask "decrypting backup" "$(cat <<-TASK
        cat "$backup_enc" | \
          mcrypt --decrypt --keyfile "$encryption_key" > "$backup_tar"
    TASK
    )"
    run_subtask "restoring backup directory" "$(cat <<-TASK
        ls "$backup_dir" > /dev/null || tar -pxvfz "$backup_tar" -C /
    TASK
    )"
GROUPED
)"

run_grouped "restoring configs" "$(cat <<-GROUPED
    run_subtask "Restoring postgres config" "$(cat <<-TASK
        cp -fp "$postgresql_config_backup" "$postgresql_config_original" && \
        cp -fp "$postgresql_hba_backup" "$postgresql_hba_original"
    TASK
    )"
    run_subtask "restoring MatterMost config" "$(cat <<-TASK
        cp -fp "$mm_config_backup" "$mm_config_original"
    TASK
    )"
    run_subtask "restoring Nginx config" "$(cat <<-TASK
        cp -fp "$nginx_config_backup" "$nginx_config_original"
    TASK
    )"
    run_subtask "restoring sshd settings" "$(cat <<-TASK
        cp -fp "$ssh_config_backup" "$ssh_config_original"
    TASK
    )"
    run_subtask "restoring iptables" "$(cat <<-TASK
        cp -fp "$iptables_config_backup" "$iptables_config_original"
    TASK
    )"
    run_subtask "restoring cron jobs" "$(cat <<-TASK
        cp -fp "$cron_backup_backup" "$cron_backup_original" && \
        cp -fp "$cron_certbot_reneval_backup" "$cron_certbot_reneval_original"
    TASK
    )"
    run_subtask "backing up network jobs" "$(cat <<-TASK
        cp -fp "$network_if_pre_up_iptables_backup" \
               "$network_if_pre_up_iptables_original"
    TASK
    )"
GROUPED
)"

run_task "restoring admins" "$(cat <<-TASK
    cp -rfp "$admins_backup"/* "$admins_original"
TASK
)"

run_grouped "restoring data" "$(cat <<-GROUPED
    run_subtask "restoring database" "$(cat <<-TASK
        sudo -u postgres \
          psql --set ON_ERROR_STOP=on mattermost < "$database_backup" 2> /dev/null
    TASK
    )"
    run_subtask "restoring local data" "$(cat <<-TASK
        cp -rfp "$local_data_backup"/* "$local_data_original"
    TASK
    )"
    run_subtask "restoring certificates" "$(cat <<-TASK
        if [ ! -d "$certificates_original" ]; then
            mkdir "$certificates_original" -p > /dev/null
        fi
        cp -rfp "$certificates_backup"/* "$certificates_original"
    TASK
    )"
GROUPED
)"

echo "[done]"

The last thing to do is write a cron job to perform a backup once a day:

sudo nano /etc/cron.daily/complete_backup
sudo chmod +x /etc/cron.daily/complete_backup
#!/bin/sh
/opt/backup/backup.sh >> /var/log/complete_backup.log 2>&1

Mattermost tweaks

Out of the box, Mattermost would have some generic site name. Of all settings that we would like to tweak after installation, that is the only one we can only access by editing /opt/mattermost/config.config.json (TeamSettings/SiteName). Any other config that would be interesting to us could be accessed via the System Console window of Mattermost. That change would require us to restart Mattermost manually to take effect.

Once we change the site name to something more preferable we can also change the title of notification emails (if we’ll use them) and admin’s email address. If we decide to allow push notifications for mobile devices (Android and iOS clients), then we’ll have to decide on the server - we can use test server (TPNS) which does not encrypt our messages (then I would recommend sending generic description with user and channel names) or set up an encrypted push notification server manually - problem with the later is that official clients only support Mattermost servers. So one has to either pay for Mattermost’s encrypted push notification service or fork clients and release one’s own version.

I would also recommend skimming through other options to e.g. regenerate hashes for links and turning off public user registration. If we decide to have several unrelated teams on one server, we might also disable option for cross-team communication.

As for customization, for now, we can only set up global default localization options. An option to change default fonts or themes is still an open ticket on the issue tracker.

Mattermost vs Slack

How does Mattermost actually compare to Slack in action? I won’t deny it - for me, it looks less polished. Of 10 possible fonts, only 2 (Open Sans and Roboto) seems to handle Polish characters correctly and 1 of them (Roboto) seems to fail when we use bold (e.g. when in notifications from some channel). We cannot change the size of the font, and by default, they seem a little bit too small (after zooming in, they seem to be a bit too big). The default theme has annoying sets of colors and I am unable to change these settings without manually editing source code files and preparing my own build.

Some other potentially useful options are also lacking. I am unable to configure the server so that only admin could invite users. Instead, there is a registration link with a hash which could be hidden from the landing page and (optionally) requiring email registration from a fixed domain. As an admin, I can only block any registration to prevent users from sending it to whoever they want. Alternatively, I can disable any registration and use Mattermost command line interface for creating my users (probably the best solution for enterprises). And newly registered users cannot be automatically invited into a predefined list of channels.

Mobile clients seem to be build using an embedded browser - I only tested the Android client, and it has significant startup overhead before I’ll see that my chat is even loading. I attribute it to the browser initialization. When I receive a notification and click it to go to the message, client reloads the browser, even if I just have the said conversation opened. Currently, the client does not have an option to open menus by sliding the screen to the side, though that’s exactly how they open once we click on a menu icon. You can also only have one server configured in your app. (And you cannot remove it easily - when I switched from HTTP to HTTPS the app stuck on an infinite server load and I had to reinstall it in order to enter a new URL). Overall, I got the feeling that the mobile clients are slow and heavy (at least Android one is. AFAIK iOS client is built the same way so I assume it shares it’s brother’s issues).

Other quirks that brought my attention was no ability to add reaction emoticons (we use them on Slack heavily) and worse, viewing thumbnails for assigned files - however the former should be solved with 3.5 release. When you remove a message ugly message removed notice would stay (it disappears after reloading but still). If you spent a lot of time on Slack, these small quirks will add up and you won’t be able to help but notice the difference in the user experience. So does Mattermost have any advantages besides the pricing?

I’d say yes. Even if currently each user would have to apply customization settings manually, they are still able to change the color of almost everything. (I weren’t able to locate the option to disable my own messages’ highlighting - each message you send has a different background than all the rest. This color is also used for highlighting the message your cursor is over it.) Having the ability to respond to a message directly (kind of like on Reddit) might also be a nice thing if 2 parallel discussions emerge on one channel.

When looking at the message bar you’ll notice that it lacks Slack’s plus button, it only has an assignment. There is no distinction on a photo or generic file upload (which is a minus) and there is no option to publish a snippet. The latter is not that much of a problem, though, as Mattermost has better markdown support than Slack, so you can paste your code like on GitHub:

val s = Seq()

and with the ability to respond to a message it works similar to commenting assignments.

It is also useful to share server for several teams because if you want those teams to communicate, you can still allow that. And anyone on the server could then talk to anyone else, without the need to register on each team separately.

Having access to the whole archive out-of-the-box and without additional fees is also a big plus. And if you are really concerned about the security, privacy, and ownership of your data, having your own secured server will always beat trusting third-party vendor locked-in solutions.

Summary

Setting up my own Mattermost server was a great learning experience. And a lot of fun. Though the service is less polished than a Slack, it suits my use case and my pocket. I am looking forward to new releases which will bring new functionalities and get rid of aforementioned quirks.

I cannot recommend this solution to everyone - limited ability to control who can register might be a deal breaker for many companies. However, if you are managing a small team (of friends?) or your company is OK with manually managing users via command line, or the Mattermost server is only available in the intranet or via VPN it might be a quite good Slack alternative. In such a case, though, you should consider subscribing for paid push-notifications to make sure they are distributed via secured connections. Or, if your company is reaally concerned about owning everything, forking mobile clients to use your own dedicated push-notification server (for a while forking the platform might be the only way of providing your own color-scheme defaults). Or perhaps just go straight to Mattermost Enterprise Edition which should ease all those issues and add some additional perks like rebranding. But here I wanted to focus on Team Edition.

In the end, it’s all about the tradeoffs. If your primary concerns are flawless user experience and polished UI - go with Slack. If you want to go an extra mile to make sure that you actually own your data - Mattermost is the way to go.

These are the sources I read while configuring my server. I admit that a major part of my text is just a creative ripoff ;)

  • Production Install on Debian Jessie - complete manual on setting up Mattermost on a server. I just skipped all settings required by 3-servers’ setup and replaced cloning Let’s Encrypt with certbot already available in jessie-backports.
  • Securing a Linux Server - a source I used when I configured SSH, iptables, and fail2ban.

Articles I just read through to see if I’ve got everything covered: