Setting up Mattermost
When you live far away from some of your best friends, you want to find some reasonable ways to stay in touch. After a while, we found out that social media and instant messaging apps are not really an answer to our needs, so we decided to give a chance to something that easily engages everyone. Most, if not all, of us, know about Slack and how it (usually) improves communication within teams that use it. However, if you want to fully own your data and e.g. have access to all archives without high pricing, you might start with some cheaper alternative. It appears there is one - Mattermost Team Edition. And that’s what we decided to try out.
In this post, I want to describe how I configured my Mattermost server step by step. I made some assumptions about used hosts, distros, and setup that simplified the whole process. With those assumptions in mind, I compiled knowledge about basic server security guidelines from Mattermost’s documentation and several tutorials. I’ll show that, as long as you are not afraid of command line, you can quickly setup your own secured chat server from scratch. After that, I will describe how its UX compares to Slack’s and whether or not it’s something right for you.
Originally this post was published on ScalaC blog at 2016-10-27.
Prerequisites
The setup we decided to try out is a single Debian server with TLS secured connection and daily backups of all configs and user data in the cloud. We decided to go with the cheapest Linode server, obtain a TLS certificated from Let’s Encrypt and back up our data using both Linode backup functionality and AWS S3 storage. Altogether, it should be under $13 per month. Since Let’s Encrypt won’t allow you to register a certificate to a domain like li[number]].members.linode.com, you should also obtain some nice domain if you don’t have one already.
Easy to setup
I was honestly surprised, how easy it was to set up a Mattermost server to the point where it just works. Merely following the tutorial let me set things up in about 2 hours, from the moment I registered a Linode account to the moment when I registered as the first user on working site. Most of that time I spend reading comprehensive tutorials to learn how to tweak things in the future (links to useful materials at the end of the post).
Preparing a server
First, I registered at the Linode site and configured my planning - 1 node with 2 GB of RAM, 1 CPU Core, 24 GB of space and 2TB of transfer. With that, I created a node with a Debian Jessie distribution, set up the root password and booted it.
From Remote Access tab, I read the server’s IP and used it to log in via ssh (for the sake of this post, let’s assume that the server’s IP is 1.1.1.1):
ssh root@1.1.1.1
I chose a password that is easy to remember, but long enough that no one should hack into my server during the next hour or so - I wanted to block logging in with root and password as soon as possible.
Next, I created a new admin account on the server and added it to the sudoers group. Then I logged out of the root account:
adduser admin
usermod -a -G sudo admin
exit
Then I generated an RSA key and set up ssh to use it when connecting to my server. I connected to the server using the admin user (ssh gave me a warning about not being able to connect using RSA key - that is expected as I was just about to configure it):
ssh admin@1.1.1.1
Then I added my public key on the server:
mkdir .ssh
echo "[copy pasted id_rsa.pub]" > .ssh/authorized_keys
chown -R admin:admin .ssh
chmod 700 .ssh
chmod 600 .ssh/authorized_keys
I logged out and logged in to check if logging with the RSA key works as expected. Once I confirmed that I can do that, I could apply first security measures: blocking login by password and logging into the root account.
sudo nano /etc/ssh/sshd_config
I changed the default port, let’s say to 222 (any port < 1024 and != 22 should do) and checked that some options are set like below:
Port 222
...
PermitRootLogin
...
RSAAuthentication yes
PubkeyAuthentication yes
AuthorizedKeysFile %h/.ssh/authorized_keys
...
PermitEmptyPasswords no
...
PasswordAuthentication no
To make sure that everything worked as expected I checked that:
ssh root@1.1.1.1 # would fail
ssh admin@1.1.1.1 # would fail as well
ssh admin@1.1.1.1 -p 222 # would log in using RSA key
Finally, I made sure that the server is up to date:
sudo apt-get update
sudo apt-get upgrade
That is a nice moment to take care of configuring the domain. Usually, DNS propagation can take up to 72 hours and Let’s Encrypt would require us to have it already configured. Considering that I wanted my connection to be secure, outdated DNS would be a blocker. I assume that it is chat.mypage.com.
And now Mattermost!
Setup database
Official guide for Debian suggests using 3 machines:
- database and files storage,
- Mattermost node,
- nginx load balancer.
However, if your team is really small (as mine), you can get away with a simpler single-server setup. Here nginx will be used to handle SSL termination, HTTP to HTTPS redirection and port mapping.
I decided to go with a PostgreSQL database. I installed it and logged into postgres:
sudo apt-get install postgresql postgresql-contrib -y
sudo -i -u postgres
pql
and created a database:
CREATE DATABASE mattermost;
CREATE USER mmuser WITH PASSWORD 'mmuser_password';
GRANT ALL PRIVILEGES ON DATABASE mattermost to mmuser;
\q;
view raw
then quit the postgres account (and a postgres REPL) with exit.
You don’t have to name your database mattermost, your user mmuser and make password mmuser_password. And I strongly suggest you change them.
Official guide suggests you to make some changes to the PostgreSQL configuration, but as in this example I’m not setting up the DB as a separate server, it is not needed.
Installing Mattermost
I downloaded and extracted the Mattermost installation inside /opt with the data directory under /opt/mattermost/data:
wget https://releases.mattermost.com/3.4.0/mattermost-team-3.4.0-linux-amd64.tar.gz
tar -xvzf mattermost-team-3.4.0-linux-amd64.tar.gz
rm mattermost-team-3.4.0-linux-amd64.tar.gz
sudo mkdir -p /opt
sudo mv mattermost /opt
sudo mkdir -p /opt/mattermost/data
According to good practices, I created a dedicated account for the Mattermost service (since I run it as a service) and made my admin part of the group allowed to mess with the installation:
sudo useradd -r mattermost -U
sudo chown -R mattermost:mattermost /opt/mattermost
sudo chmod -R g+w /opt/mattermost
sudo usermod -aG mattermost admin
Halfway through the setup! I opened the settings with en editor:
nano /opt/mattermost/config/config.json
to change "DataSource"’s value into "postgres://mmuser:mmuser_password@127.0.0.1:5432/mattermost?sslmode=disable&connect_timeout=10". At this point I could check if the server could be actually be run:
cd /opt/mattermost/bin
./platform
When I connected to 1.1.1.1:8065 I could see that the server was working as expected. I used that opportunity to quickly register the first user (which automatically becomes server administrator).
I got back to ssh and terminated the server with ctrl+c. The only thing left was turning it into service. I created the file /etc/systemd/system/mattermost.service and filled it with:
[Unit]
Description=Mattermost is an open source, self-hosted Slack-alternative
After=syslog.target network.target
[Service]
Type=simple
User=mattermost
Group=mattermost
ExecStart=/opt/mattermost/bin/platform
PrivateTmp=yes
WorkingDirectory=/opt/mattermost
Restart=always
RestartSec=30
LimitNOFILE=49152
[Install]
WantedBy=multi-user.target
and then loaded my newly created service with:
sudo systemctl daemon-reload
sudo systemctl enable mattermost
sudo systemctl start mattermost
With that, I had Mattermost running at (default) port 8065. Personally, I preferred to change that port (because why not) by editing /etc/systemd/system/mattermost.service and restarting the service.
Configure nginx
Well, actually at this point I already had some functioning chat service, right? But I didn’t intend to stop on that - it still required passing the port number and the connection was not secure. To address these issues, first I had to install nginx.
sudo apt-get install nginx -y
After accessing http://1.1.1.1 I could confirm that nginx was working. So now I had to configure it to redirect :80 calls to Mattermost. I’ve created file /etc/nginx/sites-available/mattermost:
server {
server_name chat.mypage.com;
location / {
client_max_body_size 50M;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_pass http://127.0.0.1:8065;
}
}
Now I had to link it as an enabled site, remove default config and restart nginx:
sudo rm /etc/nginx/sites-enabled/default
sudo ln -s /etc/nginx/sites-available/mattermost /etc/nginx/sites-enabled/mattermost
sudo service nginx restart
With that, I had a nicely working server… which still needed some security tweaks.
Securing server
At that moment I’d only adjusted ssh logging in a little bit. I had to add a few more things to introduce minimal security of the server. First I started by making sure that one won’t try to attack ssh by brute force. Such an attacker could be banned with fail2ban.
fail2ban
sudo apt-get install fail2ban -y
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
Now we would like to make sure that ssh configuration point to the right port:
...
[ssh]
...
port = 222
filter = sshd
...
[sshd]
...
port = 222
filter = sshd-ddos
...
sudo service fail2ban restart
AFAIR the rest of the default setup on Debian Jessie was good enough. Then I needed to adjust iptables to filter out any other port scanning attempts.
iptables
I created a file with iptables rules with:
sudo mkdir /etc/iptables
sudo nano /etc/iptables/rules
and filled it with the following content:
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
# Accept any related or established connections
-I INPUT 1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-I OUTPUT 1 -m state --state RELATED,ESTABLISHED -j ACCEPT
# Allow all traffic on the loopback interface
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
# Allow outbound DHCP request - Some hosts (Linode) automatically assign the primary IP
-A OUTPUT -p udp --dport 67:68 --sport 67:68 -j ACCEPT
# Outbound DNS lookups
-A OUTPUT -o eth0 -p udp -m udp --dport 53 -j ACCEPT
# Outbound PING requests
-A OUTPUT -p icmp -j ACCEPT
# Outbound Network Time Protocol (NTP) request
-A OUTPUT -p udp --dport 123 --sport 123 -j ACCEPT
# SSH
-A INPUT -i eth0 -p tcp -m tcp --dport 222 -m state --state NEW -j ACCEPT
# Outbound HTTP
-A OUTPUT -o eth0 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A OUTPUT -o eth0 -p tcp -m tcp --dport 443 -m state --state NEW -j ACCEPT
# Inbound HTTP
-A INPUT -m state --state NEW -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -p tcp --dport 443 -j ACCEPT
COMMIT
It is slightly different from the original file from the tutorial I based my setup on - I had to uncomment outbound DHCP request as I used Linode host and it was essential for the server to work. Another change was allowing inbound HTTP - without it, some Mattermost requests were misbehaving: no message send confirmation, etc.
I had to test these rules before accepting them permanently:
sudo iptables-apply --timeout 60 /etc/iptables/rules
With command above I had 60 seconds to check if I could create new connections to the server. I recommend checking:
- whether a new ssh connection can be made (in another terminal/console),
- whether one can establish a connection to Mattermost server, log in and send a message with confirmation.
If the answer to both questions was true, then I could confirm that iptables were set up correctly and apply these settings permanently. I needed a few tries before it worked as intended, but once I introduced changes described above anything worked as expected. I only needed to add a hook to make sure settings would be applied on network startup:
sudo nano /etc/network/if-pre-up.d/iptables
sudo chmod +x /etc/network/if-pre-up.d/iptables
where /etc/network/if-pre-up.d/iptables was filled with:
#!/bin/sh
iptables-restore < /etc/iptables/rules
TSL and nginx again
Now the only thing I had to take care of was creating a SSL (TLS) certificate and using it to secure the connection. Let’s Encrypt allows us to get a certificate for free so I gave it a shot. First, I added jessie-backports to /etc/apt/sources.list and updated apt cache. With that I was able to install certbot from repositories:
#!/bin/sh
iptables-restore < /etc/iptables/rules
view raw
At this point I already had my domain - I really recommend setting it up early on as I REALLY had to wait about 72 hours before my ISP’s DNS cache had it. And with the settings I want to describe here, I could no longer access the site through IP or Linode’s domain. (A workaround was to move away from my ISP’s DNS cache as it surprisingly worked more reliably and faster, but that’s a tale for another time). I’d like to remind that I assume here that the domain is chat.mypage.com.
Since certbot’s support for nginx is experimental and not turned on by default I used standalone routine - it required me to turn off nginx for a moment, so that certbot could set up a temporary server, and use it for serving content to Let’s Encrypt servers to prove that the domain truly belongs to me (or specifically that the domain points to the server that cerbot is currently run on).
sudo apt-get install certbot -t jessie-backports -y
Sometimes running certbot requires several attempts - to avoid spamming with unnecessary requests, their server has quotas for certificate requests. If your request was denied, just try again later on. At some point, you will succeed. Once I obtained the certificate, I was able to access it in /etc/letsencrypt/live/chat.mypage.com. With that, I could finally make nginx use it to secure HTTP(S) connections. I changed the content of /etc/nginx/sites-available/mattermost to:
sudo service nginx stop
sudo certbot certonly --standalone -d chat.mypage.com
Finally, I started nginx again:
server {
listen 80;
server_name chat.mypage.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
server_name chat.mypage.com;
ssl on;
ssl_certificate /etc/letsencrypt/live/chat.mypage.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/chat.mypage.com/privkey.pem;
ssl_session_timeout 5m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
location / {
gzip off;
proxy_set_header X-Forwarded-Ssl on;
client_max_body_size 50M;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Frame-Options SAMEORIGIN;
proxy_pass http://127.0.0.1:6324;
}
}
to confirm that http://chat.mypage.com redirects to https://chat.mypage.com which in turn uses secure TLS connections.
To make sure that certificate would be renewed every 30 days (Let’s Encrypt gives certificates valid for only 90 days) I also created a cron job:
sudo nano /etc/cron.monthly/certbot_renewal
sudo chmod +x /etc/cron.monthly/certbot_renewal
with content:
#!/bin/sh
certbot renew --standalone --pre-hook "service nginx stop" \
--post-hook "service nginx start" \
--force-renew >> /var/log/certificate_renewal.log 2>&1
Basic security was done.
Back up your data
While the application was ready to use and I could actually start allowing users on the server (after I played around with Mattermost’s settings), I needed to make sure that whatever data would be generated - user’s archives, uploaded files and so on - would be preserved for certain. First thing I could do was enabling backup on Linode - it costs me $2.5 a month and creates a snapshot of my whole node once a week.
But once a week is not enough. What if I wanted to restore content 6 days after the last backup? What if I wanted to move it to another node/server? For that, I needed to find some additional solution.
Upload to AWS S3
I decided to go with AWS S3 storage. Uploading all the config files and user data once a day should give me enough confidence and with AWS’ prices, it is even cheaper than Linode’s backup.
Installation of AWS command line tools is rather simple:
cd /opt
curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
unzip awscli-bundle.zip
sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
rm awscli-bundle.zip
For AWS S3 I needed an Amazon account paired with my credit card (for billings). There, in S3 services settings, I created a new bucket for my backups. Finally, I used Security Credentials for creating keys just for my backups. I used them for creating an aws_upload bash script:
#!/bin/bash
aws_upload () {
AWS_ACCESS_KEY_ID=[my access key] \
AWS_SECRET_ACCESS_KEY=[secret access key] \
AWS_DEFAULT_REGION=[bucket region] \
aws s3 cp "$backup_enc" "s3://my.backup.bucket/$s3_new_backup_name"
}
After sourcing it into my backup script I would be able to upload backup file as backup-from-${current-date}.tar.gz with command like:
backup_enc='backup.tgz.enc'
s3_new_backup_name="backup-from-$(date '+%Y-%m-%d').tgz.enc"
aws_upload
Since it keeps credentials in a separate file I could check backup script into git ignoring aws_upload and avoiding credentials exposure.
Encryption
Because I have limited trust to third-party storages I would still encrypt the backup with something like mcrypt before the upload.
Installation of mcrypt was even faster:
sudo apt-get install mcrypt -y
Since it requires some key for encryption I decided to generate a random one:
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 512 | head -n 1 > ./keyfile
Then I could encrypt/decrypt backups with something like:
cat backup | mcrypt --keyfile ./keyfile > backup_enc
cat backup_enc | mcrypt --decrypt --keyfile ./keyfile > backup
Similarly like with credentials I added the keyfile to ignored so that the key would not be published in a repository.
My backup scripts
Long story short - backup scripts would just copy all config/data files into one place, archive them and publish on AWS S3. The only exception would be Mattermost DB which would be backed up with pqdump to create a dump file which can just be read into the DB like any other PostgreSQL script. To avoid typos in hardcoded paths, I extracted all paths into variables and wrote a few helpers to make the scripts more readable (to me :D).
.common.sh:
#!/bin/bash
# Task runners
run_task () {
echo " - $1"
eval "$2" > /dev/null
if [ $? -eq 0 ]; then
echo " [done]"
else
echo " [failed]"
exit -1
fi
}
run_grouped () {
echo " - $1"
eval "$2"
echo " [done]"
}
run_subtask () {
echo " - $1"
eval "$2" > /dev/null
if [ $? -eq 0 ]; then
echo " [done]"
else
echo " [failed]"
exit -1
fi
}
# Common directories and files
encryption_key="$(dirname $0)/keyfile"
backup_dir=/var/backup
backup_rel=`python -c "import os.path; print os.path.relpath('$backup_dir', '/')"`
backup_tar=/var/backup.tgz
backup_enc=/var/backup.tgz.enc
postgresql_dir=/etc/postgresql/9.4
postgresql_config_original="$postgresql_dir/main/postgresql.conf"
postgresql_config_backup="$backup_dir/postgresql.conf"
postgresql_hba_original="$postgresql_dir/main/pg_hba.conf"
postgresql_hba_backup="$backup_dir/pg_hba.conf"
mm_dir=/opt/mattermost
mm_config_original="$mm_dir/config/config.json"
mm_config_backup="$backup_dir/config.json"
nginx_config_original=/etc/nginx/sites-available/mattermost
nginx_config_backup="$backup_dir/nginx.conf"
ssh_config_original=/etc/ssh/sshd_config
ssh_config_backup="$backup_dir/sshd.config"
iptables_config_original=/etc/iptables/rules
iptables_config_backup="$backup_dir/iptables.rules"
cron_backup_original=/etc/cron.daily/complete_backup
cron_backup_backup="$backup_dir/complete_backup"
cron_certbot_reneval_original=/etc/cron.monthly/certbot_renewal
cron_certbot_reneval_backup="$backup_dir/certbot_renewal"
network_if_pre_up_iptables_original=/etc/network/if-pre-up.d/iptables
network_if_pre_up_iptables_backup="$backup_dir/iptables.if-pre-up"
admins_original=/home
admins_backup="$backup_dir/admins"
database_backup="$backup_dir/db.dump"
local_data_original="$mm_dir/data"
local_data_backup="$backup_dir/files"
certificates_original=/etc/letsencrypt/live/chat.mypage.com
certificates_backup="$backup_dir/certificates"
s3_new_backup_name="backup-from-$(date '+%Y-%m-%d').tgz.enc"
# S3 Upload functions
# Should follow this convention:
#
# aws_upload () {
# AWS_ACCESS_KEY_ID=[my access key] \
# AWS_SECRET_ACCESS_KEY=[secret access key] \
# AWS_DEFAULT_REGION=[bucket region] \
# aws s3 cp "$backup_enc" "s3://chat.mypage.com/$s3_new_backup_name"
# }
source "$(dirname $0)/aws_upload" # should define aws_upload
Then backup is just a matter of defining tasks:
#!/bin/bash
source "$(dirname $0)/.commons.sh"
echo "Backing up Server"
if [ ! -d $backup_dir ]; then
mkdir $backup_dir -p > /dev/null
fi
run_grouped "backing up configs" "$(cat <<-GROUPED
run_subtask "backing up postgres config" "$(cat <<-TASK
cp -fp "$postgresql_config_original" "$postgresql_config_backup" && \
cp -fp "$postgresql_hba_original" "$postgresql_hba_backup"
TASK
)"
run_subtask "backing up MatterMost config" "$(cat <<-TASK
cp -fp "$mm_config_original" "$mm_config_backup"
TASK
)"
run_subtask "backing up Nginx config" "$(cat <<-TASK
cp -fp "$nginx_config_original" "$nginx_config_backup"
TASK
)"
run_subtask "backing up sshd settings" "$(cat <<-TASK
cp -fp "$ssh_config_original" "$ssh_config_backup"
TASK
)"
run_subtask "backing up iptables" "$(cat <<-TASK
cp -fp "$iptables_config_original" "$iptables_config_backup"
TASK
)"
run_subtask "backing up cron jobs" "$(cat <<-TASK
cp -fp "$cron_backup_original" "$cron_backup_backup" && \
cp -fp "$cron_certbot_reneval_original" "$cron_certbot_reneval_backup"
TASK
)"
run_subtask "backing up network jobs" "$(cat <<-TASK
cp -fp "$network_if_pre_up_iptables_original" \
"$network_if_pre_up_iptables_backup"
TASK
)"
GROUPED
)"
run_task "backing up admins" "$(cat <<-TASK
if [ ! -d "$backup_dir/admins" ]; then
mkdir "$backup_dir/admins" -p > /dev/null
fi
cp -rfp "$admins_original"/* "$admins_backup"
TASK
)"
run_grouped "backing up data" "$(cat <<-GROUPED
run_subtask "backing up database" "$(cat <<-TASK
sudo -u postgres pg_dump mattermost > "$database_backup"
TASK
)"
run_subtask "backing up local data" "$(cat <<-TASK
if [ ! -d "$local_data_backup" ]; then
mkdir "$local_data_backup" -p > /dev/null
fi
cp -rfp "$local_data_original"/* "$local_data_backup"
TASK
)"
run_subtask "backing up certificates" "$(cat <<-TASK
if [ ! -d "$certificates_backup" ]; then
mkdir "$certificates_backup" -p > /dev/null
fi
cp -rfp "$certificates_original"/* "$certificates_backup"
TASK
)"
GROUPED
)"
run_grouped "upload backup to S3" "$(cat <<-GROUPED
run_subtask "compressing backup" "$(cat <<-TASK
tar -pcvzf "$backup_tar" -C / "$backup_rel"
TASK
)"
run_subtask "encrypting backup" "$(cat <<-TASK
cat "$backup_tar" | \
mcrypt --keyfile "$encryption_key" > "$backup_enc" 2> /dev/null
TASK
)"
run_subtask "uploading backup" "$(cat <<-TASK
mjk_upload && jelz_upload
TASK
)"
GROUPED
)"
echo "[done]"
Similarly restoring:
#!/bin/bash
source "$(dirname $0)/.commons.sh"
echo "Restoring Server setup"
run_grouped "restoring backup files" "$(cat <<-GROUPED
run_subtask "decrypting backup" "$(cat <<-TASK
cat "$backup_enc" | \
mcrypt --decrypt --keyfile "$encryption_key" > "$backup_tar"
TASK
)"
run_subtask "restoring backup directory" "$(cat <<-TASK
ls "$backup_dir" > /dev/null || tar -pxvfz "$backup_tar" -C /
TASK
)"
GROUPED
)"
run_grouped "restoring configs" "$(cat <<-GROUPED
run_subtask "Restoring postgres config" "$(cat <<-TASK
cp -fp "$postgresql_config_backup" "$postgresql_config_original" && \
cp -fp "$postgresql_hba_backup" "$postgresql_hba_original"
TASK
)"
run_subtask "restoring MatterMost config" "$(cat <<-TASK
cp -fp "$mm_config_backup" "$mm_config_original"
TASK
)"
run_subtask "restoring Nginx config" "$(cat <<-TASK
cp -fp "$nginx_config_backup" "$nginx_config_original"
TASK
)"
run_subtask "restoring sshd settings" "$(cat <<-TASK
cp -fp "$ssh_config_backup" "$ssh_config_original"
TASK
)"
run_subtask "restoring iptables" "$(cat <<-TASK
cp -fp "$iptables_config_backup" "$iptables_config_original"
TASK
)"
run_subtask "restoring cron jobs" "$(cat <<-TASK
cp -fp "$cron_backup_backup" "$cron_backup_original" && \
cp -fp "$cron_certbot_reneval_backup" "$cron_certbot_reneval_original"
TASK
)"
run_subtask "backing up network jobs" "$(cat <<-TASK
cp -fp "$network_if_pre_up_iptables_backup" \
"$network_if_pre_up_iptables_original"
TASK
)"
GROUPED
)"
run_task "restoring admins" "$(cat <<-TASK
cp -rfp "$admins_backup"/* "$admins_original"
TASK
)"
run_grouped "restoring data" "$(cat <<-GROUPED
run_subtask "restoring database" "$(cat <<-TASK
sudo -u postgres \
psql --set ON_ERROR_STOP=on mattermost < "$database_backup" 2> /dev/null
TASK
)"
run_subtask "restoring local data" "$(cat <<-TASK
cp -rfp "$local_data_backup"/* "$local_data_original"
TASK
)"
run_subtask "restoring certificates" "$(cat <<-TASK
if [ ! -d "$certificates_original" ]; then
mkdir "$certificates_original" -p > /dev/null
fi
cp -rfp "$certificates_backup"/* "$certificates_original"
TASK
)"
GROUPED
)"
echo "[done]"
The last thing to do is write a cron job to perform a backup once a day:
sudo nano /etc/cron.daily/complete_backup
sudo chmod +x /etc/cron.daily/complete_backup
#!/bin/sh
/opt/backup/backup.sh >> /var/log/complete_backup.log 2>&1
Mattermost tweaks
Out of the box, Mattermost would have some generic site name. Of all settings that we would like to tweak after installation, that is the only one we can only access by editing /opt/mattermost/config.config.json (TeamSettings/SiteName). Any other config that would be interesting to us could be accessed via the System Console window of Mattermost. That change would require us to restart Mattermost manually to take effect.
Once we change the site name to something more preferable we can also change the title of notification emails (if we’ll use them) and admin’s email address. If we decide to allow push notifications for mobile devices (Android and iOS clients), then we’ll have to decide on the server - we can use test server (TPNS) which does not encrypt our messages (then I would recommend sending generic description with user and channel names) or set up an encrypted push notification server manually - problem with the later is that official clients only support Mattermost servers. So one has to either pay for Mattermost’s encrypted push notification service or fork clients and release one’s own version.
I would also recommend skimming through other options to e.g. regenerate hashes for links and turning off public user registration. If we decide to have several unrelated teams on one server, we might also disable option for cross-team communication.
As for customization, for now, we can only set up global default localization options. An option to change default fonts or themes is still an open ticket on the issue tracker.
Mattermost vs Slack
How does Mattermost actually compare to Slack in action? I won’t deny it - for me, it looks less polished. Of 10 possible fonts, only 2 (Open Sans and Roboto) seems to handle Polish characters correctly and 1 of them (Roboto) seems to fail when we use bold (e.g. when in notifications from some channel). We cannot change the size of the font, and by default, they seem a little bit too small (after zooming in, they seem to be a bit too big). The default theme has annoying sets of colors and I am unable to change these settings without manually editing source code files and preparing my own build.
Some other potentially useful options are also lacking. I am unable to configure the server so that only admin could invite users. Instead, there is a registration link with a hash which could be hidden from the landing page and (optionally) requiring email registration from a fixed domain. As an admin, I can only block any registration to prevent users from sending it to whoever they want. Alternatively, I can disable any registration and use Mattermost command line interface for creating my users (probably the best solution for enterprises). And newly registered users cannot be automatically invited into a predefined list of channels.
Mobile clients seem to be build using an embedded browser - I only tested the Android client, and it has significant startup overhead before I’ll see that my chat is even loading. I attribute it to the browser initialization. When I receive a notification and click it to go to the message, client reloads the browser, even if I just have the said conversation opened. Currently, the client does not have an option to open menus by sliding the screen to the side, though that’s exactly how they open once we click on a menu icon. You can also only have one server configured in your app. (And you cannot remove it easily - when I switched from HTTP to HTTPS the app stuck on an infinite server load and I had to reinstall it in order to enter a new URL). Overall, I got the feeling that the mobile clients are slow and heavy (at least Android one is. AFAIK iOS client is built the same way so I assume it shares it’s brother’s issues).
Other quirks that brought my attention was no ability to add reaction emoticons (we use them on Slack heavily) and worse, viewing thumbnails for assigned files - however the former should be solved with 3.5 release. When you remove a message ugly message removed notice would stay (it disappears after reloading but still). If you spent a lot of time on Slack, these small quirks will add up and you won’t be able to help but notice the difference in the user experience. So does Mattermost have any advantages besides the pricing?
I’d say yes. Even if currently each user would have to apply customization settings manually, they are still able to change the color of almost everything. (I weren’t able to locate the option to disable my own messages’ highlighting - each message you send has a different background than all the rest. This color is also used for highlighting the message your cursor is over it.) Having the ability to respond to a message directly (kind of like on Reddit) might also be a nice thing if 2 parallel discussions emerge on one channel.
When looking at the message bar you’ll notice that it lacks Slack’s plus button, it only has an assignment. There is no distinction on a photo or generic file upload (which is a minus) and there is no option to publish a snippet. The latter is not that much of a problem, though, as Mattermost has better markdown support than Slack, so you can paste your code like on GitHub:
val s = Seq()
and with the ability to respond to a message it works similar to commenting assignments.
It is also useful to share server for several teams because if you want those teams to communicate, you can still allow that. And anyone on the server could then talk to anyone else, without the need to register on each team separately.
Having access to the whole archive out-of-the-box and without additional fees is also a big plus. And if you are really concerned about the security, privacy, and ownership of your data, having your own secured server will always beat trusting third-party vendor locked-in solutions.
Summary
Setting up my own Mattermost server was a great learning experience. And a lot of fun. Though the service is less polished than a Slack, it suits my use case and my pocket. I am looking forward to new releases which will bring new functionalities and get rid of aforementioned quirks.
I cannot recommend this solution to everyone - limited ability to control who can register might be a deal breaker for many companies. However, if you are managing a small team (of friends?) or your company is OK with manually managing users via command line, or the Mattermost server is only available in the intranet or via VPN it might be a quite good Slack alternative. In such a case, though, you should consider subscribing for paid push-notifications to make sure they are distributed via secured connections. Or, if your company is reaally concerned about owning everything, forking mobile clients to use your own dedicated push-notification server (for a while forking the platform might be the only way of providing your own color-scheme defaults). Or perhaps just go straight to Mattermost Enterprise Edition which should ease all those issues and add some additional perks like rebranding. But here I wanted to focus on Team Edition.
In the end, it’s all about the tradeoffs. If your primary concerns are flawless user experience and polished UI - go with Slack. If you want to go an extra mile to make sure that you actually own your data - Mattermost is the way to go.
Links
These are the sources I read while configuring my server. I admit that a major part of my text is just a creative ripoff ;)
- Production Install on Debian Jessie - complete manual on setting up Mattermost on a server. I just skipped all settings required by 3-servers’ setup and replaced cloning Let’s Encrypt with certbot already available in
jessie-backports. - Securing a Linux Server - a source I used when I configured SSH, iptables, and fail2ban.
Articles I just read through to see if I’ve got everything covered: