WIP: Debian installation instructions

docs/changelog.rst

@@ -21,9 +21,7 @@ Changelog
 * [feature] can now import artist and releases from youtube and musicbrainz.
   This requires a YouTube API key for the search
 * [breaking] we now check for user permission before serving audio files, which requires
-a specific configuration block in your reverse proxy configuration:
-
-.. code-block::
+  a specific configuration block in your reverse proxy configuration::
 
     location /_protected/media {
         internal;

docs/conf.py

@@ -17,10 +17,12 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 #
-# import os
-# import sys
-# sys.path.insert(0, os.path.abspath('.'))
+import os
+import sys
 
+sys.path.insert(0, os.path.abspath('../api'))
+
+import funkwhale_api  # NOQA
 
 # -- General configuration ------------------------------------------------
 
@@ -55,9 +57,11 @@ author = 'Eliot Berriot'
 # built documents.
 #
 # The short X.Y version.
-version = '0.1'
+# version = funkwhale_api.__version__
+# @TODO use real version here
+version = 'feature/22-debian-installation'
 # The full version, including alpha/beta/rc tags.
-release = '0.1'
+release = version
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -152,6 +156,3 @@ texinfo_documents = [
      author, 'funkwhale', 'One line description of project.',
      'Miscellaneous'),
 ]
-
-
-

docs/importing-music.rst

@@ -5,10 +5,18 @@ From music directory on the server
 ----------------------------------
 
 You can import music files in funkwhale assuming they are located on the server
-and readable by the funkwhale application.
+and readable by the funkwhale application. Your music files should contain at
+least an ``artist``, ``album`` and ``title`` tags.
 
-Assuming your music is located at ``/music`` and your music files contains at
-least an ``artist``, ``album`` and ``title`` tag, you can import those tracks as follows:
+You can import those tracks as follows, assuming they are located in
+``/srv/funkwhale/data/music``:
+
+.. code-block:: bash
+
+    python api/manage.py import_files "/srv/funkwhale/data/music/**/*.ogg" --recursive --noinput
+
+When you use docker, the ``/srv/funkwhale/data/music`` is mounted from the host
+to the ``/music`` directory on the container:
 
 .. code-block:: bash
 
@@ -17,6 +25,7 @@ least an ``artist``, ``album`` and ``title`` tag, you can import those tracks as
 For the best results, we recommand tagging your music collection through
 `Picard <http://picard.musicbrainz.org/>`_ in order to have the best quality metadata.
 
+
 .. note::
 
     This command is idempotent, meaning you can run it multiple times on the same
@@ -26,6 +35,18 @@ For the best results, we recommand tagging your music collection through
 
     At the moment, only OGG/Vorbis and MP3 files with ID3 tags are supported
 
+.. note::
+
+    The --recursive flag will work only on Python 3.5+, which is the default
+    version When using Docker or Debian 9. If you use an older version of Python,
+    remove the --recursive flag and use more explicit import patterns instead::
+
+        # this will only import ogg files at the second level
+        "/srv/funkwhale/data/music/*/*.ogg"
+        # this will only import ogg files in the fiven directory
+        "/srv/funkwhale/data/music/System-of-a-down/*.ogg"
+
+
 
 Getting demo tracks
 ^^^^^^^^^^^^^^^^^^^
@@ -34,10 +55,10 @@ If you do not have any music on your server but still want to test the import
 process, you can call the following methods do download a few albums licenced
 under creative commons (courtesy of Jamendo):
 
-.. code-block:: bash
+.. parsed-literal::
 
-    curl -L -o download-tracks.sh "https://code.eliotberriot.com/funkwhale/funkwhale/raw/master/demo/download-tracks.sh"
-    curl -L -o music.txt "https://code.eliotberriot.com/funkwhale/funkwhale/raw/master/demo/music.txt"
+    curl -L -o download-tracks.sh "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/demo/download-tracks.sh"
+    curl -L -o music.txt "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/demo/music.txt"
     chmod +x download-tracks.sh
     ./download-tracks.sh music.txt
 

docs/installation/debian.rst

@@ -0,0 +1,280 @@
+Debian installation
+===================
+
+.. note::
+
+    this guide targets Debian 9, which is the latest debian, but should work
+    similarly on Debian 8.
+
+External dependencies
+---------------------
+
+The guides will focus on installing funkwhale-specific components and
+dependencies. However, funkwhale requires a
+:doc:`few external dependencies <./external_dependencies>` for which
+documentation is outside of this document scope.
+
+Install utilities
+-----------------
+
+You'll need a few utilities during this guide that are not always present by
+default on system. You can install them using:
+
+.. code-block:: shell
+
+    sudo apt-get update
+    sudo apt-get install curl python3-venv git unzip
+
+
+Layout
+-------
+
+All funkwhale-related files will be located under ``/srv/funkwhale`` apart
+from database files and a few configuration files. We will also have a
+dedicated ``funwhale`` user to launch the processes we need and own those files.
+
+You are free to use different values here, just remember to adapt those in the
+next steps.
+
+.. _create-funkwhale-user:
+
+Create the user and the directory:
+
+.. code-block:: shell
+
+    sudo adduser --system --home /srv/funkwhale funkwhale
+    cd /srv/funkwhale
+
+Log in as the newly created user from now on:
+
+.. code-block:: shell
+
+    sudo -u funkwhale -H bash
+
+Now let's setup our directory layout. Here is how it will look like::
+
+    .
+    ├── config      # config / environment files
+    ├── api         # api code of your instance
+    ├── data        # persistent data, such as music files
+    ├── front       # frontend files for the web user interface
+    └── virtualenv  # python dependencies for funkwhale
+
+Create the aforementionned directories:
+
+.. code-block:: shell
+
+    mkdir -p config api data/static data/media data/music front
+
+The ``virtualenv`` directory is a bit special and will be created separately.
+
+Download latest funkwhale release
+----------------------------------
+
+Funkwhale is splitted in two components:
+
+1. The API, which will handle music storage and user accounts
+2. The frontend, that will simply connect to the API to interact with its data
+
+Those components are packaged in subsequent releases, such as 0.1, 0.2, etc.
+You can browse the :doc:`changelog </changelog>` for a list of available releases
+and pick the one you want to install, usually the latest one should be okay.
+
+In this guide, we'll assume you want to install the latest version of funkwhale,
+which is |version|:
+
+First, we'll download the latest api release.
+
+.. parsed-literal::
+
+    curl -L -o "api-|version|.zip" "https://code.eliotberriot.com/funkwhale/funkwhale/-/jobs/artifacts/|version|/download?job=build_api"
+    unzip "api-|version|.zip" -d extracted
+    mv extracted/api api
+    rmdir extracted
+
+
+Then we'll download the frontend files:
+
+.. parsed-literal::
+
+    curl -L -o "front-|version|.zip" "https://code.eliotberriot.com/funkwhale/funkwhale/-/jobs/artifacts/|version|/download?job=build_front"
+    unzip "front-|version|.zip" -d extracted
+    mv extracted/front .
+    rmdir extracted
+
+You can leave the ZIP archives in the directory, this will help you know
+which version you've installed next time you want to upgrade your installation.
+
+System dependencies
+-------------------
+
+First, switch to the api directory:
+
+.. code-block:: shell
+
+    cd api
+
+A few OS packages are required in order to run funkwhale. The list is available
+in ``api/requirements.apt`` or by running
+``./install_os_dependencies.sh list``.
+
+.. note::
+
+    Ensure you are running the next commands as root or using sudo
+    (and not as the funkwhale) user.
+
+You can install those packages all at once:
+
+.. code-block:: shell
+
+    ./install_os_dependencies.sh install
+
+From now on you can switch back to the funkwhale user.
+
+Python dependencies
+--------------------
+
+Go back to the base directory:
+
+.. code-block:: shell
+
+    cd /srv/funkwhale
+
+To avoid collisions with other software on your system, Python dependencies
+will be installed in a dedicated
+`virtualenv <https://docs.python.org/3/library/venv.html>`_.
+
+First, create the virtualenv:
+
+.. code-block:: shell
+
+    python3 -m venv /srv/funkwhale/virtualenv
+
+This will result in a ``virtualenv`` directory being created in
+``/srv/funkwhale/virtualenv``.
+
+In the rest of this guide, we'll need to activate this environment to ensure
+dependencies are installed within it, and not directly on your host system.
+
+This is done with the following command:
+
+.. code-block:: shell
+
+    source /srv/funkwhale/virtualenv/bin/activate
+
+Finally, install the python dependencies:
+
+.. code-block:: shell
+
+    pip install wheel
+    pip install -r api/requirements.txt
+
+.. important::
+
+    further commands involving python should always be run after you activated
+    the virtualenv, as described earlier, otherwise those commands will raise
+    errors
+
+
+Environment file
+----------------
+
+You can now start to configure funkwhale. The main way to achieve that is by
+adding an environment file that will host settings that are relevant to your
+installation.
+
+Download the sample environment file:
+
+.. parsed-literal::
+
+    curl -L -o config/.env "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/env.prod.sample"
+
+You can then edit it: the file is heavily commented, and the most relevant
+configuration options are mentionned at the top of the file.
+
+Especially, populate the ``DATABASE_URL`` and ``CACHE_URL`` values based on
+how you configured your PostgreSQL and Redis servers in
+:doc:`external dependencies <./external_dependencies>`.
+
+
+When you want to run command on the API server, such as to create the
+database or compile static files, you have to ensure you source
+the environment variables.
+
+This can be done like this::
+
+    export $(cat config/.env | grep -v ^# | xargs)
+
+The easier thing to do is to store this in a script::
+
+    cat > /srv/funkwhale/load_env <<'EOL'
+    #!/bin/bash
+    export $(cat /srv/funkwhale/config/.env | grep -v ^# | xargs)
+    EOL
+    chmod +x /srv/funkwhale/load_env
+
+You should now be able to run the following to populate your environment
+variables easily:
+
+.. code-block:: shell
+
+    source /srv/funkwhale/load_env
+
+.. note::
+
+    Remember to source ``load_env`` whenever you edit your .env file.
+
+Database setup
+---------------
+
+You should now be able to import the initial database structure:
+
+.. code-block:: shell
+
+    python api/manage.py migrate
+
+This will create the required tables and rows.
+
+.. note::
+
+    You can safely execute this command any time you want, this will only
+    run unapplied migrations.
+
+
+Create an admin account
+-----------------------
+
+You can then create your first user account:
+
+.. code-block:: shell
+
+    python api/manage.py createsuperuser
+
+If you ever want to change a user's password from the command line, just run:
+
+.. code-block:: shell
+
+    python api/manage.py changepassword <user>
+
+Collect static files
+--------------------
+
+Static files are the static assets used by the API server (icon PNGs, CSS, etc.).
+We need to collect them explicitely, so they can be served by the webserver:
+
+.. code-block:: shell
+
+    python api/manage.py collectstatic
+
+This should populate the directory you choose for the ``STATIC_ROOT`` variable
+in your ``.env`` file.
+
+Systemd unit file
+------------------
+
+See :doc:`./systemd`.
+
+Reverse proxy setup
+--------------------
+
+See :ref:`reverse-proxy <reverse-proxy-setup>`.

docs/installation/docker.rst

@@ -7,17 +7,17 @@ First, ensure you have `Docker <https://docs.docker.com/engine/installation/>`_
 
 Download the sample docker-compose file:
 
-.. code-block:: bash
+.. parsed-literal::
 
     mkdir -p /srv/funkwhale
     cd /srv/funkwhale
-    curl -L -o docker-compose.yml "https://code.eliotberriot.com/funkwhale/funkwhale/raw/master/deploy/docker-compose.yml"
+    curl -L -o docker-compose.yml "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/docker-compose.yml"
 
 Create your env file:
 
-.. code-block:: bash
+.. parsed-literal::
 
-    curl -L -o .env "https://code.eliotberriot.com/funkwhale/funkwhale/raw/master/deploy/env.prod.sample"
+    curl -L -o .env "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/env.prod.sample"
 
 Ensure to edit it to match your needs (this file is heavily commented)
 

docs/installation/external_dependencies.rst

@@ -0,0 +1,62 @@
+External dependencies
+=====================
+
+
+.. note::
+
+    Those dependencies are handled automatically if you are
+    :doc:`deploying using docker <./docker>`
+
+Database setup (PostgreSQL)
+---------------------------
+
+Funkwhale requires a PostgreSQL database to work properly. Please refer
+to the `PostgreSQL documentation <https://www.postgresql.org/download/>`_
+for installation instructions specific to your os.
+
+On debian-like systems, you would install the database server like this:
+
+.. code-block:: shell
+
+    sudo apt-get install postgresql
+
+The remaining steps are heavily inspired from `this Digital Ocean guide <https://www.digitalocean.com/community/tutorials/how-to-set-up-django-with-postgres-nginx-and-gunicorn-on-ubuntu-16-04>`_.
+
+Open a database shell:
+
+.. code-block:: shell
+
+    sudo -u postgres psql
+
+Create the project database and user:
+
+.. code-block:: shell
+
+    CREATE DATABASE funkwhale;
+    CREATE USER funkwhale;
+    GRANT ALL PRIVILEGES ON DATABASE funkwhale TO funkwhale;
+
+Assuming you already have :ref:`created your funkwhale user <create-funkwhale-user>`,
+you should now be able to open a postgresql shell:
+
+.. code-block:: shell
+
+    sudo -u funkwhale -H psql
+
+Cache setup (Redis)
+-------------------
+
+Funkwhale also requires a cache server:
+
+- To make the whole system faster, by caching network requests or database
+  queries
+- To handle asynchronous tasks such as music import
+
+On debian-like distributions, a redis package is available, and you can
+install it:
+
+.. code-block:: shell
+
+    sudo apt-get install redis-server
+
+This should be enough to have your redis server set up.

docs/installation/index.rst

@@ -18,7 +18,10 @@ Available installation methods
 .. toctree::
    :maxdepth: 1
 
+   external_dependencies
+   debian
    docker
+   systemd
 
 
 .. _frontend-setup:
@@ -33,10 +36,10 @@ Frontend setup
 
 Files for the web frontend are purely static and can simply be downloaded, unzipped and served from any webserver:
 
-.. code-block:: bash
+.. parsed-literal::
 
     cd /srv/funkwhale
-    curl -L -o front.zip "https://code.eliotberriot.com/funkwhale/funkwhale/builds/artifacts/master/download?job=build_front"
+    curl -L -o front.zip "https://code.eliotberriot.com/funkwhale/funkwhale/builds/artifacts/|version|/download?job=build_front"
     unzip front.zip
 
 .. _reverse-proxy-setup:
@@ -58,8 +61,8 @@ Ensure you have a recent version of nginx on your server. On debian-like system,
 
 Then, download our sample virtualhost file:
 
-.. code-block:: bash
+.. parsed-literal::
 
-    curl -L -o /etc/nginx/sites-enabled/funkwhale.conf "https://code.eliotberriot.com/funkwhale/funkwhale/raw/master/deploy/nginx.conf"
+    curl -L -o /etc/nginx/sites-enabled/funkwhale.conf "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/nginx.conf"
 
 Ensure static assets and proxy pass match your configuration, and check the configuration is valid with ``nginx -t``. If everything is fine, you can restart your nginx server with ``service nginx restart``.

docs/installation/systemd.rst

@@ -0,0 +1,42 @@
+Systemd configuration
+----------------------
+
+Systemd offers a convenient way to manage your funkwhale instance if you're
+not using docker.
+
+We'll see how to setup systemd to proprely start a funkwhale instance.
+
+First, download the sample unitfiles:
+
+.. parsed-literal::
+
+    curl -L -o "/etc/systemd/system/funkwhale.target" "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/funkwhale.target"
+    curl -L -o "/etc/systemd/system/funkwhale-server.service" "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/funkwhale-server.service"
+    curl -L -o "/etc/systemd/system/funkwhale-worker.service" "https://code.eliotberriot.com/funkwhale/funkwhale/raw/|version|/deploy/funkwhale-worker.service"
+
+This will download three unitfiles:
+
+- ``funkwhale-server.service`` to launch the funkwhale web server
+- ``funkwhale-worker.service`` to launch the funkwhale task worker
+- ``funkwhale.target`` to easily stop and start all of the services at once
+
+You can of course review and edit them to suit your deployment scenario
+if needed, but the defaults should be fine.
+
+Once the files are downloaded, reload systemd:
+
+.. code-block:: shell
+
+    systemctl daemon-reload
+
+And start the services:
+
+.. code-block:: shell
+
+    systemctl start funkwhale.target
+
+You can check the statuses of all processes like this:
+
+.. code-block:: shell
+
+    systemctl status funkwhale-\*