Serving IOP4 in production ========================== .. _production_web_server: Setting up a production web server ---------------------------------- .. warning:: This section requires some familiarity with system administration and web server configuration. IOP4 comes with the iop4site project to allow a fast and easy way of serving results. However, it should only be used for **local** debugging and not offered to the public. For production use, you should use a proper web server. Here we provide some details on how to set up IOP4 with nginx and gunicorn. This guide is not complete and should be taken only as a starting point, since setting up a real server is a complex task and depends on how you want to integrate IOP4 with your existing infrastructure. This setup has been tested on Ubuntu Server 22.04 LTS. You need to install nginx and gunicorn (within your virtual environment). Example nginx site configuration (to be placed at ``/etc/nginx/sites-available/``) .. literalinclude:: production_example/nginx_example_site :language: nginx where you should replace ``domain``, ``domain2`` with the domain names that you will be using, and ``/path/to/static/`` should be accessible by the user running nginx (e.g. under ``/var/www/html/mysite/static/``). Example gunicorn socket configuration (to be placed at ``/etc/systemd/system/``) .. literalinclude:: production_example/gunicorn.socket :language: ini Example gunicorn service configuration (to be placed at ``/etc/systemd/system/``) .. literalinclude:: production_example/gunicorn.service :language: ini where ``/path/to/your/django/site`` is the path to your Django site project (your modified `iop4site` or a different one) and ``/path/to/gunicorn/executable`` is the path to the gunicorn executable (usually within your virtual environment, you can get it by typing ``which gunicorn`` when your virtual environment is activated). The most important settings to change in your new ``settings.py`` file within your Django project are: .. code-block:: python SECRET_KEY = "new generated key" # SECURITY WARNING: don't run with debug turned on in production! DEBUG = True INTERNAL_IPS = [] ALLOWED_HOSTS = ["domain", "domain2"] # Configure static files to the path served by nginx STATIC_ROOT = '/path/to/static/' .. _production_cron_job: Creating a cron job for reducing new observations ------------------------------------------------- You might be interested in creating a cron job that routinely reduces new observations every morning. Here we provide an example. This assumes you are working in a linux system. Other OS provide different methods to create jobs. Use ``crontab -e`` to edit your crontab file and add a line like the following .. code-block:: text 00 08 * * * /home/vhega/run_iop4_daily.sh > /home/vhega/run_iop4_daily.log 2>&1 Then, create a file ``run_iop4_daily.sh``, give it execution permissions (``chmod +x run_iop4_daily.sh``) and add the following content: .. code-block:: bash #!/usr/bin/bash # save the current datetime printf -v date '%(%Y-%m-%d_%H%M)T' -1 echo "#########################################" echo "Run daily job for IOP4: $date" echo "#########################################" . /home/vhega/miniconda3/bin/activate iop4 # make sure all files created by iop4 are editable by the current user only umask 0022 # Run iop4 for new observations (i.e. last night) iop4 --discover-missing -o log_file=/home/vhega/iop4data/logs/daily_$date.log # save return code rc=$? # Create and send a summary of the results for last night, passing the return code iop4-night-summary --fromaddr '{{YOUR SENDER ADDRESS}}' \ --mailto '{{ADDRESS 1}},{{ADDRESS 2}},{{ADDRESS 3}}' \ --contact-name '{{CONTACT NAME}}' \ --contact-email '{{CONTACT EMAIL}}' \ --site-url '{{DEPLOYMENT SITE URL}}' \ --saveto "/home/vhega/iop4data/logs/daily_$date.html" --rc $rc The above script will run iop4 every morning, disovering and proccessing new observations. The last command is optional, and will send an email with a summary with the results from last night to the specified email addresses. The cron job does not need to be run on a daily basis, and you can run it whenever you expect new observations to become available in the telescope remote archives (e.g. every few hours). Alternatively, you can pass an argument to ``iop4-night-summary`` specifying the night that you want to generate the summary for. The email will be in HTML format (viewable in any modern browser or email client), and can optionally be saved to any path. You can indicate the url of your deployed site so the links in the email (e.g. for files with error) point directly to the corresponding page in the iop4 web interface or admin site. Note: for iop4-night-summary to be able to send an email, your computer needs a local SMTP server (e.g. postfix). .. _production_share_datadir: Sharing the data directory with other system users -------------------------------------------------- If you are running IOP4 in a server with multiple users, and have created an user to run the IOP4 pipeline as a service, you might be interested in making the local archive available to other users, so they can process the data independently. By default, IOP4 will remove write permissions on raw files, protecting them from accidental modification. For example, you can link your raw directory to the service account raw directory, .. code-block:: bash ln -s /home/iop4user/.iop4data/raw ~/home/myuser/.iop4data/raw The other directories can also be linked, but keep in mind that other users might not be able to modify and reprocess reduced files (they will still be able to inspect them). You will still need to create your database following the installation instructions. Multiple users can also share the same data directory, but this is not recommended. You should not confuse system users (which can run the iop4 pipeline) with IOP4 portal users, that can access and inspect data from the web interface. These should be created only after following See :ref:`Setting up a production web server ` (the debug web server is not recommended for multiple users).