Skip to content
Snippets Groups Projects
Commit dceedb8a authored by sheepmax's avatar sheepmax
Browse files

Moved docker files to separate folder; Updated READMEs; Renamed shell scripts;...

Moved docker files to separate folder; Updated READMEs; Renamed shell scripts; Changed default interpreter for scripts to bash; No longer start http server by default
parent a0b58d95
No related branches found
No related tags found
2 merge requests!37Publish current website,!23Build and deploy with Docker
Pipeline #205476 passed
File deleted
......@@ -2,4 +2,4 @@
# If applying this setup to a git repo that is already initialized, you should run these two commands:
# git rm --cached -r .
# git reset --hard
* text=auto eol=lf
\ No newline at end of file
*.sh eol=lf
......@@ -3,6 +3,9 @@ __pycache__/
*.py[cod]
*$py.class
# MacOS directory information
.DS_Store
# C extensions
*.so
......@@ -129,4 +132,4 @@ dmypy.json
.pyre/
# Jupyter Book
_build/
\ No newline at end of file
_build/
This diff is collapsed.
#!/bin/sh
#!/bin/bash
set -euo pipefail
# Build the jupyter book, everything else is post-processing
jupyter-book build book
......@@ -6,31 +8,40 @@ jupyter-book build book
# Note: the structure of thebe_lite mimicks where thing are needed in the html folder
cp book/thebe_lite/* book/_build/html/ -r
# Check whether python has the alias 'python' or 'python3'
if command -v python > /dev/null 2>&1
then
python_command="python"
else
if command -v python3 > /dev/null 2>&1
then
python_command="python3"
else
echo "Could not find python, cannot start webserver."
exit 1
fi
fi
# Copy all non notebook, markdown or build files into the build for local access in pyodide etc.
# The commands do as follows:
# 1. find: finds all paths in book/, filtering to see if they are files (test -f returns 0)
# 1. find: finds all paths in book/, filtering to see if they are files
# 2. grep: remove all files which begin in book/_, or have a .md/.ipynb extension or are have thebe_lite in their path
# 3. cut: remove the first 5 letters of each path, this corresponds to book/
# 4. xargs: will execute the script inside the quotes
# 5. grep: finds a file's parent's path by matching against the section of the string ending in a '/'
# 6. mkdir: makes all the parent directories, -p will do so recrusively
# 7. cp: finally copies all files from the /book folder to /book/_build/html
find book/ -exec test -f {} \; -print | grep -v "^book/_.*\|.*\.\(md\|ipynb\)\|thebe_lite" | cut -c 6- | xargs -i sh -c 'echo "book/_build/html/{}" | grep -o "^.*/" | xargs -d "\n" mkdir -p; cp book/"{}" book/_build/html/"{}"'
find book/ -type f | grep -v "^book/_.*\|.*\.\(md\|ipynb\)\|thebe_lite" | cut -c 6- | xargs -i sh -c 'echo "book/_build/html/{}" | grep -o "^.*/" | xargs -d "\n" mkdir -p; cp book/"{}" book/_build/html/"{}"'
# Check whether python has the alias 'python' or 'python3'
if command -v python3 > /dev/null 2>&1
then
python_command="python3"
else
if command -v python > /dev/null 2>&1
then
python_command="python"
fi
fi
# Serves the files on port 8000
$python_command -m http.server 8000 --directory book/_build/html &
if [ "$python_command" = "" ] ; then
echo "Book successfully built. If you want to use interactive elements, start a local HTTP server for the _build/html folder."
exit 0
fi
# Serves the files on port 8000, localhost (127.0.0.1:8000)
START_SERVER=${1:-false}
if [ "$START_SERVER" = true ] ; then
echo "Starting server on port 8000"
$python_command -m http.server 8000 --directory book/_build/html
else
echo "Book successfully built. If you want to use interactive elements, start a server locally using the command: $python_command -m http.server 8000 --directory book/_build/html. Or run this script again: ${0} true"
fi
#!/bin/sh
docker compose down
docker compose up -d --build
docker logs -f mude-book
#!/bin/bash
set -euo pipefail
docker compose -f ./docker/docker-compose.yml down
START_SERVER=${1:-true} docker compose -f ./docker/docker-compose.yml up -d
docker logs -f mude-book
#!/bin/sh
./build-lite.sh
bash
# Docker for book building & deployment
## What is it and why use it?
Docker allows us to run "containers", which are defined on [the Docker website](https://www.docker.com/resources/what-container/) as:
> A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.
Esentially Docker will allow you to build the book across different computers and setups without having to worry about Python packages breaking, or features not being supported on your computer - that is, as long as you get Docker up and running!
## Install and use Docker
You should follow the instructions on the Docker website to install [Docker Desktop](https://docs.docker.com/desktop/) (for Windows the downloaded exe was over 400MB; you will need to log out of windows to complete installation). These instructions were written for Windows users; if you are using a Mac or Linux you can follow the instruction on the [Docker website](https://docs.docker.com/engine/install/); you don't have to install the full Docker desktop, just the engine will do!
- there is no need to create a Docker account
- Windows users should use the Git Bash terminal. If you have never used it: it comes installed with git and provides Unix-like commands. You will need to make it aware of your Anaconda distribution by executing `conda init bash`. Test it by calling a Python package like `jupyter-book`.
- you will also need to install Windows Subsystem for Linux (WSL); it provides a Linux environment on your Wndows machine. Docker Desktop will warn you about WSL not being installed when you start it up for the first time. Try [this](https://learn.microsoft.com/nl-nl/windows/wsl/install-manual#step-4---download-the-linux-kernel-update-package) to install. There is a WSL extension in VS Code but it does not work for Docker.
-
A couple "tricks" are needed to get it running smoothly:
1. Administrative privelages: a) running Docker Desktop in admin mode works, and b) you can also try to add your Windows user to the Docker users group, as described [here](https://stackoverflow.com/questions/61530874/docker-how-do-i-add-myself-to-the-docker-users-group-on-windows). In short, open a windows command prompt as administrator and execute: `net localgroup docker-users "your-user-id" /ADD`. You won't get confirmation, but it seems to work.
2. You might also need to use the first two answers [here](https://stackoverflow.com/questions/43041331/docker-forever-in-docker-is-starting-at-windows-task) (done by Robert). Note that the "Switch to Windows containers..." solution is not needed after you restart a few times (but if the container tab does not load, check it).
3. You may need to use git bash with admin rights, but once set up, your default approach should be OK.
Once Docker is set up correctly, you should see a default page on the Containers tab in Docker Desktop that reads "Your running containers show up here." The book contains some Docker configuration and shell scripts to semi-automate the process of using a container. The main idea is that we define the tools needed to build the book with a Docker _image_ (`docker/Dockerfile`), then we create the Docker _container_ to run the scripts to build to book (`build-book.sh`). When the container runs, `docker/docker-compose.yml` provides the instructions for hosting the book on a local webserver so that you can view the book.
One way to test whether your Docker Desktop installation is working is to run through a couple of the basic tutorials in Docker Desktop. THe first two that are advertised in the software should be sufficient as long as they take you through the process of creating a container and viewing the website in your browser.
### Building the book on your computer (Docker)
This is the custom setup for our MUDE book. It assumes you are a Windows user and are using Git Bash as a terminal, with Docker Desktop installed and working. For Mac or Linux, we assume you installed Docker Desktop and that you are using a terminal and everything works perfectly without problems because...well...it's not Windows.
Initial build times may take a few mintutes, but then the Docker image will be cached and not be rebuilt unless needed. This setup will also produce the `_build` folder on your machine. Trying to build locally after building in docker will cause a rebuild for some reason (maybe incompatible settings?), and visa-versa.
The typical workflow is:
- Open Docker Desktop and make sure the empty container page loads. This means the Docker Engine is ready to build your book.
- Run the `deploy-docker.sh` script. You should see the book build output in the terminal window.
- _Note: after you run_ `deploy-docker.sh` _successfully for the first time, you can start the container directly in Docker Desktop._
- View the book at [http://localhost:8000/](http://localhost:8000/) (_not_ the local build at `./book/_build/html/index.html`---the interactive features won't work!). You can also open this by clicking the link in the Container tab
- once the book is successfully deployed, press Ctrl-C to exit back to terminal so you can run further commands (e.g. `stop-deployment.sh` discussed below). The container will keep running in the background.
- Do whatever book tasks you have: read, edit, use, etc
- If you need to rebuild the book to check your changes, run `deploy-docker.sh` again. You may also need to refresh the page you're viewing, and potentially clear the browser cache to see the changes (contact Robert if there is a problem).
- Once you are finished, stop the container using the `stop-deployment.sh` script or by pushing the stop button in Docker Desktop
- Push any commits to GL
### Docker notes for Windows
Besides the installation tricks mentioned above, which are required to get a container running, there is one other issue we (Max/Robert) had in the setup. The build would proceed as desired from `deploy-docker.sh` until the `CMD` line in the `docker/Dockerfile` was reached, where the container terminal could not find the shell script `build-book.sh`, terminating the process in an error. Docker Desktop would show that the container was exited. All of the commands in `build-book.sh` worked fine when entered in the terminal, which helped (eventually) indicate the problem was with line endings in Windows. This was fixed using `* text=auto eol=lf` in `.gitattributes`.
If applying the `.gitattributes` setup to a git repo that is already initialized, you should run these two commands:
```
git rm --cached -r .
git reset --hard
```
See also the discussion in [MR23](https://gitlab.tudelft.nl/mude/book/-/merge_requests/23).
For future troubleshooting, try the docker commands `docker [network, image, container] ls` to see what is running (choose one of the three options in `[ ]`). You can also delete the image and cnotainer from Docker Desktop. If a line in the `docker/Dockerfile` is causing problems, it can be commented and the container terminal can be used to test things, once the container is activated. If you encounter further issues, contact Max.
......@@ -3,4 +3,5 @@ WORKDIR /book
RUN apt-get update && apt-get -y install git
COPY requirements.txt requirements.txt
RUN pip install -r requirements.txt
CMD ["./docker.sh"]
ENTRYPOINT ["./build-book.sh"]
CMD ["true"] # This gets appended to the ENTRYPOINT, but we can also override it by passing arguments to docker run
services:
web:
container_name: mude-book
build: .
build:
context: ${PWD}
dockerfile: ${PWD}/docker/Dockerfile
ports:
- "8000:8000"
- "8000:8000"
volumes:
- type: bind
source: ${PWD}
target: /book
command: $START_SERVER
stdin_open: true
tty: true
#!/bin/sh
pid=$(ps aux | grep http.server | head -n 1 | awk '{ print $2 }')
kill -9 $pid
#!/bin/sh
docker compose down
#!/bin/bash
docker compose -f ./docker/docker-compose.yml down
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment