Skip to content
Jump to bottom

Initial multi-purpose Grouper image #6

Merged
5 commits merged into from Jan 11, 2018
Merged

Initial multi-purpose Grouper image #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
73c6f21
Lots of clean-up: scim/tomee working; setuid added for Tomcat and Shibd;
Dec 30, 2017
529421d
Merge branch 'multi-purpose-grouper-image' of github.internet2.edu:do…
Dec 30, 2017
175d047
updated readme and other clean-up
Jan 5, 2018
4d364aa
test-compose documentation and other refinements
Jan 5, 2018
d7618eb
Added additional notes
Jan 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .dockerignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
.git/
test-compose/
*.md
manualBuild.sh
manualBuild.sh
LICENSE
66 changes: 40 additions & 26 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,11 @@ RUN yum update -y \
&& yum install -y wget tar unzip dos2unix \
&& yum clean all

ENV ANT_VERSION="1.10.1" \
GROUPER_VERSION=2.3.0 \
JAVA_HOME=/opt/openjdk8 \
TOMCAT_MAJOR=8 \
TOMCAT_VERSION="8.5.23"

RUN java_version=8.0.131; \
zulu_version=8.21.0.1; \
ENV GROUPER_VERSION=2.3.0 \
JAVA_HOME=/opt/java

RUN java_version=8.0.131; \
zulu_version=8.21.0.1; \
echo 'Downloading the OpenJDK Zulu...' \
&& wget -q http://cdn.azul.com/zulu/bin/zulu$zulu_version-jdk$java_version-linux_x64.tar.gz \
&& echo "1931ed3beedee0b16fb7fd37e069b162 zulu$zulu_version-jdk$java_version-linux_x64.tar.gz" | md5sum -c - \
Expand Down Expand Up @@ -40,30 +37,40 @@ RUN echo 'Installing Grouper'; \
cd /opt/grouper/$GROUPER_VERSION/ \
&& $JAVA_HOME/bin/java -cp :grouperInstaller.jar edu.internet2.middleware.grouperInstaller.GrouperInstaller

#The Grouper Installer is corrupting the Messaging Jar files.
RUN cp /opt/grouper/2.3.0/grouper.rabbitMq-2.3.0/lib/* /opt/grouper/2.3.0/grouper.apiBinary-2.3.0/lib/grouper/ \
&& cp /opt/grouper/2.3.0/grouper.activeMq-2.3.0/lib/* /opt/grouper/2.3.0/grouper.apiBinary-2.3.0/lib/grouper/


FROM centos:centos7 as cleanup

COPY --from=installing /opt/grouper/2.3.0/grouper.apiBinary-2.3.0/ /opt/grouper/grouper.apiBinary
COPY --from=installing /opt/grouper/2.3.0/grouper.ui-2.3.0/dist/grouper/ /opt/grouper/grouper.ui/dist/grouper/
COPY --from=installing /opt/grouper/2.3.0/grouper.ws-2.3.0/grouper-ws/build/dist/grouper-ws/ /opt/grouper/grouper.ws/dist/grouper-ws/
COPY --from=installing /opt/grouper/2.3.0/apache-tomcat-8.5.12/ /opt/tomcat/
COPY --from=installing /opt/grouper/2.3.0/apache-tomee-webprofile-7.0.0/ /opt/tomee/
COPY --from=installing /opt/grouper/2.3.0/grouper.clientBinary-2.3.0/ /opt/grouper/grouper.clientBinary/
ENV GROUPER_VERSION=2.3.0 \
TOMCAT_VERSION=8.5.12 \
TOMEE_VERSION=7.0.0

COPY --from=installing /opt/grouper/$GROUPER_VERSION/grouperInstaller.jar /opt/grouper/
COPY --from=installing /opt/grouper/$GROUPER_VERSION/grouper.apiBinary-$GROUPER_VERSION/ /opt/grouper/grouper.apiBinary/
COPY --from=installing /opt/grouper/$GROUPER_VERSION/grouper.ui-$GROUPER_VERSION/dist/grouper/ /opt/grouper/grouper.ui/
COPY --from=installing /opt/grouper/$GROUPER_VERSION/grouper.ws-$GROUPER_VERSION/grouper-ws/build/dist/grouper-ws/ /opt/grouper/grouper.ws/
COPY --from=installing /opt/grouper/$GROUPER_VERSION/grouper.ws-$GROUPER_VERSION/grouper-ws-scim/targetBuiltin/grouper-ws-scim/ /opt/grouper/grouper.scim/
#COPY --from=installing /opt/grouper/$GROUPER_VERSION/grouper.clientBinary-$GROUPER_VERSION/ /opt/grouper/grouper.clientBinary/
COPY --from=installing /opt/grouper/$GROUPER_VERSION/apache-tomcat-$TOMCAT_VERSION/ /opt/tomcat/
COPY --from=installing /opt/grouper/$GROUPER_VERSION/apache-tomee-webprofile-$TOMEE_VERSION/ /opt/tomee/

RUN cd /opt/grouper/grouper.apiBinary/; \
rm -fr ddlScripts/ grouper.lck grouper.log grouper.script grouper.tmp/ gshAddGrouperSystemWsGroup.gsh logs/
rm -fr ddlScripts/ grouper.lck grouper.log grouper.script grouper.tmp/ gshAddGrouperSystemWsGroup.gsh logs/

RUN cd /opt/tomcat/; \
rm -fr webapps/docs/ webapps/examples/ webapps/host-manager/ webapps/manager/ logs/* temp/* work/* \
&& mkdir -p logs/grouperUi logs/grouperWs

RUN cd /opt/tomcat/webapps/; \
rm -fr docs/ examples/ host-manager/ manager/ logs/*
RUN cd /opt/tomee/; \
rm -fr webapps/docs/ webapps/host-manager/ webapps/manager/ logs/* temp/* work/*

RUN cd /opt/tomee/webapps/; \
rm -fr docs/ host-manager/ manager/
RUN sed -i "s/\/opt\/grouper\/$GROUPER_VERSION\/apache-tomcat-$TOMCAT_VERSION/\/opt\/tomcat/g" /opt/grouper/grouper.ui/WEB-INF/classes/log4j.properties \
&& sed -i "s/\/opt\/grouper\/$GROUPER_VERSION\/apache-tomcat-$TOMCAT_VERSION/\/opt\/tomcat/g" /opt/grouper/grouper.ws/WEB-INF/classes/log4j.properties \
&& sed -i 's/${grouper.home}/\/opt\/tomee\//g' /opt/grouper/grouper.scim/WEB-INF/classes/log4j.properties

COPY container_files/tomcat/ /opt/tomcat/
COPY container_files/tomee/ /opt/tomee/

COPY container_files/server.xml /opt/tomcat/conf/


FROM tier/shibboleth_sp
Expand All @@ -74,8 +81,9 @@ LABEL author="tier-packaging@internet2.edu <tier-packaging@internet2.edu>" \
ImageName=$imagename \
ImageOS=centos7

ENV JAVA_HOME=/opt/openjdk8 \
PATH=$PATH:$JAVA_HOME/bin
ENV JAVA_HOME=/opt/java \
PATH=$PATH:$JAVA_HOME/bin \
GROUPER_HOME=/opt/grouper/grouper.apiBinary

RUN ln -sf /usr/share/zoneinfo/UTC /etc/localtime

Expand All @@ -84,12 +92,18 @@ RUN yum update -y \
&& pip install --upgrade pip \
&& pip install supervisor \
&& yum clean -y all

COPY --from=installing $JAVA_HOME $JAVA_HOME
COPY --from=cleanup /opt/tomcat/ /opt/tomcat/
COPY --from=cleanup /opt/tomee/ /opt/tomee/
COPY --from=cleanup /opt/grouper/ /opt/grouper/

RUN groupadd -r tomcat \
&& useradd -r -m -s /sbin/nologin -g tomcat tomcat \
&& mkdir -p /opt/tomcat/logs/ /opt/tomcat/temp/ /opt/tomcat/work/ \
&& chown -R tomcat:tomcat /opt/tomcat/logs/ /opt/tomcat/temp/ /opt/tomcat/work/ \
&& chown -R tomcat:tomcat /opt/tomee/logs/ /opt/tomee/temp/ /opt/tomee/work/

COPY container_files/tier-support/ /opt/tier-support/
COPY container_files/usr-local-bin /usr/local/bin/
COPY container_files/httpd/* /etc/httpd/conf.d/
Expand Down
106 changes: 75 additions & 31 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,11 +26,12 @@ Grouper is an enterprise access management system designed for the highly distri
![logo](https://www.internet2.edu/media/medialibrary/2013/10/15/image_grouper_logowordmark_bw.png)

# How to use this image
This image provides support for each of the Grouper components/roles: Grouper Daemon/Loader, Grouper UI, and Grouper Web Services.

This image provides support for each of the Grouper components/roles: Grouper Daemon/Loader, Grouper UI, Grouper Web Services, and Grouper SCIM Server.

## Starting each role

While TIER recommends/supports using Docker Swarm for orchestrating the Grouper environment, these containers can be run directly. Both examples are shown below. It should be noted that these examples will not run independently, but required additional configuration components to be provided before each container will start as expected.
While TIER recommends/supports using Docker Swarm for orchestrating the Grouper environment, these containers can be run directly (or with other orchestration products). Both examples are shown below. It should be noted that these examples will not run independently, but required additional configuration to be provided before each container will start as expected.

### Daemon/Loader

Expand All @@ -46,6 +47,20 @@ Run the Grouper Daemon/Loader as a standalone container.
$ docker run --detach --name grouper-daemon tier/grouper:latest daemon
```

### SCIM Server

Runs the Grouper SCIM Server as a service.

```console
$ docker service create --detach --publish 9443:443 --name grouper-ws tier/grouper:latest scim
```

Runs the Grouper Web Services in a standalone container.

```console
$ docker run --detach --publish 9443:443 --name grouper-daemon tier/grouper:latest scim
```

### UI

Runs the Grouper UI as a service.
Expand Down Expand Up @@ -76,7 +91,7 @@ $ docker run --detach --publish 8443:443 --name grouper-daemon tier/grouper:late

### UI and Web Services

This good when first starting to work with Grouper, but when scaling Grouper UI or Web Services it is advisable to use the individual roles noted above.
> This method is good when first starting to work with Grouper, but when scaling Grouper UI or Web Services it is advisable to use the individual roles noted above.

Runs the Grouper UI and Web Services as a combined service. (You should really run these as individual roles to take advantage of Docker service replicas.)

Expand All @@ -101,93 +116,122 @@ $ docker run -it --rm tier/grouper:latest bin/gsh <optional GSH args>
# Configuration

## Grouper Configurations

There are several things that are required for this image to successfully start. At a minimum, the `grouper.hibernate.properties` and `subject.properties` (or the old `sources.xml` equivalent) files need to be customized and available to the container at start-up.

Grouper config files maybe placed into `/opt/grouper/conf` and these files will be put into the appropriate location based on the role the container assumes. Docker Secrets starting with the name `grouper_` should take precedence over these files. (See below.)

## Web Apps Configuration
If starting the container to serve the Grouper UI or Grouper Web Services components, a TLS key and cert(s) need to be applied to those containers.

If starting the container to serve the Grouper UI, Grouper Web Services, Grouper SCIM Server components, a TLS key and cert(s) need to be applied to those containers.

The Grouper UI also requires some basic Shibboleth SP configuration. The `/etc/shibboleth/shibboleth2.xml` file should be modified to set:
- an entityId for the SP
- load IdP or federation metadata
- set the SP's encryption keys
- the identity attribute of the subject to be passed to Grouper

If encrpytion keys are defined in the `shibboleth2.xml` file, then the key/cert should be provided as well. The `attribute-map.xml` file has most of the common identity attributes pre-configured, but it (and other Shibbolrth SP files) can be overlaid/replaced as necessary.
If encryption keys are defined in the `shibboleth2.xml` file, then the key/cert files should be provided as well. The `attribute-map.xml` file has most of the common identity attributes pre-configured, but it (and other Shibboleth SP files) can be overlaid/replaced as necessary.

## General Configuration Mechanism
There are three primary ways to provide the Grouper and additional configuration to the container: Docker Config/Secrets, customized images, and bind mounts. Depending upon your needs you may use a combination of two or three of these options.

There are three primary ways to provide Grouper and additional configuration files to the container: Docker Config/Secrets, customized images, and bind mounts. Depending upon your needs you may use a combination of two or three of these options.

### Secrets/Configs

Docker Config and Docker Secrets is Docker's way of providing configurations files to a container. The primary difference between the Config and Secrets functionality is that Secrets is designed to protect resrouces that sensitive files.
Docker Config and Docker Secrets are Docker's way of providing configurations files to a container at runtime. The primary difference between the Config and Secrets functionality is that Secrets is designed to protect resources/files that are sensitive.

This image will make any secrets (containing a period in the secret name) available to the appropriate Grouper component's conf directory (i.e. `<GROUPER_HOME>/conf` or `WEB-INF/classes`). These file will supercede any in the underlying image.
This container will make any secrets with secret names prepended with `grouper_` available to the appropriate Grouper component's conf directory (i.e. `<GROUPER_HOME>/conf` or `WEB-INF/classes`). Any secrets with secret names starting with `shib_` will be available in the Shibboleth SP `/etc/shibboleth/` directory. Any secrets with secret names starting with `httpd_` will be available to `/etc/httpd/conf.d` directory. Finally, if a secret with the name of `host-key.pem` will be mapped to the httpd TLS cert used by Grouper UI, Grouper WS, and Grouper SCIM Server containers. These files will supercede any found in the underlying image.

Secrets can be managed using the `docker secret` command: `docker secret create grouper.hibernate.properties ./grouper.hibernate.properties`. This will securely store the file in the swarm. Secrets can then be assigned to the service `docker service create -d --name daemon --secret grouper.hibernate.properties --secret sources.xml tier/grouper daemon`.
Secrets can be managed using the `docker secret` command: `docker secret create grouper_grouper.hibernate.properties ./grouper.hibernate.properties`. This will securely store the file in the swarm. Secrets can then be assigned to the service `docker service create -d --name daemon --secret grouper_grouper.hibernate.properties --secret grouper_sources.xml tier/grouper daemon`.

> `docker run` does not support secrets; Bind mounts need to be used instead.

### Bind Mounts

Bind mounts can be used to connect files on the Docker host into the container. When not running in swarm mode, the secrets are supported, so we can use a bind mount to provide the container with the configuration files.
Bind mounts can be used to connect files/folders on the Docker host into the container's file system. Unless running in swarm mode, the secrets are not supported, so we can use a bind mount to provide the container with the configuration files.

```
docker run --detach --name daemon \
--mount type=bind,src=$(pwd)/grouper.hibernate.properties,dst=/run/secrets/grouper.hibernate.properties \
--mount type=bind,src=$(pwd)/sources.xml,dst=/run/secrets/sources.xml \
```console
$ docker run --detach --name daemon \
--mount type=bind,src=$(pwd)/grouper.hibernate.properties,dst=/run/secrets/grouper_grouper.hibernate.properties \
--mount type=bind,src=$(pwd)/sources.xml,dst=/run/secrets/grouper_sources.xml \
tier/grouper daemon
```

### Customized Images

Deployers will undoubtedly want to add in their files to the container; things like addtional jar files defining Grouper Hooks, or things like own images, css files, anything. This can be accomplished by building custom images. **Deployers should NOT use this method to store sensitive configuration files.**
Deployers will undoubtedly want to add in their files to the container. Things like additional jar files defining Grouper Hooks, or things like images and css files. This can be accomplished by building custom images. **Deployers should NOT use this method to store sensitive configuration files.**

To add a favicon to the Grouper UI, we use the tier/grouper images as a base and `COPY` our local `favicon.ico` into the image. While we are at it, we define this image as a UI image by specifying the default commnd (i.e `CMD`) of `ui`.

```Dockerfile
FROM tier/grouper:latest

COPY favicon.ico /opt/grouper/grouper.ui/dist/grouper/
COPY favicon.ico /opt/grouper/grouper.ui/

CMD ui
```

To build our image:

```
docker build --tag=org/grouper-ui .
```console
$ docker build --tag=org/grouper-ui .
```

This image can now be used locally or pushed to an organization's Docker repository.


## Memory Limits
(TODO)
## Environment Variables

Deployers can set runtime variables to both the Grouper Shell and Loader/Daemon and to Tomcat/Tomcat EE using environment variables. These can be set using the `docker run` and `docker service creates`'s `--env` paramater.

### Grouper Shell/Loader

The following environment variables are used by the Grouper Shell/Loader:
- MEM_START: corresponds to the java's `-Xms`.
- MEM_MAX: corresponds to java's `-Xmx`.

### Tomcat/TomEE

Amongst others variables defined in the `catalina.sh`, the following variables would like be useful for deployers:
- CATALINA_OPTS: Java runtime options to only be used by Tomcat itself.

# File System Endpoints
Significant directories and files that deployers should be aware of.

- `/grouper/conf/`: a common directory to place non-sensitive config files that will be placed into the appropriate location for each Grouper component at container start-up.
- `/grouper/lib/`: a common directory to place additional jar files that will be placed into the appropriate location for each Grouper component at container start-up.
Here is a list of significant directories and files that deployers should be aware of.

- `/opt/grouper/conf/`: a common directory to place non-sensitive config files that will be placed into the appropriate location for each Grouper component at container start-up.
- `/opt/grouper/lib/`: a common directory to place additional jar files that will be placed into the appropriate location for each Grouper component at container start-up.
- `/opt/grouper/grouper.apiBinary/`: location to overlay Grouper GSH or Daemon/Loader files.
`/opt/grouper/grouper.scim/`: location for overlaying Grouper SCIM Server web application files (expanded `grouper-ws-scim.war`).
- `/opt/grouper/grouper.ui/`: location for overlaying Grouper UI web application files (expanded `grouper.war`).
- `/opt/grouper/grouper.ws/`: location for overlaying Grouper Web Services web application files (expanded `grouper-ws.war`).
- `/etc/httpd/conf.d/ssl-enabled.conf`: Can be overlaid to change the TLS settings when running Grouper UI or Web Servicse.
- `/etc/shibboleth/`: location to overlay the Shibboleth SP configuration files used by the image.
- `/opt/grouper/grouper.apiBinary/`: location to overlay Grouper GSH or Daemon/Loader files.
- `/opt/grouper/grouper.ui/dist/grouper/`: location to overlay Grouper UI web application files.
- `/opt/grouper/grouper.ws/dist/grouper-ws/`: location to overlay Grouper Web Services web application files.
- `/opt/tomcat/`: used to run Grouper UI and Grouper WS
- `/opt/tomee/`: used to run the Grouper SCIM Server.
- `/var/run/secrets`: location where Docker Secrets are mounted into the container. Secrets starting with `grouper_`, `shib_`, and `httpd_` have special meaning. See `Secrets/Configs` above.

To examine baseline image files, one might run `docker run --name=temp -it tier/grouper bash` and browse through these endpoints. While the container is running one may copy files out of the image/container using something like `docker cp temp:/opt/grouper/grouper.api/conf/grouper.properties .`, which will copy the `grouper.properties` to the host's present working directory. These files can then be edited and applied via the mechanisms outlined above.
To examine baseline image files, one might run `docker run --name=temp -it tier/grouper bash` and browse through these file system endpoints. While the container is running one may copy files out of the image/container using something like `docker cp containerId:/opt/grouper/grouper.api/conf/grouper.properties .`, which will copy the `grouper.properties` to the Docker client's present working directory. These files can then be edited and applied via the mechanisms outlined above.

# Provisioning a Grouper Database
(TODO)

```
docker run --detach --rm --name daemon \
--mount type=bind,src=$(pwd)/grouper.hibernate.properties,dst=/run/secrets/grouper.hibernate.properties \
Using standard methods, create a MariaDb Server and an empty Grouper database. Create a database user with privileges to create and populate schema objects. Set the appropriate database connection properties in `grouper.hibernate.properties`. Be sure to the user created with schema manipulation privileges as the db user.

Next populate the database by using the following command.

```console
$ docker run -it --rm \
--mount type=bind,src=$(pwd)/grouper.hibernate.properties,dst=/run/secrets/grouper_grouper.hibernate.properties \
tier/grouper gsh -registry -check -runscript -noprompt
```

Note: a less privileged database user maybe used when running the typical Grouper roles. This user need SELECT, INSERT, UPDATE, and DELETE privileges on the schema objects.

# Misc Notes

- [HTTP Strict Transport Security (HSTS)](https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security) is enabled on the Apache HTTP Server.
- morphStrings functionality in Grouper is supported. It is recommended that the various morphString files be associated with the containers as Docker Secrets. Set the configuration file properties to use `/var/run/secrets/secretname`.

# License

Expand Down
Loading