Merge branch 'master' of https://github.internet2.edu/internet2/InCom…

…monTAP-Examples

Components/Grouper/README.md

@@ -0,0 +1,64 @@
+The `test-compose` directory contains an example Grouper environment that starts up the various Grouper components. This example demonstrates how one might go about customizing and deploying their Grouper containers, using the TIER Grouper image as a base image. 
+
+In this example, the following cases are covered by this example:
+
+- A demo directory and SIS database are included, populated with approximately 1,000 test subjects.
+- Grouper is configured to use this directory as the subject source.
+- Grouper Loader creates groups based on the data in the SIS table.
+- Grouper UI is protected by a Shibboleth IdP (included) that connects to this directory server.
+- Grouper WS is protected by http basic auth that authenticates against the directory server.
+- Grouper publishes event data to a RabbitMQ instance (included). 
+
+It should be noted that while this example uses Docker Compose as a build and deployment vehicle, ideally one should use a CI server to build and publish institution specific images to an image repository as changes to the institution's customizations are committed to the source repository. These images would then be deployed to Docker Swarm, assuming that the appropriate Docker Secrets and Configs have been published to the swarm.
+
+# Getting Started
+
+From `test-compose` directory, run:
+
+```console
+$ docker-compose up -d
+```
+
+This will build each of our customized images after downloading the TIER Grouper image. It will create containers for each of our components using the configuation specified in the `docker-compose.yml` file.
+
+To stop the Grouper environment, run:
+
+```console
+$ docker-compose down
+```
+
+When doing iterative work, such as testing UI changes or configuration changes, I find if handy to use the following command:
+
+```console
+$ docker-compose kill; docker-compose rm -f; docker-compose build && docker-compose up
+```
+
+This command will clear out any remaining containers, as defined by the `docker-compose.yml` file, from the Docker host, rebuild our custom images, and start new instances of them. Because we do not specify the `-d` on the `up` command, the containers will not be forked causing the container logs to be displayed to the console, and the command prompt will not return until hitting `Ctrl+C`, which will kill the running containers.
+
+# Testing Endpoints
+
+The components can be accessed at the following urls, with
+
+Grouper UI: https://localhost/grouper (username: banderson, password: password (from ldap) or password1 (from tomcat-users.xml))
+Grouper WS: https://localhost:8443/grouper-ws/status?diagnosticType=all
+Grouper SCIM: https://localhost:9443/grouper-ws-scim/ (username: banderson, password: password (from tomcat-users.xml))
+RabbmitMQ: http://localhost:15672/ (username: guest, password: guest) 
+MariaDB: Port 3306 (username: root, password: (no password) )
+389-ds Directory: Port 389 (username: cn=Directory Manager, password: password)
+
+Note that when accessing the Grouper UI, Grouper WS, or Shibboleth IdP, your browser will prompt you about an untrusted certificate. It is OK to ignore the warning while working with this example configuration.
+
+# Additional Notes
+
+- In this example, we use a variety of ways to pass in passwords (Grouper database, LDAP, Grouper Client, and RabbitMQ). The point is to demonstrate possibilities and not demonstrating what is required. (See the image readme for more details.)
+- Docker `configs` are not supported by Docker Compose (when run in a non-Swarm mode), so those are represented in the `docker-compose.yml` file as bind mount volumes.
+- The Grouper config files in the `data` image's `conf` directory are used to build the sample grouper database and ldap store. They are not used when the container is instantiated as there is no Grouper runtime in this container.
+- The containers will use Docker Secrets and bind mounts for non-sensitive files that are read from the `configs-and-secrets` directory in the `test-compose` directory.
+- With regard to RabbitMQ, the deployer must manually add a queue named `sampleQueue` to see Grouper messages in RabbitMQ. Messages will be dropped by RabbitMQ (and the Grouper Deamon will log errors) until this occurs.
+- In this example, we don't care about the IdP secrets. They are baked into the overlay instead of using Docker Secrets. (This is not best practice for an IdP configuration, but that isn't the focus of this example.)
+
+# Future TODOs
+
+- Add a Docker Stack example
+
+> This docker-stack.yml file uses the `configs` syntax which is part of the Compose file format v3.3 and requires Docker Engine version 17.06.0+ (released on 2017-06-28). Users of older engine versions will need convert `config` references to use bind mounts. After this change, everything else should work as expected.

Components/Grouper/compose.sh

@@ -0,0 +1,2 @@
+#!/bin/sh
+docker-compose up --build -d

Components/Grouper/configs-and-secrets/grouper/grouper-loader.properties

@@ -0,0 +1,63 @@
+#################################
+## LDAP connections
+#################################
+# specify the ldap connection with user, pass, url
+# the string after "ldap." is the ID of the connection, and it should not have
+# spaces or other special chars in it.  In this case is it "personLdap"
+
+#note the URL should start with ldap: or ldaps: if it is SSL.  
+#It should contain the server and port (optional if not default), and baseDn,
+#e.g. ldaps://ldapserver.school.edu:636/dc=school,dc=edu
+ldap.demo.url = ldap://data:389/dc=internet2,dc=edu
+
+#optional, if authenticated
+ldap.demo.user = cn=admin,dc=internet2,dc=edu
+
+#optional, if authenticated note the password can be stored encrypted in an external file
+ldap.demo.pass.elConfig = ${java.lang.System.getenv().get('SUBJECT_SOURCE_LDAP_PASSWORD_FILE') != null ? org.apache.commons.io.FileUtils.readFileToString(java.lang.System.getenv().get('SUBJECT_SOURCE_LDAP_PASSWORD_FILE'), "utf-8") : java.lang.System.getenv().get('SUBJECT_SOURCE_LDAP_PASSWORD')}
+#ldap.demo.pass = password
+
+#optional, if you are using tls, set this to true.  Generally you will not be using an SSL URL to use TLS...
+ldap.demo.tls = false
+
+#optional, if using sasl
+#ldap.personLdap.saslAuthorizationId =
+#ldap.personLdap.saslRealm =
+
+#optional (note, time limit is for search operations, timeout is for connection timeouts),
+#most of these default to vt-ldap defaults.  times are in millis
+#validateOnCheckout defaults to true if all other validate methods are false
+#ldap.personLdap.batchSize =
+#ldap.personLdap.countLimit =
+#ldap.personLdap.timeLimit =
+#ldap.personLdap.timeout =
+#ldap.personLdap.minPoolSize =
+#ldap.personLdap.maxPoolSize =
+#ldap.personLdap.validateOnCheckIn =
+#ldap.personLdap.validateOnCheckOut =
+#ldap.personLdap.validatePeriodically =
+#ldap.personLdap.validateTimerPeriod =
+#ldap.personLdap.pruneTimerPeriod =
+#if connections expire after a certain amount of time, this is it, in millis, defaults to 300000 (5 minutes)
+#ldap.personLdap.expirationTime =
+
+#make the paths fully qualified and not relative to the loader group.
+loader.ldap.requireTopStemAsStemFromConfigGroup=false
+
+#####################################
+## Messaging integration with change log
+#####################################
+changeLog.consumer.rabbitMqMessagingSample.quartzCron = 0 * * * * ?                                                          
+
+# note, change "messagingSample" in key to be the name of the consumer.  e.g. changeLog.consumer.someNameAnyName.class
+changeLog.consumer.rabbitMqMessagingSample.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbConsumer
+
+changeLog.consumer.rabbitMqMessagingSample.publisher.class = edu.internet2.middleware.grouper.changeLog.esb.consumer.EsbMessagingPublisher
+changeLog.consumer.rabbitMqMessagingSample.publisher.messagingSystemName = rabbitmq
+# note, routingKey property is valid only for rabbitmq. For other messaging systems, it is ignored.
+changeLog.consumer.rabbitMqMessagingSample.publisher.routingKey = 
+## queue or topic
+changeLog.consumer.rabbitMqMessagingSample.publisher.messageQueueType = queue
+changeLog.consumer.rabbitMqMessagingSample.publisher.queueOrTopicName = sampleQueue
+## this is optional if not using "id" for subjectId, need to be a subject attribute in the sources.xml
+#changeLog.consumer.rabbitMqMessagingSample.publisher.addSubjectAttributes = email

Components/Grouper/configs-and-secrets/grouper/grouper.client.properties

@@ -0,0 +1,112 @@
+#
+# Copyright 2014 Internet2
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+# Grouper client configuration
+# $Id: grouper.client.example.properties,v 1.24 2009-12-30 04:23:02 mchyzer Exp $
+#
+
+# The grouper client uses Grouper Configuration Overlays (documented on wiki)
+# By default the configuration is read from grouper.client.base.properties
+# (which should not be edited), and the grouper.client.properties overlays
+# the base settings.  See the grouper.client.base.properties for the possible
+# settings that can be applied to the grouper.client.properties
+
+########################################
+## LDAP connection settings
+########################################
+
+# url of directory, including the base DN (distinguished name)
+# e.g. ldap://server.school.edu/dc=school,dc=edu
+# e.g. ldaps://server.school.edu/dc=school,dc=edu
+grouperClient.ldap.url =
+
+# kerberos principal used to connect to ldap
+grouperClient.ldap.login =
+
+# password for shared secret authentication to ldap
+# or you can put a filename with an encrypted password
+grouperClient.ldap.password =
+
+########################################
+## Web service Connection settings
+########################################
+
+# url of web service, should include everything up to the first resource to access
+# e.g. http://groups.school.edu:8090/grouper-ws/servicesRest
+# e.g. https://groups.school.edu/grouper-ws/servicesRest
+grouperClient.webService.url = https://ws/grouper-ws/servicesRest
+
+# kerberos principal used to connect to web service
+grouperClient.webService.login = banderson
+
+# password for shared secret authentication to web service
+# or you can put a filename with an encrypted password
+grouperClient.webService.password.elConfig = ${java.lang.System.getenv().get('GROUPER_CLIENT_WEBSERVICE_PASSWORD_FILE') != null ? org.apache.commons.io.FileUtils.readFileToString(java.lang.System.getenv().get('GROUPER_CLIENT_WEBSERVICE_PASSWORD_FILE'), "utf-8") : java.lang.System.getenv().get('GROUPER_CLIENT_WEBSERVICE_PASSWORD') }
+
+
+################################
+## Grouper Messaging System
+################################
+
+# name of messaging system which is the default
+grouper.messaging.default.name.of.messaging.system = rabbitmq
+
+# name of a messaging system.  note, "grouperBuiltinMessaging" can be arbitrary
+# grouper.messaging.system.grouperBuiltinMessaging.name = grouperBuiltinMessaging
+
+# class that implements edu.internet2.middleware.grouperClient.messaging.GrouperMessagingSystem
+# grouper.messaging.system.grouperBuiltinMessaging.class = edu.internet2.middleware.grouper.messaging.GrouperBuiltinMessagingSystem
+
+# name of a messaging system.  note, "grouperBuiltinMessaging" can be arbitrary
+grouper.messaging.system.rabbitmqSystem.name = rabbitmqSystem
+
+# class that implements edu.internet2.middleware.grouperClient.messaging.GrouperMessagingSystem
+grouper.messaging.system.rabbitmqSystem.class = edu.internet2.middleware.grouperMessagingRabbitmq.GrouperMessagingRabbitmqSystem
+
+# host address of rabbitmq queue
+grouper.messaging.system.rabbitmqSystem.host = rabbitmq
+
+# virtual host of rabbitmq queue
+grouper.messaging.system.rabbitmqSystem.virtualhost =
+
+# port of rabbitmq queue
+grouper.messaging.system.rabbitmqSystem.port =
+
+grouper.messaging.system.rabbitmqSystem.defaultPageSize = 10
+
+grouper.messaging.system.rabbitmqSystem.maxPageSize = 50
+
+
+# name of a messaging system, required
+grouper.messaging.system.rabbitmq.name = rabbitmq
+
+# default system settings to this messaging system, note, there is only one level of inheritance
+grouper.messaging.system.rabbitmq.defaultSystemName = rabbitmqSystem
+
+grouper.messaging.system.rabbitmq.user = guest
+
+#pass
+grouper.messaging.system.rabbitmq.password.elConfig = ${java.lang.System.getenv().get('RABBITMQ_PASSWORD_FILE') != null ? org.apache.commons.io.FileUtils.readFileToString(java.lang.System.getenv().get('RABBITMQ_PASSWORD_FILE'), "utf-8") : java.lang.System.getenv().get('RABBITMQ_PASSWORD') }
+# set the following three properties if you want to use TLS connection to rabbitmq. All three need to be populated.
+# TLS Version
+#grouper.messaging.system.rabbitmqSystem.tlsVersion = TLSv1.1
+
+# path to trust store file
+#grouper.messaging.system.rabbitmqSystem.pathToTrustStore =
+
+# trust passphrase
+#grouper.messaging.system.rabbitmqSystem.trustPassphrase =

Components/Grouper/configs-and-secrets/grouper/grouper.hibernate.properties

@@ -0,0 +1,29 @@
+#
+# Grouper Hibernate Configuration
+# $Id: grouper.hibernate.example.properties,v 1.9 2009-08-11 20:18:09 mchyzer Exp $
+#
+
+# The grouper hibernate config uses Grouper Configuration Overlays (documented on wiki)
+# By default the configuration is read from grouper.hibernate.base.properties
+# (which should not be edited), and the grouper.hibernate.properties overlays
+# the base settings.  See the grouper.hibernate.base.properties for the possible
+# settings that can be applied to the grouper.hibernate.properties
+
+########################################
+## DB settings
+########################################
+
+# e.g. mysql:           jdbc:mysql://localhost:3306/grouper
+# e.g. p6spy (log sql): [use the URL that your DB requires]
+# e.g. oracle:          jdbc:oracle:thin:@server.school.edu:1521:sid
+# e.g. hsqldb (a):      jdbc:hsqldb:dist/run/grouper;create=true
+# e.g. hsqldb (b):      jdbc:hsqldb:hsql://localhost:9001/grouper
+# e.g. postgres:        jdbc:postgresql://localhost:5432/database
+# e.g. mssql:           jdbc:sqlserver://localhost:3280;databaseName=grouper
+hibernate.connection.url = jdbc:mysql://data:3306/grouper?CharSet=utf8&useUnicode=true&characterEncoding=utf8
+
+hibernate.connection.username         = root
+# If you are using an empty password, depending upon your version of
+# Java and Ant you may need to specify a password of "".
+# Note: you can keep passwords external and encrypted: https://bugs.internet2.edu/jira/browse/GRP-122
+hibernate.connection.password.elConfig = ${java.lang.System.getenv().get('GROUPER_DATABASE_PASSWORD_FILE') != null ? org.apache.commons.io.FileUtils.readFileToString(java.lang.System.getenv().get('GROUPER_DATABASE_PASSWORD_FILE'), "utf-8") : java.lang.System.getenv().get('GROUPER_DATABASE_PASSWORD') }

Components/Grouper/configs-and-secrets/grouper/grouper.properties

@@ -0,0 +1,25 @@
+#
+# Grouper Configuration
+# $Id: grouper.example.properties,v 1.48 2009-12-16 06:02:30 mchyzer Exp $
+#
+
+# Grouper uses Grouper Configuration Overlays (documented on wiki)
+# By default the configuration is read from grouper.base.properties
+# (which should not be edited), and the grouper.properties overlays
+# the base settings.  See the grouper.base.properties for the possible
+# settings that can be applied to the grouper.properties
+
+#if groups like the wheel group should be auto-created for convenience (note: check config needs to be on)
+configuration.autocreate.system.groups = true
+
+# A wheel group allows you to enable non-GrouperSystem subjects to act
+# like a root user when interacting with the registry.
+groups.wheel.use                      = true
+
+# Set to the name of the group you want to treat as the wheel group.
+# The members of this group will be treated as root-like users.
+groups.wheel.group                    = etc:sysadmingroup
+
+# Used to allow Include Exclude groups
+grouperIncludeExclude.use = true
+grouperIncludeExclude.requireGroups.use = true

Components/Grouper/configs-and-secrets/grouper/rabbitmq_password.txt

@@ -0,0 +1 @@
+guest