Dynamic Depot Assignment

Introduction

With the standard multi-depot support in opsi, the clients are permanently assigned to the respective depots. This can be extended by a mechanism with which a client can recognize from which depot it can best obtain its software. The specific implementation of a sensible algorithm for this depends on the respective environment and is therefore configurable. Assignment according to IP addresses is the simplest and most suitable solution in many areas. In other network topologies, this is not sufficient.

To offer the option, that the client can detect the suitable depot according to the current network conditions, it must be ensured, that the alternative depots are synchronized, which means they offer the same software packages. In practice the depots will not be synchronized at all times. So the list depots offered to a client is limited to those depots, which are synchronized with the master depot of the client. The master depot of a client is the depot, to which the client is assigned to. The master depot thus determines which software and which version can be installed on the client.

Our concept for this is as follows:

The opsi config server provides a client script,which is transferred to the client and interpreted there if necessary. This script determines which of the available depots is used. For this client script is defined: the interface with which the script gets the list of available servers and the current client configuration (IP address, netmask, gateway) and to return the result of the selection procedure. Furthermore there are interfaces for logging and information about the ongoing process.

The specific implementation of this script can easily be adapted to the requirements of the particular opsi environment.

Resulting from this concept, the sequence of a client connection then looks as follows:

  1. The client reports to the opsi config server via the web service.

  2. The opsi config server sends the client the list of software packages to be installed.

  3. The opsi config server transmits to the client the script for detecting the best depot and the list of available depots.

  4. The client executes the script and uses it to determine the best depot.

  5. The client connects to the selected depot to get the required software packages.

  6. The installation status is reported back to the opsi config server.

Configuration

To activate the dynamic depot selection, the host parameter clientconfig.depot.dynamic must be set to true. This can be client-specific, depot-specific or global.

In addition, the clientconfig.depot.selection_mode parameter can be used to select which algorithm is to be used to determine the best depot.

The following selection options are available:

latency

The depot with the lowest network latency (ping) is used.

master_and_latency

Like latency, but only depots that are assigned to the master depot of the client via masterDepotId are considered.

network_address

The depot whose configured network address (networkAddress) matches the current IP address of the client is used.

random

The depot is selected at random.

Editing depot properties

The properties of a depot can be changed in the opsi Management Interface as well as on the command line.

Depot properties in opsi configed
Figure 1. Depot properties in 'opsi-configed'

On the command line the depot properties can be shown with the method host_getObjects. Here e.g. for the depot 'dep1.uib.local'.

opsi-admin -d method host_getObjects [] '{"id":"dep1.uib.local"}'

This example results in the following output:

[
          {
          "masterDepotId" : "masterdepot.uib.local",
          "ident" : "dep1.uib.local",
          "networkAddress" : "192.168.101.0/255.255.255.0",
          "description" : "Depot 1 Master Depot",
          "inventoryNumber" : "",
          "ipAddress" : "192.168.105.1",
          "repositoryRemoteUrl" : "webdavs://dep1.uib.local:4447/repository",
          "depotLocalUrl" : "file:///var/lib/opsi/depot",
          "isMasterDepot" : true,
          "notes" : "",
          "hardwareAddress" : "52:54:00:37:c6:8b",
          "maxBandwidth" : 0,
          "repositoryLocalUrl" : "file:///var/lib/opsi/repository",
          "opsiHostKey" : "6a13da751fe76b9298f4ede127280809",
          "type" : "OpsiDepotserver",
          "id" : "dep1.uib.local",
          "depotWebdavUrl" : "webdavs://dep1.uib.local:4447/depot",
          "depotRemoteUrl" : "smb://dep1/opsi_depot"
          }
]

To edit the depot properties from the command line, the output is written to a file:

opsi-admin -d method host_getObjects [] '{"id":"dep1.uib.local"}' > /tmp/depot_config.json

The resulting file (/tmp/depot_config.json) can now be edited and written back with the following command:

opsi-admin -d method host_createObjects < /tmp/depot_config.json

The depot properties, which are relevant in the context of dynamic depot allocation are:

  • isMasterDepot
    Must be true for assigning a client to this depot. If false is entered here, no clients can be assigned, but the depot is still used in the dynamic depot assignment.

  • networkAddress
    Network addresses for which this depot is responsible. The network address can be specified in two notations:

    • network/netmask, example: 192.168.101.0/255.255.255.0

    • network/maskbits, example: 192.168.101.0/24

Whether the networkAddress is actually evaluated to determine the depot depends on the algorithm in the selection script.

Synchronization of depots

To keep the depots in sync, opsi provides several tools:

  • opsi-package-manager

  • opsi-package-updater

When installing an opsi package, the opsi-package-manager can be instructed to install the package not only on the current server but on all known depots by using the parameter -d ALL. Example:

opsi-package-manager -i opsi-template_1.0-20.opsi -d ALL

By using the parameter -D, opsi-package-manager can be instructed to list the differences between depots. A list of depots must be specified with the -d option or all known depots must be selected with -d ALL. Example:

opsi-package-manager -D -d ALL

opsi-package-manager is also the tool used for a 'push' synchronization. On the other hand, the tool opsi-package-updater is intended to synchronize depots using the 'pull' method.

opsi-package-updater can run on the depots as a cronjob. This enables easy automation. Please refer to the chapter opsi-package-updater for further information on the configuration.

If a package is installed on a opsi-Server with opsi-package-manager -i (without -d), it does not end up in the repository directory. In order for it to be copied there, you can either specify the name of the depot during installation with -d or explicitly instruct the upload to the repository directory with opsi-package-manager -u <name of the package>.

Please also note the description of the two tools in the corresponding chapters of the opsi manual.

Processing

If the dynamic depot assignment is activated for a client via the host parameter 'clientconfig.depot.dynamic', the client retrieves the script via the web service and executes it.

The script that the client uses to select the depot is stored on the server as the file:
/etc/opsi/backendManager/extend.d/70_dynamic_depot.conf

The following parameters are transferred to the 'selectDepot' function defined in this script:

  • clientConfig
    Information of the current client configuration (hash).
    The clientConfig hash keys are currently:

    • clientId: opsi host ID of the client (FQDN)

    • ipAddress: IP address of the network interface used to connect to the config server

    • netmask: network mask of the network interface

    • defaultGateway: default gateway

  • masterDepot
    Information about the master depot ('opsi-Depotserver'-object). The master depot is the depot to which the client is assigned in the management interface. The attributes of the transferred 'opsi-Depotserver'-object correspond to the attributes as given by host_getObjects (see Editing depot properties).

  • alternativeDepots
    Information about the alternative depots (list of 'opsi-Depotserver'-objects). The list of alternative depots is determined from the depots which are synchronised to the master depot, in regard to the software packages currently required.

Based on this information, the algorithm can now select a depot from the list. The 'opsi-Depotserver'-object of the depot to be used must be returned by the function. If the algorithm does not find a suitable depot from the list of alternative depots or if this is empty, the master depot should be returned.

Selection script template

Three functions for selecting a depot are already implemented in the template script.
The function depotSelectionAlgorithmByNetworkAddress checks the network addresses of the depots and selects the depot where the current IP address is in the network of the depot.
The function depotSelectionAlgorithmByLatency sends ICMP echo request packets (ping) to depots and selects the depot with the lowest latency.
The function depotSelectionAlgorithmByMasterDepotAndLatency is intended for environments with several master depots, which can have slave depots assigned. From the set of master depots of the client and the associated slave depots, the depot that has the lowest latency is selected.
The function getDepotSelectionAlgorithmByNetworkAddressBestMatch works similarly to depotSelectionAlgorithmByNetworkAddress, but prefers the best matching (smallest) Network. The function getDepotSelectionAlgorithmByRandom chooses one of the available depots at random. This can be used for balencing load, but it is important to keep all package versions on the depot at the same level. The function getDepotSelectionAlgorithm is called by the client and returns the algorithm to be used for the selection of the depot. Without changes to the template script, the function depotSelectionAlgorithmByNetworkAddress will be returned.

After changing the selected Algorithm (by commenting in/out in the getDepotSelectionAlgorithm function) the opsiconfd has to be restarted for the changes to take effect.

# -*- coding: utf-8 -*-

global showDepotInfoFunction
showDepotInfoFunction = \
'''
	def showDepotInfo():
		logger.info("Choosing depot from list of depots:")
		logger.info("   Master depot: %s", masterDepot)
		for alternativeDepot in alternativeDepots:
			logger.info("   Alternative depot: %s", alternativeDepot)
'''

global getDepotWithLowestLatencyFunction
getDepotWithLowestLatencyFunction = \
'''
	def getDepotWithLowestLatency(latency):
		"""
		Given a dict with depot as key and latency as value it will \
return the depot with the lowest latency.

		Will return None if no depot can be determined.
		"""
		selectedDepot = None
		if latency:
			minValue = 1000
			for (depot, value) in latency.items():
				if value < minValue:
					minValue = value
					selectedDepot = depot
			logger.notice("Depot with lowest latency: %s (%0.3f ms)", selectedDepot, minValue*1000)

		return selectedDepot
'''

global getLatencyInformationFunction
getLatencyInformationFunction = \
'''
	def getLatencyInformation(depots):
		"""
		Pings the given depots and returns the latency information in \
a dict with depot as key and the latency as value.

		Depots that can't be reached in time will not be included.
		"""
		from OPSI.Util.Ping import ping
		from urllib.parse import urlparse

		latency = {}
		for depot in depots:
			if not depot.repositoryRemoteUrl:
				logger.info("Skipping {depot} because repositoryRemoteUrl is missing.", depot)
				continue

			try:
				host = urlparse(depot.repositoryRemoteUrl).hostname
				# To increase the timeout (in seconds) for the ping you
				# can implement it in the following way:
				#  depotLatency = ping(host, timeout=5)
				depotLatency = ping(host)

				if depotLatency is None:
					logger.info("Ping to depot %s timed out.", depot)
				else:
					logger.info("Latency of depot %s: %0.3f ms", depot, depotLatency * 1000)
					latency[depot] = depotLatency
			except Exception as e:
				logger.warning(e)

		return latency
'''


def getDepotSelectionAlgorithmByMasterDepotAndLatency(self):
	return '''\
def selectDepot(clientConfig, masterDepot, alternativeDepots=[]):
	{getLatencyInformationFunction}
	{getDepotWithLowestLatencyFunction}
	{showDepotInfoFunction}

	showDepotInfo()

	if alternativeDepots:
		from collections import defaultdict

		# Mapping of depots to its master.
		# key: Master depot
		# value: All slave depots + master
		depotsByMaster = defaultdict(list)

		allDepots = [masterDepot] + alternativeDepots

		for depot in allDepots:
			if depot.masterDepotId:
				depotsByMaster[depot.masterDepotId].append(depot)
			else:
				depotsByMaster[depot.id].append(depot)

		depotsWithLatency = getLatencyInformation(depotsByMaster[masterDepot.id])
		depotWithLowestLatency = getDepotWithLowestLatency(depotsWithLatency)

		if not depotWithLowestLatency:
			logger.info('No depot with lowest latency. Falling back to master depot.')
			depotWithLowestLatency = masterDepot

		return depotWithLowestLatency

	return masterDepot
'''.format(
	showDepotInfoFunction=showDepotInfoFunction,
	getLatencyInformationFunction=getLatencyInformationFunction,
	getDepotWithLowestLatencyFunction=getDepotWithLowestLatencyFunction
)

def getDepotSelectionAlgorithmByLatency(self):
	return '''\
def selectDepot(clientConfig, masterDepot, alternativeDepots=[]):
	{getLatencyInformationFunction}
	{getDepotWithLowestLatencyFunction}
	{showDepotInfoFunction}

	showDepotInfo()

	selectedDepot = masterDepot
	if alternativeDepots:
		depotsWithLatency = getLatencyInformation([masterDepot] + alternativeDepots)
		selectedDepot = getDepotWithLowestLatency(depotsWithLatency)

		if not selectedDepot:
			logger.info('No depot with lowest latency. Falling back to master depot.')
			selectedDepot = masterDepot

	return selectedDepot
'''.format(
	showDepotInfoFunction=showDepotInfoFunction,
	getLatencyInformationFunction=getLatencyInformationFunction,
	getDepotWithLowestLatencyFunction=getDepotWithLowestLatencyFunction
)

def getDepotSelectionAlgorithmByRandom(self):
	return '''\
def selectDepot(clientConfig, masterDepot, alternativeDepots=[]):
	{showDepotInfoFunction}

	showDepotInfo()

	import random

	allDepots = [masterDepot]
	allDepots.extend(alternativeDepots)
	return random.choice(allDepots)
'''.format(
	showDepotInfoFunction=showDepotInfoFunction
)

def getDepotSelectionAlgorithmByNetworkAddress(self):
	return '''\
def selectDepot(clientConfig, masterDepot, alternativeDepots=[]):
	{showDepotInfoFunction}

	showDepotInfo()

	selectedDepot = masterDepot
	if alternativeDepots:
		from OPSI.Util import ipAddressInNetwork

		depots = [masterDepot]
		depots.extend(alternativeDepots)
		for depot in depots:
			if not depot.networkAddress:
				logger.warning("Network address of depot '%s' not known", depot)
				continue

			if ipAddressInNetwork(clientConfig['ipAddress'], depot.networkAddress):
				logger.notice("Choosing depot with networkAddress %s for ip %s", depot.networkAddress, clientConfig['ipAddress'])
				selectedDepot = depot
				break
			else:
				logger.info("IP %s does not match networkAddress %s of depot %s", clientConfig['ipAddress'], depot.networkAddress, depot)

	return selectedDepot
'''.format(
	showDepotInfoFunction=showDepotInfoFunction,
)


def getDepotSelectionAlgorithmByNetworkAddressBestMatch(self):
	return '''\
def selectDepot(clientConfig, masterDepot, alternativeDepots=[]):
	{showDepotInfoFunction}

	showDepotInfo()
	logger.debug("Alternative Depots are: %s", alternativeDepots)
	selectedDepot = masterDepot
	if alternativeDepots:
		from OPSI.Util import ipAddressInNetwork
		import ipaddress

		depots = [masterDepot]
		depots.extend(alternativeDepots)
		logger.debug("All considered Depots are: %s",depots)
		sorted_depots = sorted(depots, key=lambda depot: ipaddress.ip_network(depot.networkAddress), reverse=True)
		logger.debug("Sorted depots: %s", sorted_depots)
		for depot in sorted_depots:
			logger.debug("Considering Depot %s with NetworkAddress %s", depot, depot.networkAddress)
			if not depot.networkAddress:
				logger.warning("Network address of depot '%s' not known", depot)
				continue

			if ipAddressInNetwork(clientConfig['ipAddress'], depot.networkAddress):
				logger.notice("Choosing depot with networkAddress %s for ip %s", depot.networkAddress, clientConfig['ipAddress'])
				selectedDepot = depot
				break
			else:
				logger.info("IP %s does not match networkAddress %s of depot %s", clientConfig['ipAddress'], depot.networkAddress, depot)

	return selectedDepot
'''.format(
	showDepotInfoFunction=showDepotInfoFunction,
)

def getDepotSelectionAlgorithm(self):
	""" Returns the selected depot selection algorythm.	"""
	# return self.getDepotSelectionAlgorithmByMasterDepotAndLatency()
	# return self.getDepotSelectionAlgorithmByLatency()
	return self.getDepotSelectionAlgorithmByNetworkAddress()
	# return self.getDepotSelectionAlgorithmByNetworkAddressBestMatch
	# return self.getDepotSelectionAlgorithmByRandom()

Logging

If the dynamic depot assignment is activated, the corresponding entries from the depot selection can be found in opsiclientd.log. Here is the shortened log of an example session. In this example the server bonifax.uib.local is the config server and master depot for the client pctrydetlef.uib.local. The master server bonifax.uib.local has the network address 192.168.1.0/255.255.255.0. Available as an alternative depot is stb-40-srv-001.uib.local with the network address 192.168.2.0/255.255.255.0. The client pctry4detlef.uib.local has the IP address 192.168.2.109, and is therefore in the network of the alternative depot.

(...)
[6] [Dec 02 18:25:27] [ opsiclientd                   ] Connection established to: 192.168.1.14   (HTTP.pyo|421)
[5] [Dec 02 18:25:28] [ event processing gui_startup  ]    [ 1] product opsi-client-agent:   setup   (EventProcessing.pyo|446)
[5] [Dec 02 18:25:28] [ event processing gui_startup  ] Start processing action requests   (EventProcessing.pyo|453)
[5] [Dec 02 18:25:28] [ event processing gui_startup  ] Selecting depot for products [u'opsi-client-agent']   (Config.pyo|314)
[5] [Dec 02 18:25:28] [ event processing gui_startup  ] Selecting depot for products [u'opsi-client-agent']   (__init__.pyo|36)
(...)
[6] [Dec 02 18:25:28] [ event processing gui_startup  ] Dynamic depot selection enabled   (__init__.pyo|78)
(...)
[6] [Dec 02 18:25:28] [ event processing gui_startup  ] Master depot for products [u'opsi-client-agent'] is bonifax.uib.local   (__init__.pyo|106)
[6] [Dec 02 18:25:28] [ event processing gui_startup  ] Got alternative depots for products: [u'opsi-client-agent']   (__init__.pyo|110)
[6] [Dec 02 18:25:28] [ event processing gui_startup  ] 1. alternative depot is stb-40-srv-001.uib.local   (__init__.pyo|112)
(...)
[6] [Dec 02 18:25:28] [ event processing gui_startup  ] Verifying modules file signature   (__init__.pyo|129)
[5] [Dec 02 18:25:28] [ event processing gui_startup  ] Modules file signature verified (customer: uib GmbH)   (__init__.pyo|143)
(...)
[6] [Dec 02 18:25:28] [ event processing gui_startup  ] Choosing depot from list of depots:   (<string>|4)
[6] [Dec 02 18:25:28] [ event processing gui_startup  ]    Master depot: <OpsiConfigserver id 'bonifax.uib.local'>   (<string>|5)
[6] [Dec 02 18:25:28] [ event processing gui_startup  ]    Alternative depot: <OpsiDepotserver id 'stb-40-srv-001.uib.local'>   (<string>|7)
[5] [Dec 02 18:25:28] [ event processing gui_startup  ] Choosing depot with networkAddress 192.168.2.0/255.255.255.0 for ip 192.168.2.109   (<string>|40)
[5] [Dec 02 18:25:28] [ event processing gui_startup  ] Selected depot is: <OpsiDepotserver id 'stb-40-srv-001.uib.local'>   (__init__.pyo|171)
(...)
[5] [Dec 02 18:25:28] [ event processing gui_startup  ] Mounting depot share smb://stb-40-srv-001/opsi_depot   (EventProcessing.pyo|415)
(...)