opsi WAN/VPN extension

The WAN/VPN extension offers the possibility to efficiently manage clients that are connected via slow lines. This documentation is intended to explain the functionality of this extension of opsi and provide a guide on how to configure and maintain this extension. Please note that the extensions Installation on shutdown and WAN/VPN cannot be used simultaneously on one client.

Preconditions for using the WAN/VPN extension

This opsi extension is currently in the cofunding process and not free. For more details see opsi extensions.

General overview of the WAN/VPN extension

opsi software deployment is mainly doing the following steps:

  • The 'opsi-Loginblocker' at client system startup prevents the users from logging on.

  • The 'opsiclientd' service running on the client connects the opsi config server.

  • If any product actions are set for the client, it mounts a share from the opsi-depot.

  • The 'opsi-winst' is starting and also connects to the opsi config server.

  • The 'opsi-winst' executes the product actions, using the share from the opsi-depot.

  • If a reboot is required, it executes and the process starts all over.

  • When all the product actions are completed, the logfiles are transferred to the opsi config server and the user logon is unblocked.

Now we will look at the special circumstances of a client, which is located in a remote branch, connected via 'WAN' to the 'LAN', where the opsi config server and opsi depot server are:

  • During communication with the opsi config server small amounts of data are transferred, so there is no noticeable slowdown of the software deployment in a WAN.

  • But processing the product actions might consume a very long time, depending on the packet sizes, bandwidth and latency of the 'WAN' connection. There also might occur timeouts during file access.

  • Therefore, during the installation is processing, the user has to wait for an unreasonably long time before logon is granted.

As an alternative to providing a dedicated 'opsi-Depotserver' in the remote branch network, the remote clients can be connected via WAN/VPN extension module. Using the WAN/VPN extension module, the 'opsi-client-agent' can be configured this way:

  • At system startup, if there are no opsi products cached and ready for installation, the user can logon immediately.

  • When there are product actions set for the client, the 'opsiclientd', which is running on the client, starts downloading the required installation files from the opsi-depot to the local file system. This is done in the background while the user is working. The maximum download bandwidth can be configured and also can be dynamically adapted to the current network traffic status.

  • When the synchronization of the opsi products is completed, a reboot request is triggered.

  • The logged on user can accept the reboot request, or the client will be rebooted at some time later on.

  • At the next system startup, the cache is found to be filled with the opsi products to be installed and the installation starts. In this case, the installation will use the downloaded files from the local file system, which increases the speed of installation even compared to a standard 'LAN' installation.

Now we examine the situation of a notebook, which at system startup often cannot connect the opsi config server:

  • Trying to connect the opsi config server at system startup will result in a timeout.

  • Connecting the opsi config server might be possible when a user logs on and a 'VPN' connection to the corporate network is established.

  • Without connection the opsi config server no software deployment is available.

This situation also can be solved by using the WAN/VPN extension module.
The 'opsi-client-agent' can be configured the following way:

  • At system startup, if there are no opsi products cached and ready for installation, the user can logon immediately.

  • Triggered by network activation or a by timer event, the 'opsiclientd' running on the client tries to connect the opsi config server.

  • If the opsi config server is reachable, the 'opsiclientd' starts to:

    • synchronize the configurations

    • download the required files from the opsi-depot to the local file system.
      In combination with the opsi extension module 'Dynamic Depot Selection', the download is done from the best fitting opsi-depot.

  • When the synchronization of the opsi products is completed, a reboot request is triggered.

  • The logged on user can accept the reboot request or the client will be rebooted at some time later on.

  • At the next system startup, the cache is found to be filled with the opsi products to be installed and the installation starts. In this case, the installation will use the downloaded files from the local file system, which increases the speed of installation even compared to a standard 'LAN' installation. So the 'opsiclientd' takes over the function of both the opsi config server and the opsi depot server.

  • At the next connect to the opsi config server the results of the installation process (configuration change, logfiles …​) will be synchronized.

The download mechanism of product synchronization is multiple interruptible and will continue at the point of interruption. So files that are already downloaded will not have to be downloaded again.

The WAN/VPN extension module allows to connect clients, that are outside of the secure corporate network. Therefore additional security mechanisms are required regarding the communication between client and server.
So the 'opsiclientd' now offers the ability to verify the identity of an 'opsi-Server'. Therefore the key pair of the SSL certificate of the 'opsiconfd' is used.
By this mechansim the opsi config server as well as the opsi depot server can be verified, assumed the communication is performed via 'opsiconfd' and 'SSL'. In case of an opsi-depot the file access must be done by the 'opsiconfd' using 'HTTPS'/'WEBDAVS'. Access done via 'CIFS'/'SMB' will not be checked.

Caching of opsi products

The caching of opsi products is handled by the ProductCacheService, which is part of the opsiclientd. The ProductCacheService synchronizes the local copies of the files contained in an opsi product with the files of the opsi product on an opsi depot. The base directory of the product cache is configurable and set to %SystemDrive%\opsi.org\cache\depot by default.

Communication Protocol for accessing an opsi-depot

For transferring the product files, two different protocols are used:

  • 'CIFS'/'SMB'

  • 'HTTP(S)'/'WEBDAV(S)'

In case of using 'CIFS'/'SMB', a connection to the 'depotRemoteUrl' will be established as configured with the properties of the opsi-depot. In case of using 'HTTP(S)'/'WEBDAV(S)', the 'depotWebdavUrl' is connected, which as well is to be configured with the properties of the opsi-depot.

Which protocol is to be used, can be configured client specific by the 'Hostparameter' clientconfig.depot.protocol. Available values to be set as clientconfig.depot.protocol are cifs and webdav.

Also the 'opsi-linux-bootimage' is evaluating this setting and uses the specified protocol.
With opsiclientd, a different protocol can be used for individual events via the depot_protocol attribute.

Using the .files file for Synchronization

The synchronization process is based on the file <product-id>.files, which is located in the base directory of each opsi product on the opsi-depot. This file contains information about the files, directories ans symbolic links used by an opsi product. Each line of that file contains such information. Different types of information are separated by a blank.
The first character of a line defines the type of the following entry. Available values are:

  • d for a directory

  • f for a file

  • l for a symbolic link

Separated by a blank follows the relative path, which is single quoted.
The next entry gives the sizes of the file (which is 0 for directories and symbolic links).
The final entry in case of a file is the MD5-sum of the file, in case of a symbolic link it is the target referred to by the symbolic link.

Example excerpt of a '.files' file:

d 'utils' 0
f 'utils/patch_config_file.py' 2506 d3007628addf6d9f688eb4c2e219dc18
l 'utils/link_to_patch_config_file.py' 0 '/utils/patch_config_file.py'

The '.files' file is generated automatically when installing product packages (after running the postinst-Script).

When using the WAN/VPN extension, the files of opsi products on the opsi-depot should not be changed manually, otherwise the information contained in the '.files' file would be outdated, causing errors during the synchronization process.

Internal processing of opsi product caching

The synchronization of a local copy of an opsi product processes as follows:

  • The '.files' file of the opsi product is transferred to the local client.

  • Then it is checked, whether there is enough local disk space available to cache the opsi products. If there isn’t enough disc space available, some old opsi products will be deleted, which haven’t been used (synchronized) for a long time.

  • The local caching directory will be created if it doesn’t exist.

  • Referring to the '.files' file, any old files and directories, which aren’t in use anymore, will be deleted from the local opsi product cache.

  • Then the '.files' file will be processed in the following order.

    • missing directories are created.

    • missing files are transferred.

    • existing files will be checked by size and MD5-sum and be synchronized again if necessary.

The synchronization results in an exact local copy of the opsi product from the opsi-depot.

On windows systems, no symbolic links will be created. Instead of links there will be copies of the link target.

When the opsi product has completed successfully,

Configuring the opsi product caching

The opsi product caching is configured in the section [cache_service] of the opsiclientd.conf.

  • product_cache_max_size (integer): The maximum size of the opsi product cache in byte. This is important to limit the disk space to be used by opsi product caching.

  • storage_dir (string): the path to the directory, in which the base directory depot for the opsi product caching is to be created.

Further configurations can be done event specific.
Within an event configuration section [event_<event-config-id>] the following options are available:

  • cache_products (boolean): if the value of this option is true, in case of the event the 'ProductCacheService' will start to cache opsi products, for which a product action is set. If additionally the value of the option use_cached_products is set to true, the further processing of this section will be suspended until the caching of opsi products is completed.

  • cache_max_bandwidth (integer): the maximum bandwidth in byte/s to be used for caching. If this value is set to 0 or less, the bandwidth is unlimited.

  • cache_dynamic_bandwidth (boolean): if the value of this option is set to true, the bandwidth will be adapted dynamically. Therefore the network traffic at the network interface to the opsi-depot will be monitored. If any traffic is detected, which is not caused by the opsi product caching, the bandwidth for the caching will be sharply reduced, to allow other processes to work at (almost) full speed. If the caching works at reduced bandwidth and no more other network activity but the opsi product caching is detected, the caching process will continue with unlimited bandwidth. The value of cache_max_bandwidth will be used to limit the maximum dynamic bandwidth.

  • use_cached_products (boolean): if the value of this option is set to true, the local opsi product cache will be used for processing product actions. If caching of the opsi products is not completed, the processing of this event will stop and return an error code.

Limitation of simultaneously running product synchronizations

To protect the network bandwidth and the opsi depot server, the number of simultaneously running caching processes can be limited by using the host parameter opsiconfd.transfer.slots_opsiclientd_product_sync. The value defined for a depot server determines the maximum number of simultaneously running product synchronizations for the respective depot, whereby the configured default value of the host parameter applies to depots without a specific value.

If the host parameter does not yet exist, it can be created as a standard configuration parameter via opsi-configed. Alternatively, the host parameter can also be created via the command line on the opsi config server.

opsi-admin -d method config_createUnicode opsiconfd.transfer.slots_opsiclientd_product_sync "Maximum number of simultaneous product synchronizations" 100 100 true false

Caching of configurations

The caching of configurations is done by the 'ConfigCacheService', which is part of the 'opsiclientd'.
The 'ConfigCacheService' synchronizes a local 'Client-Cache-Backend' with the 'Config-Backend' of the opsi config server.
The 'opsiclientd' provides per 'WebService' an access point to the backend and thereby provides quite the same functionality as the 'opsiconfd'.

The local 'Client-Cache-Backend'

The local 'Client-Cache-Backend' is based on 'SQLite' and mainly consists of a local working copy, a snapshot an a modification tracker, which records all changes of the local working copy.
The base directory of the config cache can be configured and defaults to %SystemDrive%\opsi.org\cache\config. The snapshot reflects the configuration status on the opsi config server at the time of the last synchronization.
At the start of the processing, the local working copy is a copy of the snapshot, but will be modified during processing.

Internal processing of configuration synchronizing

The synchronization of the local changes of the 'Client-Cache-Backend' with the 'Config-Backend' of the opsi config server is processed as follows:

  • The changes of the local working copy registered by the modification-tracker are transferred to the opsi config server. Any changes of the configuration on the opsi config server since the last synchronization will be detected by comparing to the snapshot. If there are any conflicts deteced, the following rules apply:

    • In case of inventory data, the local client data have priority.

    • For 'Action-Requests' the value of the client is valid, unless the version of the corresponding package has changed in the meantime on the server side. Then the server value is preferred.

    • In case of 'Installationsstatus' and 'Aktionsergebnis', the client data have priority.

    • If the opsi licenense management modul is switched on (config: 'license-management.use=true'), the config server tries to find a license pool for the product by the assignment pool to productId. I free license of this pool will be reserved and this license will be replicated. Any unused licenses, which have been reserved during replication, will be released again.

    • The new state of 'Hostparametern' and 'Product-Properties' is only transferred to the server if they have not been changed server-side in the meantime.

  • The modification tracker will be cleared.

  • The logfiles will be transferred.

The 'Config-Backend' replication of the opsi config server to the 'Client-Cache-Backend' is processed as follows:

  • The replication only takes place, if any 'Action-Requests' are set on the opsi config server. The product action always does not count in this respect. The replication process will start only if the status of 'Action-Requests' is changed since the last replication.

  • The modification tracker and the local working copy are cleared.

  • The configurations required for local processing will be replicated.

  • If 'Action-Requests' are set for opsi products which are marked as to require a license, the required software license will be reserved from a 'Lizenzpool', which is assigned to that opsi product.

  • Additionally required data, as there are the auditHardwareConfig and the modules, will be transferred.

  • The snapshot and the local working copy will be updated, so they have the same content.

A successful replication from server to client results in:

The sync completed is running until the next reboot, or until it is canceled by manually fired event (e.g. on_demand). In the latter case the config cache is marked invalid (which implies that the config has to be resynced - in case of changes) and the other event is processed.

Configuration of config caching

The configuration of the config caching is mainly done event specific:
Within an event configuration section [event_<event-config-id>], the following options are available:

  • sync_config_to_server (boolean): if the value of this option is set to true, the 'ConfigCacheService' in case of that event starts to transfer the changes registered by the modification tracker to the opsi config server. The process will wait for that task to complete.

  • sync_config_from_server (boolean): if this value is set to true, the 'ConfigCacheService' starts with the replication. If additionally the value of the option use_cached_config is set to true, the processing of this event is suspended until the replication is completed.

  • use_cached_config (boolean): if the value of this option is set to true, the 'Client-Cache-Backend' will be used for processing the product actions. If the synchronization is not completed, the processing of this event will be stopped and return an error code.

  • post_sync_config_to_server (boolean): has the same functionality as sync_config_to_server, but will be evaluated after the product actions have been completed.

  • post_sync_config_from_server (boolean): has the same functionality as sync_config_from_server, but will be evaluated after the product actions have been completed.

The 'opsi-client-agent'-package comes with a opsiclientd.conf prepared for the WAN/VPN extension.
For activating the WAN/VPN extension, just enabling of some events and disabling of some others is required.
The configuration of the 'opsi-client-agent' also can be done from the web service (see: opsi-client-agent web service), so it is recommended to create the following 'Hostparameter':

  • opsiclientd.event_gui_startup.active (boolean, default: true)

  • opsiclientd.event_gui_startup\{user_logged_in\}.active (boolean, default: true)

  • opsiclientd.event_net_connection.active (boolean, default: false)

  • opsiclientd.event_timer.active (boolean, default: false)

By these 'Hostparameter', events can be enabled or disabled client specific. The 'Hostparameter' can be created using the 'opsi-configed' or 'opsi-admin'.

For creating the 'Hostparameter' by 'opsi-admin', the following commands have to be executed on the opsi config server:

opsi-admin -d method config_createBool opsiclientd.event_gui_startup.active "gui_startup active" true
opsi-admin -d method config_createBool opsiclientd.event_gui_startup{user_logged_in}.active "gui_startup{user_logged_in} active" true
opsi-admin -d method config_createBool opsiclientd.event_net_connection.active "event_net_connection active" false
opsi-admin -d method config_createBool opsiclientd.event_timer.active "event_timer active" false

The default values are as they come with the special opsiclientd.conf.

If you do not set the defaults like described above and skip directly to the commands below you set all your clients into WAN mode !

For a WAN/VPN client, which shall do caching of configurations and opsi products, the 'Hostparameter' have to be configured as follows:

  • opsiclientd.event_gui_startup.active: false

  • opsiclientd.event_gui_startup\{user_logged_in\}.active: false

  • opsiclientd.event_net_connection.active: true

  • opsiclientd.event_timer.active: true

The client specific 'Hostparameter' can be set by 'opsi-configed' or 'opsi-admin'.

To set the 'Hostparameter' by 'opsi-admin', the following commands have to be executed on the opsi config server: (in this example the client has the 'opsi-Host-ID' vpnclient.domain.de):

opsi-admin -d method configState_create opsiclientd.event_gui_startup.active vpnclient.domain.de false
opsi-admin -d method configState_create opsiclientd.event_gui_startup{user_logged_in}.active vpnclient.domain.de false
opsi-admin -d method configState_create opsiclientd.event_net_connection.active vpnclient.domain.de true
opsi-admin -d method configState_create opsiclientd.event_timer.active vpnclient.domain.de true

This configuration will process as follows:

  • At system start of the client there will be no connection established to the opsi config server.

  • When the activation of a network interface is detected, a connection to the opsi config server will be established (if possible) and the synchronization starts as background task.

  • A timer-Event will be established, which tries at regular intervals to trigger the synchonization process.

The caching of opsi products can be done via the protocols 'HTTPS'/'WEBDAVS' or 'CIFS'/'SMB'.

When using 'webdav', access to the opsi-depot is performed by the 'opsiconfd'.

  • advantages:

    • easy firewall configuration, for it requires just port 4447.

    • verify of the SSL-certificate of the opsi-depot available.

  • disadvantage:

    • the 'opsiconfd' generates more traffic on the opsi-depot.

By using 'cifs', access to the opsi-depot is done via 'SAMBA'.

  • advantage:

    • the 'SAMBA'-server shows a good performance, is resource-conserving and well scaleable.

  • disadvantages:

    • the firewall configuration is more complicated, access to the SAMBA ports is required.

    • no verify of the SSL-certificate of the opsi-depot is available.

An instruction for configuring the protocols is to be found in the chapter Communication Protocol for accessing an opsi-depot.

ospclientd-infopage-wan-cached
Figure 1. processing of an installation with WAN extension as displayed in the opsiclientd infopage

To activate the verifying of SSL certificates, in the opsiclientd.conf within the section [global], the option verify_server_cert is to be set to true. This, during connection to an opsiconfd, results in verifying the 'opsi-Server' by using the SSL certificate. The server certificates will be stored in the directory %SystemDrive%\opsi.org\opsiclientd\server-certs. The name of the certificate is combined from the server address (IP or name) and the extension .pem. If at connection time no stored certificate is to be found, no checking will be done.

To publish a changed certificate, the old certificate stored on the clients has to be deleted. This can be done by the RPC-method deleteServerCerts, which is available from the control interface of the 'opsiclientd'.