BLog

Imprint	Impressum
Privacy	Datenschutz
Disclaimer	Haftung
Downloads

FreeBSD: Dual Mode updating of installed ports and packages

FreeBSD offers two options for installing qualified third party software from its Ports facility. Ports can be built and installed from its sources or pre-built packages may be downloaded from a respective repository and be installed on the local machine.

Building of ports may take a considerable amount of time, and for this reason we want to install pre-built packages whenever possible. Of course packages are pre-built with a common set of options, which in most cases matches the requirements, but, there are always these notorious misfits. For example, I need devel/subversion with SASL support built-in, I want net/netatalk3 without net/avahi, I must have PostgreSQL support built-in to devel/apr, etc.

In the case of deviations from the common feature set, people recommend to built everything on the local machine from the port’s sources, and those people continue telling you that it is too risky to intermix pre-built packages with your customized port-builds.

In a certain sense, this is even correct, namely, without a serious risk assessment the best educated guess of the experts is always that many things may go wrong. And of course the conclusion from this is that Dual Mode maintenance of ports, i.e. mixed installation from locally built and pre-built packages is too risky for even trying it.

A package with the common set of options can be created locally from the corresponding port with the make package command and by leaving the default options in the configuration dialog in place. The thus locally created package would be identical to the corresponding package of the same version in the public repository of pre-built packages.

This identity of locally built vs. pre-built packages is one of the essential bases for proceeding further. The other base is that the installation phase is agnostic to the origin of the package - a package is a package is a package is ... That said, any mechanism of Dual Mode installation and updating of ports must stay on these bases, and given this constraint, the risk that something might go wrong is as high as with any other updating procedure. Anyway, the risks are:

1. First installation of a port/package

It may happen, that a selection of ports and options may conflict with installed packages. These conflicts must be resolved before execution of the maintenance script shown below, by calling pkg delete on the infringing package and adding the port’s name to the list of ports at the head of the script.

2. Updating of the installed ports/packages collections

In case pkg(8) is configured for the latest repository, the ports tree is ahead to the package repository by a few days (usually 1-3) only, therefore you might want to create and edit the following file:
mkdir -p /usr/local/etc/pkg/repos
nano /usr/local/etc/pkg/repos/FreeBSD.conf

FreeBSD: {
  url: "pkg+https://pkg.FreeBSD.org/${ABI}/latest",
}

In rare cases, it happens that a custom port depends on a new version of a pre-built package which didn’t made it yet into the remote repository. In this case we may wait for more 1 to 3 days and start the maintenance script again, or we manually run a local onetime build of the dependency before executing the script.

Special occasions, which usually show-up in the /usr/local/UPDATING file, need special actions before we can let the maintenance script shown below do its automatic job. For example the switch from mail/dovecot2 to mail/dovecot needed to be done outside of the automatic updating sequence.

The command pkg upgrade shows a list of actions to be done, and it asks whether to proceed [y/n]. If pkg wants to reinstall some of your custom ports because of changed options, then we need to tell n(o) here, because this means, that package dependencies have changed compared to our first installation. In order to resolve these kind of conflicts, we would need to customize the package, i.e. call pkg delete on it and put it into the list of the ports at the head of our script.

Now, here comes the software update shell script which I run frequently in order to keep my systems up-to date:

#!/bin/sh

### the list of the ports that shall be updated from sources
portslist="\
security/cyrus-sasl2 \
devel/apr1 \
devel/subversion \
devel/git \
graphics/cairo \
graphics/GraphicsMagick \
lang/php74 \
www/mod_php74 \
\
archivers/php74-phar \
archivers/php74-zip \
archivers/php74-zlib \
converters/php74-iconv \
converters/php74-mbstring \
databases/php74-mysqli \
databases/php74-pdo \
databases/php74-pdo_sqlite \
devel/php74-json \
devel/php74-intl \
devel/php74-tokenizer \
graphics/php74-exif \
security/php74-filter \
security/php74-openssl \
sysutils/php74-fileinfo \
sysutils/php74-posix \
textproc/php74-ctype \
textproc/php74-dom \
textproc/php74-simplexml \
textproc/php74-xml \
textproc/php74-xmlreader \
textproc/php74-xmlwriter \
www/php74-opcache \
www/php74-session \
"

### fetching FreeBSD system updates
/usr/bin/printf "Fetching FreeBSD system updates...\n"
PAGER=/bin/cat
/usr/sbin/freebsd-update fetch

### fetching updates of the FreeBSD ports tree
/usr/bin/printf "\nFetching updates of the FreeBSD ports tree...\n"
/usr/sbin/portsnap fetch update
/usr/sbin/pkg version -v
/usr/sbin/pkg updating -d `date -v-2w +%Y%m%d`

### ask and in case of y|Y run the updating processes
/usr/bin/printf "\nDo you want to continue (y/n)? "
save_stty_state=$(stty -g); stty raw -echo; answer=$(head -c 1); stty $save_stty_state
if echo "$answer" | grep -iq "^y" ; then
   /usr/local/etc/periodic/daily/411.pkg-backup

   /usr/bin/printf "\n\n"
   /usr/sbin/pkg update

   portmake=""
   pkgslist="`/usr/sbin/pkg query %o`"
   for port in $portslist ; do
      for pkg in $pkgslist ; do
         if [ "$pkg" == "$port" ] ; then
            continue 2
         fi
      done

      portmake="$portmake $port"
   done

   pkgslist=""
   outdated=`/usr/sbin/pkg version -l\< | /usr/bin/cut -f1 -w`
   for outd in $outdated ; do
      for port in $portslist ; do
         if [ "$port" == "`/usr/sbin/pkg query %o $outd`" ] ; then
            portmake="$portmake $port"
            continue 2
         fi
      done

      pkgslist="$pkgslist `/usr/sbin/pkg query %n $outd`"
   done

   pkgupgrd=""
   outdated=`/usr/sbin/pkg version -RU -l\< | /usr/bin/cut -f1 -w`
   for outd in $outdated ; do
      outd=`/usr/sbin/pkg query %n $outd`
      for pkg in $pkgslist ; do
         if [ "$pkg" == "$outd" ] ; then
            pkgupgrd="$pkgupgrd $outd"
            continue 2
         fi
      done
   done

   /usr/bin/printf "\nUpdating binary packages...\n"
   if [ "$pkgupgrd" != "" ] ; then
      /usr/sbin/pkg upgrade -U $pkgupgrd
   else
      echo "All installed packages are up-to-date."
   fi

   /usr/bin/printf "\nUpdating ports...\n"
   if [ "$portmake" != "" ] ; then
      for port in $portmake ; do
         echo "$port"
      done

      ### ask and in case of y|Y run the updating processes
      /usr/bin/printf "\nDo you want to continue (y/n)? "
      save_stty_state=$(stty -g); stty raw -echo; answer=$(head -c 1); stty $save_stty_state
      /usr/bin/printf "\n\n"
      if echo "$answer" | grep -iq "^y" ; then
         cwd=$PWD
         py="py37-"
   
         for port in $portmake ; do
            cd "/usr/ports/$port“
            if [ "$?" == "0" ] ; then
               /usr/bin/make deinstall clean
               if [ "${port#$py}" != "$port" ]; then
                  /usr/bin/make FLAVOR=py37 install clean
               else
                  /usr/bin/make install clean
               fi
            fi
         done
   
         cd "$cwd"
      fi
   else
      echo "All installed ports are up-to-date."
   fi

   /usr/bin/printf "\nCleaning up...\n"
   /usr/sbin/pkg clean -ay

   /usr/bin/find /usr/ports/distfiles -type f -mtime +4w -delete
   /usr/bin/find /usr/ports/distfiles -type d -depth +0 -and -empty -delete

else

   /usr/bin/printf "\n\n"

fi

The above script does what pkg(8) and portmaster(8) are not able to do, namely, ignore some ports of a software installation in the pkg upgrade process, and ignore the other packages when building named ports from source.

At the head of the script we need to inform the ports which must not be upgraded from the binary repository but shall be build from the ports tree - of course, the list of ports $portslist needs to be adapted for other machines, and maintained whenever a new port with custom options is installed on the system.

In the first stage, the script fetches the latest updates of the FreeBSD ports tree by the way of portsnap(8), and it shows a list of all installed ports with indication of the version status. It will also show the entries in /usr/ports/UPDATING related to the installed software of the last two weeks. Depending on that information the user might decide to stop or to proceed.

In case of proceed, it updates the local copy of the repository catalogue(s). Now the script compiles 3 lists, the list of all outdated software and more 2 divisions, namely outdated packages to be upgraded $pkgupgrd and outdated ports to be made $portmake. BTW, pkg lock does not work as advertised.

Finally it calls the command pkg upgrade -U $pkgupgrd which takes care only for the outdated packages, and then it loops through the list of the outdated ports, stepping into each of the respective port’s directory and calling the usual make deinstall/make install combo.

Discussion on Twitter: 1082813044567298049