Stages

The whole process of SRPM cleanup is divided onto 5 following stages:

• Stage I: SRPM packages are rebuilt using hsh-separate-sources(1) and a list of unused files is written down for each package.
• Stage II: SRPM packages are rebuilt using srpm-cleanup(1) and a TAR with source files is written down for each package. Comparing to the original package, the resulting TAR doesn´t contain unused files.
• Stage III: For each TAR package the corresponding SRPM package is built using srpm-cleanup(1).
• Stage IV: SRPM packages from Stage III are rebuilt with hsh-separate-sources(1) in verification mode. The verification process checks there are no new unused files. However, in some situations that check should be toolerated (see TOOLERANCE below).
• Stage V: The resulting verified SRPM packages are rebuilt with additional options (for example --with-pre and --with-post). For packages that have failed with verification the corresponding original SRPMs are rebuilt instead.

On each stage the packages are rebuilt in parallel inside a number of hasher(1) slots on a set of build hosts. The results of that builds and the build log files are transferred from build hosts and collected in the configured directories. The results of one stage are used on the next stage.

When a package is processed the output of used tools such as hsh-separate-sources(1) is written to the stage log file. The tarnscript of low-level operation goes to the build log file. After the whole stage is passed the corresponding stage status file is updated. That status file is constituted from the stage log records plus a special status record for each package that is deduced from the stage log records related to it. With the stage status file it is very easy to find the resulting status of each particular package.

Knowing the resulting status of each package it is possible to retry the stage processing for only a subset of packages based on thier status and after making the necessary configuration changes (see "Stage retries" below).

For each stage the following common targets are available:

• stageN: Run the stage N for packages that havn´t yet been processed. Run the stage on all configured BUILDERS on all architectures (see CONFIGURATION below). The values for N are 1..4, 5v and 5o. The last two are for the separate substages of Stage V: rebuilt of verified and the original packages.
• stageN-status: Show statistics for stage N.
• list-stageN-C[-A]: List names and versions of packages that fall into category C and optionally architecture A in stage N. The packages are divided onto categories using information from the stage status file.

Available architectures A are those for which BUILDERS are defined (see CONFIGURATION below).

Popular C examples among different stages are:

• done: successfully built packages,
• noatt: there were no attempts to build these packages yet,
• excl: build failed because of ExclusiveArch or ExcludeArch declaration,
• blderr: any kind of build error,
• brkndep: build failed because of broken dependencies,
• chkerr: build failed in the %check section,
• nosrpm: the builder was unable to find the SRPM file for the given package,
• nospace: build failed due to no space left on device,
• noinode: build failed due to lack of inodes,
• logerr: hsh-separate-sources(1) failed to analize the build log,
• other: any other error,
• unused: the packages are not listed in SRPM_LISTS (see CONFIGURATION below).

Stage retries

When a lot of packages are processed, building some of them are often failed --- that´s normal. In that situation the first thing is to investigate the situation, then fix the problem and re-run the stage for the failed packages only.

To help the investigation the following additional targets are available for each stage:

• list-stageN-C[-A]-logs: List paths to the build logs of packages that fall into category C and optionally architecture A in stage N.
• show-stageN-C[-A]-status: Output the stage status log entries related to the packages that fall into category C and optionally architecture A in stage N.
• show-stageN-pkg-status: Output the stage status log entries related to a single package specified with PKG=PKG. For stages I, IV-V the architecture can be additionally limited with ARCH=ARCH.

To re-run the stage for the failed packages only use:

• stageN-retry-C[-A]: Retry the stage N for packages that fall into category C and optionally architecture A.
• stageN-retry-pkgs

If INDEP_LOGS (see CONFIGURATION) is set then a number of such jobs can be run for the same stage in parallel.

To re-run the stage for all packages (failed, done and not tried) use:

•

stageN-rebuild-all: Run stage N from scratch.

When re-running the stage it is possible to alter the configuration of builders, for instance, to change the number of hasher slots --- see BUILDERS in CONFIGURATION below. Additionaly, for Stage V (both 5v and 5o parts) it is possible to run the process for only a subset of architectures: use FOR_ARCHES=<arch1,arch2...> on the command line.

Partial Stage V targets

For Stage V the following partial subtartets are available:

• stage5v|o-all: Run the stage for not yet tried packages included in all defined architectures.
• stage5v|o-A: Run the stage for not yet tried packages of architecture A.

Complete stage rebuilds

The run of a stageN target selects and process packages that weren´t processed on that stage yet (that are the packages from noatt category). In order to run the stage for all listed SRPM packages use stageN-rebuild-all target. However, the hsh-separate-sources(1) and srpm-cleanup(1) tools are known to skip the already processed packages anyway. So, to really rebuild all packages delete the existing results with stageN-clean or add -f|--force to the OPTS (see Runtime options below).

Rebuilds with manual package selection and multistage retries

• stageN-retry-pkgs: Re-run stage N for a specified list of packages. These targets expect name-version-release of packages to be specified on the command line with PKGS=<PKGS...>. In order to rebuild a package forcily pass -f|--force to the OPTS (see Runtime options below).
• reclean-pkgs: Runs stages II and III for the selected packages specified with PKGS=<PKGS...>.
• reclean-and-rebuild-pkgs: Runs stages II, III and IV for the selected packages specified with PKGS=<PKGS...>.
• relook-pkgs: Runs stage I in force mode for the selected packages specified with PKGS=<PKGS...>.
• relook-and-reclean-pkgs: Runs stages I--III for the selected packages specified with PKGS=<PKGS...>.
• relook-and-rebuild-pkgs: Runs stages I--IV for the selected packages specified with PKGS=<PKGS...>.
• clean-and-verify-noatt: Runs stages I--IV for the selected packages specified with PKGS=<PKGS...> with force mode off.

Dry runs

For each stage target, either the main stage target or any retry or additional target, there is the corresponding -dry-run target available. In the dry-run mode the stage isn´t actually run. Instead of that the list of command lines that would be executed in the normal stage run is printed out.

Moreover, for each stage target the corresponding -dry-run-run target is also available. In that "semi-dry" mode the tools are actually run (on remote hosts, if configured --- see BUILDERS), but each tools is run with --dry-run option.

In both dry-run modes no packages are built, the log files are not written and the stage status is not updated.

Building external SRPMs

It is possible to rebuild out of the repo SRPM packages passing WITH_SRPMS=EXTRA_DIR to any stage target. However, dist-tags of such packages are undefined!

Check and export

•

check-rebuild-lists: This target outputs the final statistics after stage IV separated the onto four following categories: done-included-all --- successfully rebuilt packages, included on all architectures; done-xincluded --- successfully rebuilt packages, included on some architectures (for each architecture); not-done-included-all --- failed packages, included on all architectures; not-done-xincluded --- failed packages, included on some architectures (for each architecture).

These package lists constitute a plan for stage V. The successfully rebuilt packages go to V/1 substage and the rest goes to V/2 (rebuilt of the original packages).

The target checks for intersections between the resulting lists and package coverage on each architecture (i. e. the number of all packages included in SRPM_LISTS versus the number of packages, planned for stage V).

•

check-rebuild-results-A: The target outputs statistics similar to check-rebuild-lists, but from stage V point of view: the number of planned verified and original builds can be compared with actual V/1 and V/2 results. The additional feature of the target is that the coverage is calculated twice: first as a plain number of packages successfully rebuilt on stage V, and a second time --- with additional number of packages, failed due to architecture exclusion (i. e. ExcludeArch and ExclusiveArch tags in the RPM spec file).

The statistic is output for the specified architecture A only.

• check-versions: Runs check-included and then check-disttags if DISTTAG_OPTS is defined (see Configuring Stage V).
• check-included: Output the diff between SRPM_LISTS and the packages, actually included into processing on the stages I-V. Normally, the diff should be empty.
• check-disttags: Compares the dist-tags of packages in the reference repository REPO_DIR with dist-tags in the stage V resulting packages in REBUILD_V_DIR and REBUILD_O_DIR directories (see Configuring Stage V). Normally, the diff should be empty.
• export: Copy stage V results from REBUILD_V_DIR and REBUILD_O_DIR directories along with index files into the target directory DESTDIR=DESTDIR (can be specified on the command line or configured in the Makefile). Any extra RPM files in DESTDIR are removed making the source and destination synchronized.

Additional options can be passed to rsync(1) via EXPORT_OPTS=EXPORT_OPTS.

• repo: Constructs an APT repository at OUTREPO (repo by default) with clean, orig, clean-dbg and orig-dbg packages sections.
• export-repo: Copies the repository to DESTDIR. By default the symbolic links to packages are not resolved. In order to copy the linked files add -L to EXPORT_REPO_OPTS (see rsync(1) for more information).

Clean targets

• statclean: Deletes all temporary files used to calculate statistic values for stageN-status targets.
• clean-stage1-<arch>-unused: Delete SKIPLIST_DIR lists for the packages that aren´t currently included in SRPM_LISTS for the architecture arch.
• clean-stage2-unused: Delete TAR_DIR files for the packages that aren´t currently included in SRPM_LISTS.
• clean-stage3-unused: Delete SRPM_DIR packages that aren´t currently included in SRPM_LISTS.
• clean-stage4-<arch>-unused: Delete RPM_DIR packages that aren´t currently included in SRPM_LISTS for the architecture arch.
• clean-stage5v-unused: Delete REBUILD_V_DIR packages that aren´t currently included in SRPM_LISTS.
• clean-stage5o-unused: Delete REBUILD_O_DIR packages that aren´t currently included in SRPM_LISTS.
• clean-indices: Remove the index files, first of all, the DISTTAG_INDEX file used for stage V (see Configuring Stage V). The indices can explicitly be re-created with index-rebuild target.
• stageN-clean: Delete all results and status files of stage N. WARNING! these targets are dangerous! In order to protect the results from accidentally deletion a confirmation is asked.

Other targets

• list-srpms: Output the list of all included SRPMs.
• show-builders: Output the resulting command-line for the configured BUILDERS (see the CONFIGURATION below) that is used for rebuilding of binary packages (stages I, IV-V).
• list-builders: Displays the table of the defined builders with their basic parameters.
• add-builder: Adds the host with specified parameters (see below) to the list of builders.
• del-builder: Deletes the given host from the list of builders.
• show-cleaners: Output the resulting command-line for the configured CLEANERS (see the CONFIGURATION below) that is used to clean-up source packages (stage II).
• list-cleaners: Displays the table of the defined cleaners with their basic parameters.
• add-cleaner: Adds the host with specified parameters (see below) to the list of cleaners.
• del-cleaner: Deletes the given host from the list of cleaners.
• show-srpm-builders: Output the resulting command-line for the configured BUILDERS (see the CONFIGURATION below) that is used for rebuilding of source packages (stage III).
• show-repo-dir: Displays the current value of REPO_DIR.
• set-repo-dir: Sets the REPO_DIR value to REPO_DIR=repo_dir specified on the command line.
• show-srpm-lists: Displays the current contents of SRPM_LISTS. Especially suitable if automatic source list configuration via AUTO_SRPM_LIST_DIR is set up.
• update-srpm-versions: Runs the update-versions.sed script (see below) against the SRPM_LISTS files.

Scripts

•

update-versions.sed: The script that is automatically prepared from BIN_LIST (depends on REPO_DIR) and can be used to update version numbers in the existing SRPM lists (manually or by the update-srpm-versions target).

Automatic source list configuration

•

AUTO_SRPM_LIST_DIR: Put *-<arch>.srpms files into AUTO_SRPM_LIST_DIR in order to automatically add to SRPM_LISTS the latest file for each architecture. In the example Makefile AUTO_SRPM_LIST_DIR is set to srpm-lists.

If you have two or more different kinds of lists (for example, one that defines packages for server and another for the desktop) and want to use the automatic source list configuration, you have to put them into different subdirectories within AUTO_SRPM_LIST_DIR.

To see the current contents of SRPM_LISTS run the show-srpm-lists target.

Manual source list configuration

•

SRPM_LISTS: Defines source package list files for each architecture. Each set of files should be added to SRPM_LISTS using $(call def_srpm_list,arch,files...), where arch is the name of an architecture and files is a space-separated list of files listing source RPM file names (name-version-release.src.rpm) line by line.

For example:

SRPM_LISTS = $(call def_srpm_list,ppc64le,lists/alt-sp8-server-latest-ppc64le.iso.srpms)
SRPM_LISTS += $(call def_srpm_list,aarch64,lists/alt-sp8-server-latest-aarch64.iso.srpms lists/alt-sp8-workstation-latest-aarch64.iso.srpms)
SRPM_LISTS += $(call def_srpm_list,armh,lists/alt-sp8-workstation-01-mcom02-latest-armh.img.srpm-list)
SRPM_LISTS += $(call def_srpm_list,i586,lists/alt-sp8-server-latest-i586.iso.srpms lists/alt-sp8-workstation-latest-i586.iso.srpms)
SRPM_LISTS += $(call
def_srpm_list,x86_64,lists/alt-sp8-server-latest-x86_64.iso.srpms
lists/alt-sp8-workstation-latest-x86_64.iso.srpms)

Configuring buidlers

To simplyfy adding builders there is the add-builder target:

add-builder HOST=<host> ARCH=<arch> SLOTS=<slots> REPO=REPO_DIR|<repo> [WORKDIR=<workdir>] [NPROCS=<nprocs>] [NOT_CLEANER=]´

The specified host is added to the list of BUILDERS. If NOT_CLEANER=1 isn´t passed, the specified host is also added to the list of CLEANERS.

If DEFAULT_NPROFILE is defined, then the add-builder target initializes the host.nprofile file from the specified file.

In order to delete the builder you can use the del-builder target:

del-builder HOST=<host> [NOT_CLEANER=]

The specified host is removed from the list of BUILDERS. If NOT_CLEANER=1 isn´t passed, the specified host is also removed from the list of CLEANERS.

•

BUILDERS: Defines the set of local and remote builders (hosts where hsh-separate-sources(1) can be run). Each builder should be added to BUILDERS using $(call def_builder,[user@]hostname,arches...,slots,workdir,repo[,nprofile]), where [user@]hostname defines the name of the remote host and an optional user name to connect with, arches is a space-separated list of architecture names, slots is a list of a range of Hasher slot numbers, workdir is the path to the working directory (on the remote host), repo is the path to package repository (on the remote host) and nprofile is the optional name of a CPU profile file.

All of the def_builder argumentsare match the command line options of hsh-separate-sources(1). In order to build on the local machine use : for the name of the host and don´t specify the user name.

Example:

``` BUILDERS = $(call def_builder,ppc,ppc64le,$(ppc_slots),/tmp/.private/builder/hashers,$(nfs_repo),$(nprocs))

BUILDERS += $(call def_builder,taishan1,aarch64 armh,$(arm_slots),/tmp/.private/builder/hashers,$(nfs_repo),$(nprocs))

BUILDERS += $(call def_builder,builder_new@ivkbuilder,x86_64 i586,$(ivk_slots),/tmp/.private/builder_new/hashers,$(ivk_repo),$(nprocs)) ```

Configuring cleanders

To simplyfy adding builders there is the add-cleaner target:

add-cleaner HOST=<host> SLOTS=<slots> SRPMS=<srpms> | REPO=REPO_DIR|<repo> [WORKDIR=<workdir>]

The specified host is added to the list of CLEANERS. In order to delete the cleaner you can use the del-cleaner target:

del-cleaner HOST=<host>

The specified host is removed from the list of CLEANERS.

•

CLEANERS: Defines the set of local and remote cleaners (hosts where srpm-cleanup(1) can be run). Each cleaner should be added to CLEANERS using $(call def_cleaner,[user@]hostname,slots,workdir,srpms), where [user@]hostname defines the name of the remote host and an optional user name to connect with, slots is a space-separated list of a range of Hasher slot numbers, workdir is the path to the working directory (on the remote host) and srpms is the path to the directory with SRPM files (on the remote host).

All of the def_cleaner argumentsare match the command line options of srpm-cleanup(1). In order to build on the local machine use : for the name of the host and don´t specify the user name.

Example:

CLEANERS = $(call def_cleaner,ppc,1-40,/tmp/.private/builder/hashers,$(nfs_repo)/files/SRPMS)
CLEANERS += $(call def_cleaner,taishan1,1-40,/tmp/.private/builder/hashers,$(nfs_repo)/files/SRPMS)
CLEANERS += $(call def_cleaner,builder_new@ivkbuilder,1-40,/tmp/.private/builder/hashers,$(ivk_repo)/files/SRPMS)

• EXTRA_ARCHES: Specify extra arches of those builders that aren´t currently configured.
• REPO_DIR: Defines the default repository path for add-builder and add-cleaner targets. Also used in the configuration of Stage V.
• DEFAULT_NPROFILE: Defines the name of the reference NPROC profile file that is used by add-builder target.

Remote options

• SSHDELAY: Log-in to the next remote host after a delay. Setting this to a positive non-zero value helps against SSH errors when a lot of parallel remote processes are started.
• SELFIE: Tell the tools to copy theirselves on the remote hosts.
• REMOTE_DIR: The directory to use as the current when executing tools on remote hosts. If is set to ... then a temporary directory is used which is deleted after execution. Using this value is a good alternative to CLEANUP=1.
• WITH_NPROFILE: Copy NPROC profile to remote builders. To select files automatically based on the name of the host set it to %h.nprofile (see --with-nprofile in hsh-separate-sources(1)).
• CLEANUP: Set this to enable --cleanup mode with hsh-separate-sources(1) and srpm-cleanup(1) in order to save space on the remote hosts. Note, that --cleanup is incompatible with --selfie, so the scripts have to be copied to the hosts beforehand (see also Tools).

Tools

• HSH_SEPARATE_SOURCES: Sets the path to the hsh-separate-sources(1) script on all remote hosts (no customization yet).
• SRPM_CLEANUP: Sets the path to the srpm-cleanup(1) script on all remote hosts (no customization yet).

The default is to use the script from the current directory, then --- the script in PATH.

• COMMON_OPTS: The set of common options to pass to the tools.
• BUILD_COMMON_OPTS: The set of common options passed to hsh-separate-sources(1).
• CLEAN_COMMON_OPTS: The set of common options passed to srpm-cleanup(1).

Don´t override the common options by accident as their default values contain important things about the remote options described above.

Log options

• LOG_SUF: Defines a suffix to use for stage log file names. Using the same stage log file name prevents from running multiple jobs for the same stage (for instace, retries with different options --- see Stage retries) simultaneously. That´s become possible if a different log filename suffix is used with each job. If INDEP_LOGS is set, i.e. not empty (the default) then random suffixes are used.
• INDEP_LOGS: Controls if random LOG_SUF values should be used for each stage run. The flag is set by default (a non empty value).
• BACKUP_LOGS: Set to a non-empty value to disable independent logs and enable making backups of the stage log file on each next stage run (including stage retries). Setting BACKUP_LOGS to a non-empty value prevents from running multiple jobs for the same stage simultaneously.

Configuring Stage I

• SKIPLIST_DIR: Directory to write the list of unused packages for each processed source package. The default is skip.
• INCLIST_DIR: Directory to write the list of used packages for each processed source package. The default is inc.
• STAGE1_LOG: Log file for Stage I. The default is logs/stage1.log.
• STAGE1_LOGDIR: Directory for Stage I package build logs. The default is logs/stage1.logs.
• STAGE1_STATUS: Status file for Stage I. The default is stage1.status.

Configuring Stage II

• TAR_DIR: Directory to output TAR packages to. The default is pkgs.
• STAGE2_LOG: Log file for Stage II. The default is logs/stage2.log.
• STAGE2_LOGDIR: Directory for Stage II package build logs. The default is logs/stage2.logs.
• STAGE2_STATUS: Status file for Stage II. The default is stage2.status.
• EXCEPTIONS: A file with the list of exception paths for srpm-cleanup(1) (-x|--exceptions). The default is exceptions.list. The list is also used with Stage IV.

Configuring Stage III

• SRPM_DIR: Directory to output SRPM packages to. The default is out.
• STAGE3_LOG: Log file for Stage III. The default is logs/stage3.log.
• STAGE3_LOGDIR: Directory for Stage III package build logs. The default is logs/stage3.logs.
• STAGE3_STATUS: Status file for Stage III. The default is stage3.status.

Configuring Stage IV

• RPM_DIR: Directory to output the resulting RPM packages to. The default is rpms.
• STAGE4_LOG: Log file for Stage IV. The default is logs/stage4.log.
• STAGE4_LOGDIR: Directory for Stage IV package build logs. The default is logs/stage4.logs.
• STAGE4_STATUS: Status file for Stage IV. The default is stage4.status.
• TOOLERANCE: SRPM cleanup verification toolerance, %. The default is 0%.

If EXCEPTIONS (see above) is defined, the name of the file is passed to hsh-separate-sources(1) with -x|--exceptions).

• REPO_DIR: Path to the reference package repository (on the local host). The value is also used by add-builder and add-cleaner targets as the default package repository path. See also show-repo-dir and set-repo-dir targets.
• BIN_LIST: Path to the bin.list file in the reference repository on the local host. The default is $(REPO_DIR)/files/list/bin.list.
• DISTTAG_INDEX: The name of the dist-tag index file. The dist-tag index depends on the BIN_LIST and is updated automatically. The default is disttag.index.
• DISTTAG_OPTS: Options to pass to hsh-separate-sources in order to enable dist-tags. The default is --disttag-index="$(DISTTAG_INDEX)".
• CUSTOM_TAGS: A file with a list of tab-separated pairs of package-version-release and custom dist-tag values.

Configuring Stage V

• REBUILD_OPTS: Additional options to pass to hsh-separate-sources on Stage V. For example, the --with-pre and --with-post options. There is no default value for this option.
• REBUILD_V_DIR: Directory to write the verified, cleaned-up RPM packages to. The default is rebuild.v.
• STAGE5_V_LOG: Log file for Stage V/1. The default is logs/stage5.v.log.
• STAGE5_V_LOGDIR: Directory for Stage V/1 package build logs. The default is logs/stage5.v.logs.
• STAGE5_V_STATUS: Status file for Stage V/1. The default is stage5.v.status.
• STAGE5_V_EXCLUDE_LIST: Name of the file with a list of package base names (no version and release) be excluded packages from Stage V/1. The listed packages are included in Stage V/2 even if they have sucessfully passed Stage IV. The default name is exclude.list.
• REBUILD_O_DIR: Directory to write the RPM packages resulted from rebuilding of the original SRPM packages, typically --- the packages that failed on Stage IV. The default is rebuild.o.
• STAGE5_O_LOG: Log file for Stage V/2. The default is logs/stage5.o.log.
• STAGE5_O_LOGDIR: Directory for Stage V/2 package build logs. The default is logs/stage5.o.logs.
• STAGE5_O_STATUS: Status file for Stage V/2. The default is stage5.o.status.

Runtime options

• OPTS: Any command line options to pass to a tool corresponding to the stage.
• FOR_ARCHES: Run the stage (I, IV-V) on the hosts of the specified architectures only. The architectures can be listed separated by spaces or commas.
• PKGS: List of packages to rebuild (stageN-retry-pkgs targets only).
• LIMIT: Limit the work plan by LIMIT packages (see the --limit option in hsh-separate-sources(1) and srpm-cleanup(1)).
• SLOTS: Overrides the builder-specific slots value.
• NPROCS: Overrides the builder-specific nprocs (nprofile) value.