Go to file
2020-07-20 16:07:26 +02:00
bsc impi: allow echo as compiler for mpitool 2020-07-20 16:06:00 +02:00
test Add test subset 2020-06-25 21:02:49 +02:00
.gitignore Ignore source folder 2020-07-20 16:07:26 +02:00
default.nix Remove custom nanos6 2020-07-20 16:05:32 +02:00
README Typo in git repo 2020-07-08 12:16:59 +02:00


                        BSC Nixpkgs: User guide


1 Introduction

  This repository contains a set of nix packages used in the Barcelona
  Supercomputing Center by the Programming Models group.

  Some preliminary steps must be done manually to be able to build and 
  install packages (derivations in nix jargon).

  This guide is specific for the nix installation in the xeon07 node, 
  accessed by the ssfhead.bsc.es login node, but it may be used in other 
  machines as well.

  To easily connect to xeon07 in one step, setup the SSH (for version 
  7.3 and upwards) configuration file in ~/.ssh/config adding these 
  lines:

    Host cobi
          HostName ssflogin.bsc.es
          User your-username-here

    Host xeon07
          ProxyJump cobi
          HostName xeon07
          User your-username-here

  You should be able to connect with:

    $ ssh xeon07

1.1 Network access

  In order to use nix would you need to be able to download the sources 
  from Internet. Usually the download requires the ports 22, 80 and 443 
  to be open for outgoing traffic.

  Unfortunately, in some clusters (as is the case in xeon07) access to 
  Internet is disabled. However you can tunnel the connection by SSH to 
  your local machine, and then reach the Internet.

  In order to tell nix to use the proxy connection, you will need to 
  export the "https_proxy" and "http_proxy" variables. A proxy 
  connection is already configured in xeon07 and you can automatically 
  set those variables to the correct address by loading:

    xeon07$ . /scratch/nix/internet

  Consider adding the command to your ~/.bashrc file so you don't need 
  to do it every time you want to use nix.

  Now you should be able to reach the outside world by running:

    xeon07$ curl google.com
    <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
    <TITLE>301 Moved</TITLE></HEAD><BODY>
    <H1>301 Moved</H1>
    The document has moved
    <A HREF="http://www.google.com/">here</A>.
    </BODY></HTML>

1.1 Prepare SSH keys

  Package sources are usually downloaded directly from the git server, 
  so you must be able to access all repositories without a password 
  prompt.

  Most repositories at https://pm.bsc.es/gitlab are open to read for 
  logged in users, but there are some exceptions (for example the nanos6 
  repository) where you must have explicitly granted read access.

  If you don't have a ssh key at ~/.ssh/*.pub in xeon07 create a new one 
  without password protection by running:

    xeon07$ ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (~/.ssh/id_rsa):
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in ~/.ssh/id_rsa.
    Your public key has been saved in ~/.ssh/id_rsa.pub.
    ...

  By default it will create the private key at ~/.ssh/id_rsa. Copy the 
  contents of your public ssh key in ~/.ssh/id_rsa.pub and paste it in 
  GitLab at:

    https://pm.bsc.es/gitlab/profile/keys

  Then, configure it for use in the ~/.ssh/config file, adding:

    Host bscpm02.bsc.es
      IdentityFile ~/.ssh/id_rsa

  Finally verify the SSH connection to the server works and you get a 
  greeting from the GitLab server with your username:

    xeon07$ ssh git@bscpm02.bsc.es
    PTY allocation request failed on channel 0
    Welcome to GitLab, @rarias!
    Connection to bscpm02.bsc.es closed.

  Verify that you can access rarias/nanos6 repository (otherwise you 
  first need to ask to be granted read access), at:

    https://pm.bsc.es/gitlab/rarias/nanos6
  
  Finally, you should be able to download the rarias/nanos6 git 
  repository without any password interaction by running:

    xeon07$ git clone git@bscpm02.bsc.es:rarias/nanos6.git

1.3 Prepare the bsc-nixpkgs repo

  Once you have Internet and you have granted access to the PM GitLab 
  repositories you can begin down the rabbit hole of nix. First ensure 
  that the nix binaries are available from your shell in xeon07:

    xeon07$ nix --version
    nix (Nix) 2.3.6

  Now you are set to install packages with nix. Clone the bsc-nixpkgs 
  repository:

    xeon07$ git clone git@bscpm02.bsc.es:rarias/bsc-nixpkgs.git

  Nix looks in the current folder for a file named "default.nix" for 
  packages, so go to the repo:

    xeon07$ cd bsc-nixpkgs

  Now you should be able to build nanos6 from the git repository:

    xeon07$ nix-build -A bsc.nanos6-git

  The output is placed in the "result" symbolic link.


2. Basic usage of nix

  Nix is a package manager which handles easily reproducibility and 
  configuration of packages and dependencies. See more info here:

    https://nixos.org/nix/manual/

  We will only cover the basic usage of nix for the BSC packages.

2.1 The user environment

  All nix packages are stored under the /nix directory. When you need to 
  "install" some binary from nix, a symlink is added to a folder 
  included in the $PATH variable. In particular, you should have 
  something similar added to your $PATH:

    xeon07$ echo $PATH | sed 's/:/\n/g' | grep nix
    /home/Computational/rarias/.nix-profile/bin
    /nix/var/nix/profiles/default/bin

  The first one is your custom installation of packages that are stored 
  in your home directory and the second one is the default installation 
  which contains the nix tools (which are installed in the /nix 
  directory as well).

  Use `nix search` to look for official packages in the "nixpkgs" 
  channel (the default repository of packages):

  xeon07$ nix search cowsay
  warning: using cached results; pass '-u' to update the cache
  * cowsay (cowsay)
    A program which generates ASCII pictures of a cow with a message

  * neo-cowsay (neo-cowsay)
    Cowsay reborn, written in Go

  * ponysay (ponysay-3.0.3)
    Cowsay reimplemention for ponies

  * tewisay (tewisay-unstable-2017-04-14)
    Cowsay replacement with unicode and partial ansi escape support

  When you need a program that is not available in your environment, 
  much like when you use "module load ..." you can use nix-env to modify 
  what is currently loaded. For example:

    xeon07$ nix-env -iA nixpkgs.cowsay

  Notice that you should specify the prefix "nixpkgs." before. The 
  command will download (if not found already in the nix store), compile 
  (if necessary) and load the program `cowsay` from the nixpkgs 
  repository in the environment. You should be able to run it as:

    xeon07$ cowsay "hello world"
     _____________
    < hello world >
     -------------
            \   ^__^
             \  (oo)\_______
                (__)\       )\/\
                    ||----w |
                    ||     ||

  You can now inspect the ~/.nix-profile/bin folder, and see that a new 
  symlink was added to the actual installation of the binary:

    xeon07$ file ~/.nix-profile/bin/cowsay
    /home/Computational/rarias/.nix-profile/bin/cowsay: symbolic link to 
    `/nix/store/673gczmhr5b449521srz2n7g1klykz6n-cowsay-3.03+dfsg2/bin/cowsay'

  You can list the current packages installed in your environment by 
  running:

    xeon07$ nix-env -q
    cowsay-3.03+dfsg2
    nix-2.3.6

  Notice that this setup only affects your user environment. Also, it is 
  permanent for any new session until you modify the environment again 
  and is immediate, all sessions will have the new environment 
  instantaneously.

  You can remove any package from the environment using:

    xeon07$ nix-env -e cowsay

  See the manual with `nix-env --help` if you want to know more details.

2.2 Building packages

  Usually, all official packages are already compiled and distributed 
  from a cache server so you don't need to rebuild them again. However, 
  BSC packages are distributed only in source code form as we don't have 
  any binary cache server yet.
  
  Nix will handle the build process without any user interaction (with a 
  few exceptions which you shouldn't have to worry). If any other user 
  has already built the package then the build process is not needed, 
  and the package is used as is.

  In order to build a BSC package go to the `bsc-nixpkgs` directory, and 
  run:

    xeon07$ nix-build -A bsc.dummy

  Notice the "bsc." prefix for BSC packages. The package will be built 
  and installed in the /nix directory, then a symlink is placed in the 
  result directory:

    xeon07$ find result/ -type f
    result/
    result/bin
    result/bin/dummy

  The way in which nix handles the packages and dependencies ensures 
  that the environment of the build process of any package is exactly 
  the same, so the generated output should be the same if the builds are 
  deterministic.
  
  You can check the reproducibility of the build by adding the "--check" 
  flag, which will rebuild the package and compare the checksum of every 
  file with the ones previously built:

    xeon07$ nix-build -A bsc.dummy --check
    ...
    xeon07$ echo $?
    0

  A return code of zero ensures the output is bit by bit identical to 
  the one installed. There are some packages that include 
  indeterministic information in the build process (such as the 
  timestamp of the current time) which will produce an error. Those 
  packages must be patched to ensure the output is deterministic.

  Notice that if you "cd" into the "result/" directory you will be at 
  /nix directory (as you have follow the symlink) where you don't have 
  write permission. Therefore if your program attempts to write to the 
  current directory it will fail. It is recommended to instead run your 
  program from the top directory:

    xeon07$ result/bin/dummy
    Hello world!

  Or you can install it in the environment:

    xeon07$ nix-env -i ./result

  And "cd" into any directory where you want to output some files and 
  just run it by the name:

    xeon07$ cd /tmp
    xeon07$ dummy
    Hello world!

  Finally, you can remove it from the environment if you don't need it:

    xeon07$ nix-env -e dummy

  If you want to know more details use "nix-build --help" to see the 
  manual.

2.3 The build process

  Each package is built following a programmable configuration 
  description in the nix language. Builds in nix are performed under 
  very strict conditions. No access to any file in the file system is 
  allowed, unless stated in the dependencies, which are in the /nix 
  store only.

  There is no network access in the build process and other restrictions 
  are enforced so that the build environment is reproducible. See more 
  details here:

    https://nixos.wiki/wiki/Nix#Sandboxing

  The top level "default.nix" file of the bsc-nixpkgs serves as a index 
  of all BSC packages. You can see the definition for each package, for 
  example the nbody app:

    nbody = callPackage ./bsc/apps/nbody/default.nix {
      stdenv = pkgs.gcc9Stdenv;
      mpi = intel-mpi;
      icc = icc;
      tampi = tampi;
      nanos6 = nanos6-git;
    };

  The compilation details are specified in the 
  "bsc/apps/nbody/default.nix" file.  You can configure the package by 
  changing the inputs, for example, what specific implementation of 
  nanos6 or MPI you want to use. To change the MPI implementation to the 
  official MPICH package use:

    nbody = callPackage ./bsc/apps/nbody/default.nix {
      stdenv = pkgs.gcc9Stdenv;
      mpi = pkgs.mpich; # Notice pkgs prefix for official packages
      icc = icc;
      tampi = tampi;
      nanos6 = nanos6-git;
    };

  Then you can rebuild the nbody package:

    xeon07$ nix-build -A bsc.nbody
    ...

  And verify that the binary is indeed linked to MPICH now:

    xeon07$ ldd result/bin/nbody_mpi.N2.2048.exe | grep mpi
        libmpi.so.12 => /nix/store/dwkkcv78a5bs8smflpx9ppp3klhz3i98-mpich-3.3.2/lib/libmpi.so.12 (0x00007f6be0f07000)

  If you modify a package which another package requires as a 
  dependency, nix will rebuild all required packages to propagate your 
  changes on demand.

  However, if you come back to the original configuration, the package 
  will still be in the /nix store (unless the garbage collector was 
  manually run and removed your old build), so you don't need to rebuild 
  it again.

  For example if nbody is configured back to use Intel MPI:

    nbody = callPackage ./bsc/apps/nbody/default.nix {
      stdenv = pkgs.gcc9Stdenv;
      mpi = intel-mpi;
      icc = icc;
      tampi = tampi;
      nanos6 = nanos6-git;
    };

  The build process now is not required:

    xeon07$ nix-build -A bsc.nbody
    /nix/store/rbq7wrjcmg6fzd6yhrlnkfvzcavdbdpc-nbody
    xeon07$ ldd result/bin/nbody_mpi.N2.2048.exe | grep mpi
        libmpifort.so.12 => /nix/store/jvsjvxj2a08340fpdrqbqix9z3mpp3bd-intel-mpi-2019.7.217/lib/libmpifort.so.12 (0x00007f3a00402000)
        libmpi.so.12 => /nix/store/jvsjvxj2a08340fpdrqbqix9z3mpp3bd-intel-mpi-2019.7.217/lib/libmpi.so.12 (0x00007f39fed34000)

  Take a look at the different package description files in the 
  bsc-nixpkgs repository if you want to understand more details. Also 
  the nix pills are a very good reference:

    https://nixos.org/nixos/nix-pills/

/* vim: set ts=2 sw=2 tw=72 fo=watqc expandtab spell autoindent: */