Sharing files among CÉCI users

As scientists work in teams, it is often necessary to share files (be it data or code) among several CÉCI users. Access to files is governed by the UNIX file permission system so you will need to get familiar with it to share files. A general introduction can be found on Wikipedia. Actually, sharing files with every one else is easy. It is when you need to give some people access to you files and not to others that things become a bit more convoluted.

Sharing with every one is easy: you simply need to set the correct UNIX permissions on your home directory and the files/directories you want to share. Sharing with only certain people through hiding is also easy (access by unauthorized parties is possible but improbable). Sharing with only certain people through real enforcement requires our intervention (access by unauthorized parties is then impossible).

Sharing with everyone

To share with everyone you need to make sure that your home directory has read and execute permission for others (e.g. rwxr-xr-x or 755 in octal form). Then, everyone can list the content of your home directory and access any file in there that has read access for others. The reasoning applies recursively to sub-directories.

To find which permissions apply to your home directory, run the ls command (remember that ~ is the home directory of the current user, the -l options is used to show permissions, the -d option is to list the directory itself rather than its content, and the -L option is necessary because on most CÉCI systems, the home directory is a symbolic link):

dfr@hmem00:~ $ ls -dlL ~
drwxr-xr-x 56 dfr dfr 8192 Aug 12 16:20 /home/ucl/pan/dfr

Here, the home directory of the dfr user on hmem is readable and accessible by everyone.

Should the above command return something like the following:

dfr@hmem00:~ $ ls -dlL ~
drwxr-x--- 56 dfr dfr 8192 Aug 12 16:20 /home/ucl/pan/dfr

another user (for instance here dcoligno) would not be able to see the contents:

dcoligno@hmem00:~ $ ls -l ~dfr
ls: cannot open directory /home/ucl/pan/dfr: Permission denied

Then the chmod command must be used to give access to others. The manual of the chmod command can be found by issuing the man chmod command.

dfr@hmem00:~ $ chmod o=rx ~
dfr@hmem00:~ $ ls -dlL ~
drwxr-xr-x 56 dfr dfr 8192 Aug 12 16:20 /home/ucl/pan/dfr

The above chmod command reads, in plain English, Change the permissions of my home directory so that others have (=) read and execute access to it., so that other users will be able to see its contents. Indeed, user dcoligno now can list the content of the home directory of user dfr.

dcoligno@hmem00:~ $ ls -l ~dfr
drwxr-xr-x dfr dfr Aug 12 16:20 shared_dir
drwxr-x--- dfr dfr Aug 12 16:20 private_dir
-rw-r--r-- dfr dfr Aug 12 16:20 fileA
-rw-r----- dfr dfr Aug 12 16:20 fileB

What we see is that other (all other) users have read access to shared_dir and fileA, but not to fileB for instance:

dcoligno@hmem00:~ $ cat ~dfr/fileA
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
dcoligno@hmem00:~ $ cat ~dfr/fileB
cat: /home/pan/dfr/fileB: Permission denied

Restricting access through hiding

Sometimes you do not want to share your files with all other users, though.

To share with only a restricted list of users, one option is to remove the read permission on your home directory. Then, other users won’t be able to list the content of your home directory, but if they know exactly the name of the file they need, and that file has read permission for others, they will be able to read it. If the name is not trivially guessable, that file is actually hidden but accessible to those who know it exists.

To reach this state, use the chmod command this way:

dfr@hmem00:~ $ chmod o=x ~
dfr@hmem00:~ $ ls -dlL ~
drwxr-x--x dfr dfr Aug 12 16:20 /home/ucl/pan/dfr

We see that the permissions are now drwxr-x–x rather than drwxr-xr-x, and other users can’t see what’s inside the directory:

dcoligno@hmem00:~ $ ls -l ~dfr
ls: cannot open directory /home/ucl/pan/dfr: Permission denied

Nevertheless, if dcoligno knows that dfr has a file named fileA, in his home directory, he can read its contents.

dcoligno@hmem00:~ $ cat ~dfr/fileA
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

Using this settings, dfr can decide with whom he shares a file. Setting the file or directory name to an unguessable character string, and giving that string to dcoligno, is roughly equivalent to giving dcoligno a password to access the file. Note though that anyone who knows the ‘password’ (i.e. the secret file/directory name) will be able to access it.

dfr@hmem00:~ $ ls -l ~
drwxr-xr-x dfr dfr Aug 12 16:20 shared_dir_&aqw1AQW
drwxr-x--- dfr dfr Aug 12 16:20 private_dir
-rw-r--r-- dfr dfr Aug 12 16:20 fileA
-rw-r----- dfr dfr Aug 12 16:20 fileB

Everyone who knows about shared_dir_&aqw1AQW will be able to access it but it won’t be discoverable by others.

Enforcing the restricted access (one to many)

The above solution is sufficient in most cases. But if you want to guarantee that only a specific set of users have access to your file even if they know its name, the CÉCI system administrators need to be involved.

To share with only a limited restricted list of users in a ‘one to many’ fashion (for instance you are installing software for all people in your research group, or you have all your group’s data in your home directory), and enforce that others cannot access the data, one option is to add each authorized parties to your group. Then, the above reasoning applies to the members of your group provided you replace every occurrence of others with group in the above paragraphs. Users not in your group will not be able to access your data at all (as long as your home directory has no execute permission for others of course.)

Assuming the CÉCI system administrators have included dcoligno in dfr‘s group:

dcoligno@hmem00:~$ id
uid=3000014(dcoligno) gid=3000014(dcoligno) groups=3000014(dcoligno),3000003(dfr)

then dcoligno has access to any file/directory that is group-readable. Beware, that means most often all of dfr‘s files under the default settings!

Enforcing the restricted access (many to many)

To share with a larger group, allow everyone in the group to share as well, and still enforce that others cannot access the data, the most convenient option is to create a UNIX group specifically for that group. Then, the reasoning in the previous paragraph applies provided you first change the group to which the data belong.

Assuming the CÉCI system administrators have created a group ourgroup with dfr and dcoligno,

dfr@hmem00:~ $ getent group ourgroup
ourgroup:*:4999998:dfr,dcoligno

then dfr can share the shared_dir directory with chgrp

dfr@hmem00:~ $ chgrp ourgroup shared_dir
dfr@hmem00:~ $ ls -l ~
drwxr-x--- dfr ourgroup Aug 12 16:20 shared_dir
drwxr-x--- dfr dfr      Aug 12 16:20 private_dir
-rw-r--r-- dfr dfr      Aug 12 16:20 fileA
-rw-r----- dfr dfr      Aug 12 16:20 fileB

Note that the permissions of shared_dir allow here read access to shared_dir by a member of the ourgroup group, but no write access, and also no read access to users not in the ourgroup group. Should dfr want other users in the group to be able to write files in his directory, that can be done easily:

dfr@hmem00:~ $ chmod g+w shared_dir
dfr@hmem00:~ $ ls -l ~
drwxrwx--- dfr ourgroup Aug 12 16:20 shared_dir
drwxr-x--- dfr dfr      Aug 12 16:20 private_dir
-rw-r--r-- dfr dfr      Aug 12 16:20 fileA
-rw-r----- dfr dfr      Aug 12 16:20 fileB

The above chmod command reads, in plain English, Add (+) write permission to members of the group owning shared_dir

Concluding remarks

  • UNIX groups corresponding to TIER1 project are also created on the CÉCI clusters and can be used there.
  • on the clusters, the umask – the configuration setting that dictates the permissions set to newly created files – is set to 0002 or 0022, meaning that by default, your files are created with permission 664 or 644 respectively.