Differences

This shows you the differences between two versions of the page.

--- nasac:users_documentation [2023/08/16 11:41] – [rsync] delete in favor of 'Data transfer -> rsync' Luca Capello
+++ nasac:users_documentation [2025/06/11 12:27] (current) – external edit 127.0.0.1
@@ Line 30: / Line 30: @@
   * //Windows//: in the ''File Explorer'': \\ => right-click on the folder (either at the share root or at any sub-folder), then ''Properties -> Previous Versions''
   * //Mac or Linux//: \\ => enter the ''.snapshot'' **hidden** folder
+==== Replication ====
+**NB**, this originates from (( https://gitlab.unige.ch/storage/hpfs/-/issues/3 )).
+To assure the service continuity in case of problems in the primary UNIGE DC at UniDufour, a **nightly** replication is automatically done at the secondary UNIGE DC at the Campus Biotech.
+The main differences for the NAS infrastructure at the Campus Biotech are:
+  - 2 old nodes (4080 models //vs.// the 5300 at UniDufour)
+  - nor **4hourly** neither **weekly** snapshots, but **daily** ones (automatically taken **before** the nightly replication starts)
+  - no end-user access
@@ Line 49: / Line 61: @@
 ===== Data transfer =====
+==== Migrate data to UNIGE tape solution ====
+**NB**, this originates from (( https://gitlab.unige.ch/storage/tickets/-/issues/365#note_253404 )) (( <mid:storage/ZR0P278MB0698F735FA7FC1206FF1E507BD8DA@ZR0P278MB0698.CHEP278.PROD.OUTLOOK.COM> ))
+If you want to migrated NAS data (actually, **any data**...) to the UNIGE tape solution (cf. https://catalogue-si.unige.ch/tape, based on IBM Spectrum Archive), you can use the ''ug-nas-create-zip-archive-from-folder.sh'' from the [[https://gitlab.unige.ch/storage/nas/ug-nas/-/blob/main/misc/ug-nas-create-zip-archive-from-folder.sh|ug-nas]] project, which:
+  - calculate the size of the source folder
+  - calculate the MD5 hashes for each file in the source folder
+  - create a 2GB-by-default **split** (AKA multiple-file) ''.zip'' archive in the destination folder
+  - list the destination folder content
+  - delete the source folder and all its content
+<note important>
+Given that InfoZip ''unzip'' does not support (yet) **split archives**, to decompress the files generated by ''ug-nas-create-zip-archive-from-folder.sh'' you must use ''7-Zip'' (cf. https://7-zip.org, the old ''p7zip'' Linux port works as well) on the last file, thus ''7za x ${ARCHIVE}.zip''!
+</note>
+<code>
+$ ug-nas-create-zip-archive-from-folder.sh -h
+UG-NAS: Create a split ZIP archive from a NAS folder
+Usage: ug-nas-create-zip-archive-from-folder.sh [-h] [-s <splitsize_gb>] <source> <target>
+Options:
+     -h : display this help and exit
+     -s : split at GB (defaults to 2)
+$ ug-nas-create-zip-archive-from-folder.sh \
+    "/path/to/source/folder" \
+    "/path/to/destination/folder" \
+>&1 | \
+ tee "/path/to/log/folder/$(date +%Y%d%m)_ug-nas-create-zip-archive-from-folder_${SOURCE_FOLDER}.log"
+Thu 16 Nov 2023 11:48:49 AM CET
+=====
+I: calculating source folder size...
+.9G
+=====
+Thu 16 Nov 2023 11:48:49 AM CET
+=====
+I: generating MD5 checksum for each file in source folder...
+[...]
+real    1m3.094s
+user    0m7.158s
+sys     0m1.736s
+=====
+Thu 16 Nov 2023 11:49:52 AM CET
+=====
+I: creating the split ZIP archive...
+splitsize = 2147483648
+[...]
+total bytes=4098480958, compressed=3899876027 -> 5% savings
+real    2m48.380s
+user    2m30.320s
+sys     0m5.719s
+=====
+Thu 16 Nov 2023 11:52:40 AM CET
+=====
+I: listing archive folder '/path/to/destination/folder' content:
+total 3768321
+[...]
+=====
+Thu 16 Nov 2023 11:52:40 AM CET
+=====
+I: deleting the source folder...
+real    0m0.073s
+user    0m0.004s
+sys     0m0.029s
+=====
+Thu 16 Nov 2023 11:52:40 AM CET
+=====
+I: all done!
+$
+</code>
 ==== Access from outside the UNIGE network ====
@@ Line 68: / Line 154: @@
 ==== rsync ====
-**NB**, the following instructions come from (( https://gitlab.unige.ch/prods/ies/recherche/hpc/issues/900#note_32016 ))
+**NB**, the following instructions come from (( https://gitlab.unige.ch/prods/ies/recherche/hpc/issues/900#note_32016 )) (( https://gitlab.unige.ch/storage/tickets/-/issues/40 ))
 If you need to synchronize data to another folder, you can let `rsync` **3.1.0+** saves in the log file the MD5 checksum of any transferred file (cf. https://stackoverflow.com/questions/29624524/how-can-i-print-log-the-checksum-calculated-by-rsync#45053057): <code console>
@@ Line 95: / Line 181: @@
 ===== Utilisation d'un partage SMB =====
-**NB**, this partly originate from (( https://gitlab.unige.ch/storage/tickets/-/issues/577 )).
+**NB**, this partly originates from (( https://gitlab.unige.ch/storage/tickets/-/issues/577 )) (( <mid:storage/5f3de4ce-51d5-c95f-efe4-fab7aa3c5ef2@unige.ch> )) (( <mid:storage/GV0P278MB0082B1F0968DAC3684909CE8C91EA@GV0P278MB0082.CHEP278.PROD.OUTLOOK.COM> )).
-Lorsqu'un partage a été crée et que votre CI vous a donné les droits d'accès, suivez la procédure ci-dessous pour vous connecter à votre partage. A noter que pour vous connecter à un partage depuis l'extérieur de l'université, vous devez configurer et lancer votre VPN. Veuillez vous référer à la documentation du VPN de l'unige pour ce faire.
+There are 3 different actors involved in the life of an SMB share:
+  - the **DiSTIC RISE storage team** \\ => the infrastructure administrators and share configurators (allocated space, ACL administrative access and pass-through access)
+  - the share **owner** (AKA ''contact_owner'') \\ => the one responsible for the share content (thus deciding who has access to them), also providing the billing information
+  - the share **administrators** (AKA ''contact_admin'') \\ => the share content managers
+Please follow the links on the [[https://catalogue-si.unige.ch/stockage-recherche|catalogue des services informatiques, section "Obtenir une prestation"]] to ask for a new share or a modification to an existing one, as well as to report a problem.
+<note important>
+Please check the [[#how_to_manage_smb_acls_aka_permissions|how to manage the ACLs (AKA permissions)]] section to understand how the ACLs are configured and to be managed!
+</note>
+Lorsqu'un partage a été crée et que votre CI vous a donné les droits d'accès (cf. [[#how_to_manage_smb_acls_aka_permissions|how to manage the ACLs (AKA permissions)]]), suivez la procédure ci-dessous pour vous connecter à votre partage. A noter que pour vous connecter à un partage depuis l'extérieur de l'université, vous devez configurer et lancer votre VPN. Veuillez vous référer à la documentation du VPN de l'unige pour ce faire.
 <note important>
@@ Line 164: / Line 261: @@
 === Command line ===
-  - via plain old ''mount'' command, which however requires **superuser** privileges: <code console>
+**NB**, this originates from (( https://gitlab.unige.ch/storage/tickets/-/issues/919 )).
+  - via plain old ''mount'' command, which however requires **superuser** privileges and the **cifs** kernel module: <code console>
 root@harlock:~# findmnt /mnt
 root@harlock:~# mount \
@@ Line 177: / Line 276: @@
 root@harlock:~#
 </code>
-  - via GVfs/Gio (//i.e.// the same "helper" framework used by most of the graphical tools including the default Ubuntu desktop), which is an unprivileged action: <code console>
+  - via GVfs/Gio (//i.e.// the same "helper" framework used by most of the graphical tools including the default Ubuntu desktop), which is an unprivileged action with its own non-kernel SMB implementation: <code console>
 capello@harlock:~$ gio mount smb://nasac-faculty.isis.unige.ch/ADM_HOME/DISTIC/capello
 Password required for share adm_home on nasac-faculty.isis.unige.ch
@@ Line 268: / Line 367: @@
 ==== How to manage SMB ACLs (AKA permissions) ====
-**NB**, part of the following instructions come from (( https://gitlab.unige.ch/storage/tickets/-/issues/432 )) (( https://support-si.unige.ch/openentry.html?tid=INC000000312347 ))
+**NB**, part of the following instructions come from (( https://gitlab.unige.ch/storage/tickets/-/issues/432 )) (( https://support-si.unige.ch/openentry.html?tid=INC000000312347 )) (( https://support-si.unige.ch/openentry.html?tid=INC000000317940 )).
-You must set the access rights directly from Windows :
+You need at least **2 AD groups** for an SMB share to be created:
+  - an ''admin'' one to manage the ACLs (AKA permissions) \\ => full control at the first level (thus ''\\${SERVER}\${SHARE}\''), with inheritance enabled
+  - a ''pass-through'' one to be able to mount/connect the share \\ => list the first level content only (thus no inheritance)
-- if you are a **Correspondant Informatique**, please check the internal documentation "Affectation des droits" paragraph in the "Gestion des postes de travail avec l'Active Directory (AD)" guide available on [[https://plone.unige.ch/distic/acteurs-du-si/ci-et-webmasters/formations-ci/|Plone]].
+These 2 groups will then be used to set the ACLs of the share itself (thus the mount point), ACLs that must be modified via the administrative interface only.
-- if you are a **end-user**, please check the external documentation at https://wiki.samba.org/index.php/Setting_up_a_Share_Using_Windows_ACLs#Setting_ACLs_on_a_Folder or https://docs.netapp.com/us-en/ontap/smb-admin/configure-ntfs-windows-security-tab-task.html .
+Once the share has been created, you must set the ACLs at least for the 1st-level content, directly from Windows:
+  * if you are a **Correspondant Informatique**, please check the internal documentation "Affectation des droits" paragraph in the "Gestion des postes de travail avec l'Active Directory (AD)" guide available on [[https://plone.unige.ch/distic/acteurs-du-si/ci-et-webmasters/formations-ci/|Plone]].
+  * if you are a **end-user**, please check the external documentation at https://wiki.samba.org/index.php/Setting_up_a_Share_Using_Windows_ACLs#Setting_ACLs_on_a_Folder or https://docs.netapp.com/us-en/ontap/smb-admin/configure-ntfs-windows-security-tab-task.html .
 <note important>
@@ Line 282: / Line 385: @@
 ===== How to manage an NFS share =====
-**NB** , the following instructions come from <https://gitlab.unige.ch/prods/ies/recherche/hpc/issues/716>
+**NB** , the following instructions come from (( https://gitlab.unige.ch/prods/ies/recherche/hpc/issues/716 )) (( <mid:storage/fef31707-9e80-4219-9871-06f17d35f704@unige.ch> )).
+<note important>
+To know which client/IP can connect to which NFS share, you can use the command ''/sbin/showmount -e "${EVS}" | grep -e "${SHARE}"''.  Please be aware that the output does not show the NFS options (the most common ones are **ro**, **rw** and **no_root_squash**), which are independent from the NFSv4 ACLs below!
+</note>
 Nowadays most of the GNU/Linux distribution defaults to NFSv4, which is a big step forward and differs quite a lot from NFSv3 (cf. http://www.citi.umich.edu/projects/nfsv4/ ). One of the main advantages of NFSv4 is the ACLs support into the protocol itself (cf. http://wiki.linux-nfs.org/wiki/index.php/ACLs ), ACLs that resemble very much the Windows ones.