summaryrefslogtreecommitdiffstats
path: root/src/utils
Commit message (Collapse)AuthorAgeFilesLines
* utils: drop ogCopyEfiBootLoader scriptAlejandro Sirgo Rica2024-03-262-10/+52
| | | | | | | | Implement a Python equivalent of ogCopyEfiBootLoader as the function copy_efi_bootloader. This function copies the contents of the folder of the EFI loader in the ESP into a ogBoot folder at the root of the partition target of an image creation. copy_efi_bootloader is a Windows only functionality.
* utils: add enforce_gpt argument to get_efi_partition()Alejandro Sirgo Rica2024-03-262-5/+8
| | | | | | | The Windows bootloader only supports a UEFI boot from a GPT partition. Set enforce_gpt to True in every codepath related to Windows. When enforce_gpt is set to True get_efi_partition() raises an exception when an MBR partition scheme is detected.
* revert 7f18485: improve uefi detection mechanismAlejandro Sirgo Rica2024-03-262-13/+3
| | | | | | | | | | | Make is_uefi_supported() only check for /sys/firmware/efi as get_efi_partition() will detect a missing ESP or an invalid partition scheme. Stop using get_efi_partition() inside is_uefi_supported() as the former is eventually called in every UEFI related code. UEFI supports both MBR and GPT as partition schemes and this is a required change to handle the particular case of Windows not being able to boot UEFI from a MBR partition scheme.
* src: make exception messages more contextual and explicitAlejandro Sirgo Rica2024-03-213-5/+5
| | | | | | | Provide more information in exception messages as those are the source of the logging messages. Add information about paths, files or configuration related to the operation associated to the exception.
* src: change generic exception types to be more explicitAlejandro Sirgo Rica2024-03-211-1/+1
| | | | | | | | Replace exception types to be more explicit about the nature of the error. Improve the exception raising semantics by using the 'from' keyword, this wraps an older exception into a new one so it is still considered the same object.
* utils: fix string not being defined as f-strings when they shouldAlejandro Sirgo Rica2024-03-211-2/+2
| | | | | Add the string prefix f'' to string instances formated as f-string but not properly constructed to expand the inline variables.
* src: centralize error logging into send_internal_server_errorAlejandro Sirgo Rica2024-03-215-41/+28
| | | | | | | | | | | | | | | | | | | | Use only the exception messages as the main resource for error messages. The previous error code had string duplication in the form of: logging.error('msg here') raise Exception('msg here') That approach also has the downside of having log duplication as it had the local logging.err() and a global logging.exception() inside send_internal_server_error capturing the exception message. The actual code only requires raising an exception with a proper error message. Improve exception messages to give more error context. Log every AssertionError as a backtrace. Use the 'raise Exception from e' syntax to modify the a previously raised exception 'e' into an exception with aditional context or different type. This also prevents the message that warns about newer exceptions being launch after an initial exception.
* utils: implement BIOS boot for windowsAlejandro Sirgo Rica2024-03-211-4/+10
| | | | | | | | | | | | | | | | | | Create ogboot.me and ogboot.secondboot as empty files and ogboot.firstboot with the value "iniciado" in the root of the BIOS Windows system partition. The files must contain data for GRUB to be able to write content, therefore these are created containing 3072 null bytes. The Windows boot process is handled by the "pxe" profile. There the files ogboot.me, ogboot.firstboot and ogboot.secondboot are used as a state machine to chose between booting Windows and ogLive. The first Windows boot happens if ogboot.me and ogboot.firstboot are identical, then "iniciado" is written in ogboot.firstboot. We skip this stage as we create ogboot.firstboot with 'iniciado'. The second Windows boot occurs if ogboot.me and ogboot.secondboot are boot identical, then "iniciado" is written in ogboot.secondboot. After the Windows boot ogLive is booted.
* utils: add disk index checks in get_partition_deviceAlejandro Sirgo Rica2024-03-211-1/+5
| | | | | Control the possibility of a bad disk index. Handle the checks in a similar fashion to the get_efi_partition function.
* utils: implement BIOS boot for LinuxAlejandro Sirgo Rica2024-03-212-1/+70
| | | | | | | | | Create a bios.py file to hold all the BIOS specific functions. Implement the _boot_bios_linux in Python. The new boot process tries to find the vmlinuz and initrd binaries at the desired partition. Then it tries to load them with kexec with the proper Grub boot params. One step closer to the removal of the boot legacy script.
* utils: handle exceptions caused by the hivex packageAlejandro Sirgo Rica2024-03-081-3/+6
| | | | | | The mage creation process was being interrupted by an error trying to read the Windows registry by the Hivex library. Now the exceptions are handled and an error is reported.
* utils: implement linux distro id detectionAlejandro Sirgo Rica2024-03-082-1/+27
| | | | | | | The OS probe logic must be able to check a distro programmatically, add get_linux_distro_id to return an id whitout versioning. Ensure the availability of 'ubuntu' when we need to ensure certain features are only used with a supported system.
* utils: move all boot from OS functionality into boot.pyAlejandro Sirgo Rica2024-03-081-6/+26
| | | | | | | | This change is a preparative for reimplementing the BIOS boot in order to deprecate the legacy script. All the codepaths to boot systems located at a partition are now called from the boot_os_at function enabling an easier structure for the incoming code.
* utils: improve uefi detection mechanismAlejandro Sirgo Rica2024-03-082-3/+13
| | | | | | | Checking the existence /sys/firmware/efi as it might appear sometimes in BIOS installs if the BIOS configuration is not proper. Checking for the EFI partition is the safest method to veryfy the install type.
* utils: handle missing file in getlinuxversion functionAlejandro Sirgo Rica2024-03-081-11/+15
| | | | | | The function getlinuxversion receives a path to the os-release file. The case of not being able to open it was not handled and thus causing an unwanted exception.
* utils: improve logging in the get_efi_partition functionAlejandro Sirgo Rica2024-03-081-13/+16
| | | | | Log each partition that gets checked and make the exception messages more informative.
* uefi: update EFIBOOTMGR_BIN to use /opt/opengnsys/sbin/efibootmgrv1.3.2-6OpenGnSys Support Team2024-03-051-1/+1
| | | | debian package with json support provides the binary through this path, update it.
* uefi: define EFIBOOTMGR_BIN to ease changing the efibootmgr binaryAlejandro Sirgo Rica2024-03-041-9/+11
| | | | | | The json functionality proposed upstream might be merged one day in efibootmgr so deploying a fork would not be needed anymore. This change aims to ease the migration once that day comes.
* live: drop IniciarSesion script when uefi booting into linuxAlejandro Sirgo Rica2024-03-041-6/+33
| | | | | | Replace IniciarSesion script in favor of native Python code when booting a UEFI system into Linux. This completes the implementation of booting into an OS on a UEFI compliant system.
* utils: add boot.pyJose M. Guisado2024-03-041-0/+63
| | | | | | | | | | Add utility module related to the process of booting a system from a client's partition. The main utility function to boot a clients system is boot_os_at(), from which firmware (UEFI or BIOS) and os-family specific private functions are invoked. This initial commit adds UEFI windows boot function.
* utils: add uefi.pyJose M. Guisado2024-03-041-0/+97
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add UEFI related utilities inside a new utility module: uefi.py _check_efibootmgr_json ====================== Check if the system efibootmgr executable supports JSON output. This is a private function used only by other functions from uefi.py. is_uefi_supported ================= Check if the system supports UEFI firmware. run_efibootmgr_json =================== Runs efibootmgr with json output support. Return the JSON output as a Python dict. efibootmgr_create_bootentry =========================== Create nvram boot entry. This bootentry is usually later set to boot next just once via "BootNext" nvram variable. efibootmgr_delete_bootentry =========================== Delete a nvram boot entry. Used to avoid duplicates when booting the same disk and partition from a given client. efibootmgr_bootnext =================== Set nvram "BootNext" variable to a given boot entry so after client reboot, PXE is not executed and the given boot entry takes precedence. Add dependency with efibootmgr version >= 18, and efibootmgr JSON output which is currently out of tree from util-linux repo.
* probe: add basic os family detectionJose M. Guisado2024-03-041-0/+16
| | | | | | | | Add a basic OS family enumeration: OSFamily. Add utility function that probes for an installed Linux or Windows system, returns the corresponding enum value, OSFamily.UNKNOWN otherwise.
* disk: add get_efi_partition functionJose M. Guisado2024-03-041-0/+24
| | | | | | | | | | | | | Add utility function inside disk.py to find, if any, the first ESP partition of a given disk. The disk is provided as an integer (starting at 1 following OpenGnsys scripts usual values), meaning the (n-1)th disk from the disk array returned from get_disks(). In the future a better mechanism should be put in place to fetch probed disks from a running client. This change is part of the upcoming drop of "IniciarSesion" script in favor of a Python native approach. Specifically regarding UEFI systems.
* cache: improve loggingOpenGnSys Support Team2024-02-221-1/+1
| | | | | use info instead of debug to make it easier to debug problems when creating the cache.
* fs: add swap support and improve mkfs loggingOpenGnSys Support Team2024-02-221-3/+15
|
* utils: use returncode from subprocess in ogChangeRepo()OpenGnSys Support Team2024-02-211-4/+3
| | | | | | | ... the exception shows the samba password in the logs specify the error which tells us what has happened according to man mount(8) Return Codes.
* fs: improve value parsing from command outputAlejandro Sirgo Rica2024-02-191-7/+29
| | | | | | value extraction did not have error checking and was handled in a one-liner. The actual implementation expands the parsing logic and moves it into a function.
* utils: remove repository change in restoreImageCustomOpenGnSys Support Team2024-02-151-4/+0
| | | | this is broken, it uses default uses and password, remove it.
* fs: return unknown if blkid failsOpenGnSys Support Team2024-02-151-1/+7
| | | | instead of rising an exception
* fs: logging ext4 resize errorOpenGnSys Support Team2024-02-151-2/+4
| | | | log error in case resize2fs fails.
* fs: check if ntfsresize actually succeded to shrink filesystemOpenGnSys Support Team2024-02-151-1/+4
| | | | | According to ntfsresize.c, this retuns 0 in case nothing needs to be done. It should be safe to check for non-zero error and bail out in that case.
* fs: disentagle dry-run ntfsresize loop to probe for best shrink sizeOpenGnSys Support Team2024-02-151-8/+16
| | | | | | | | Revisit 5056b8f0d5ab ("fs: validate ntfsresize dry-run output") that has introduced a possible infinity loop. Disentangle this loop while at it: iterate until best smallest size is found by probing.
* src: ogChangeRepo returns zero on success and -1 on errorOpenGnSys Support Team2024-02-151-7/+10
| | | | | | | | | | | | | | | | | | | | do not return the returncode, instead return an integer. do not use except CalledProcessError as e: it causes a another exception while handling exception. Remount the original image repository. it should be possible to simplify this further by: - stacking mounts, no need to umount initial repo and mount it again when switching to the new repo, because remount back initial repo might fail (!) - use check=False and simply check for x.returncode
* src: improve error check in image_create and image_restoreAlejandro Sirgo Rica2024-02-143-9/+23
| | | | | | | | | cover more error cases where exceptions need to be raised. check return code in the invoked subprocess. restoreImageCustom has been intentionally left behind, it is unclear what this custom script returns on success and error.
* fs: validate ntfsresize dry-run outputv1.3.2-5OpenGnSys Support Team2024-01-111-2/+3
| | | | | | | | | | | | | | | | validate 'Needed relocations: ' is in place before stepping on the split chunks (2024-01-11 10:28:16) ogClient: [ERROR] - Exception when running "image create" subprocess Traceback (most recent call last): File "/opt/opengnsys/ogClient/src/live/ogOperations.py", line 454, in image_create ogReduceFs(disk, partition) File "/opt/opengnsys/ogClient/src/utils/fs.py", line 105, in ogReduceFs _reduce_ntfsresize(partdev) File "/opt/opengnsys/ogClient/src/utils/fs.py", line 235, in _reduce_ntfsresize extra_size = int(out_resize_dryrun.split('Needed relocations : ')[1].split(' ')[0])*1.1+1024 IndexError: list index out of range if not present, no need to adjust size
* utils: use f-string in raise ValueError in run_lzop_partcloneinfo()OpenGnSys Support Team2023-12-151-1/+1
| | | | | | Otherwise it shows: ValueError: Unable to process image {image_path}
* live: report permissions and last update when creating imagev1.3.2-3OpenGnSys Support Team2023-12-121-0/+2
| | | | add .permissions and .lastupdate to json to report to ogserver.
* live: report image size when creating imageOpenGnSys Support Team2023-12-121-0/+1
| | | | add .size json field to report the real size of the image file.
* utils: sw_inventory: report unknown OS for software inventoryv1.3.2-2OpenGnSys Support Team2023-11-171-1/+2
| | | | | | Users can create an image of a filesystem that contains no OS, therefore, instead of rising an exception when no OS is detected, deliver a "unknown" OS and an empty list of software.
* live: hw_inventory: fix empty pci storage size bugJose M. Guisado2023-10-181-1/+1
| | | | | | | | | | | When a client's hardware presents an empty pci storage child there is an invalid call to _bytes_to_human a string is supplied as a default value if the storage child does not present a 'size' attribute. Fix this by checking if 'size' is present in the JSON output from lshw. If size is present then map the bytes to a human readable string using _bytes_to_human, if no size is present then use 'Empty slot' to indicate that the memory bank is not being used.
* live: hw_inventory: fix typoJose M. Guisado2023-10-181-1/+1
| | | | | | Add missing underscore to _bytes_to_human call. Fixes: 39c13287c53bd8 ("live: hw_inventory: fix empty memory bank bug")
* live: hw_inventory: fix empty memory bank bugJose M. Guisado2023-10-051-6/+9
| | | | | | | | | | | | When a client's hardware presents an empty memory bank and invalid call to _bytes_to_human is performed because None is passed as a parameter. size = _bytes_to_human(obj.get('size', None)) Fix this by checking if 'size' is present in the JSON output from lshw. If size is present then map the bytes to a human readable string using _bytes_to_human, if no size is present then use 'Empty slot' to indicate that the memory bank is not being used.
* src: improve logging messagesJose M. Guisado2023-08-011-2/+5
| | | | | | | | | | | | | | | | | | | | | | | | | Some users have mistakenly reported tiptorrent problems when the process takes a long time. Specifically by rebooting or powering off the client in the middle of the md5sum computation stage, just after the tiptorrent transfer. Same problem occurs when image creation command takes a long period of time. In order to help the user understand the different stages of commands such as image creation or image restore using tiptorrent, the following changes have been made to the current logging solution: - Add log messages to warn users not to reboot or shut down the client during a tiptorrent transfer, and also during the md5sum computation stage. - Add a log message telling the user that the image creation processes have started. - Use logging.exception inside "except:" blocks to print a traceback with the log messsage. (https://docs.python.org/3/library/logging.html#logging.exception)
* hw_inventory: use dict.getJose M. Guisado2023-07-211-23/+27
| | | | | | | | | The first stage of parsing the "lshw -json" command output is to load the json string into a Python dictionary. lshw output is large and varies from machine to machine, so it's not safe to assume that different keys will be present in the dictionary. Use dict.get() instead of dict[key] to avoid KeyError exceptions.
* fs: fix subprocess input inside _extend_resize2fsJose M. Guisado2023-06-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | The subprocess module expects bytes-like object for "input" parameter by default. Passing a string object result in the following error: (2023-06-13 14:44:43) ogClient: [ERROR] - Exception when running "image create" subprocess (2023-06-13 14:44:43) ogClient: [ERROR] - Unexpected error Traceback (most recent call last): File "/opt/opengnsys/ogClient/src/live/ogOperations.py", line 465, in image_create ogExtendFs(disk, partition) File "/opt/opengnsys/ogClient/src/utils/fs.py", line 124, in ogExtendFs _extend_ntfsresize(partdev) File "/opt/opengnsys/ogClient/src/utils/fs.py", line 250, in _extend_ntfsresize proc = subprocess.run(cmd, input='y') File "/usr/lib/python3.8/subprocess.py", line 495, in run stdout, stderr = process.communicate(input, timeout=timeout) File "/usr/lib/python3.8/subprocess.py", line 1013, in communicate self._stdin_write(input) File "/usr/lib/python3.8/subprocess.py", line 962, in _stdin_write self.stdin.write(input) TypeError: a bytes-like object is required, not 'str' Fixes: dd999bfe34e7 ("utils: rewrite ogReduceFs")
* fs: fix bug when ntfsresize reports nothing to doJose M. Guisado2023-06-131-0/+3
| | | | | | | | There is a corner case in which a target NTFS filesystem is already shrunken. When this happens ntfsresize text output parsing breaks. Check when ntfsresize reports nothing to do, warn the user about this and stop the dry-run ntfsresize loop.
* fs: fix typo inside _extend_resize2fsJose M. Guisado2023-06-131-4/+4
| | | | | | | | | | _extend_ntfsresize contains an incorrect variable name inside subprocess.run referring the resize command value. Simplify this variable name inside each specific _extend_* function: s/cmd_resize2fs/cmd s/cmd_ntfsresize/cmd
* sw_inventory: ignore invalid windows programsJose M. Guisado2023-05-091-5/+10
| | | | | | | | | | | | | | | | | | | | | | | Don't raise exception if any windows program is missing DisplayName node in the windows registry. This attribute/node should contain the program's name. This name is used as the package's name in the software set (software inventory). This patch should be considered a hotfix, python-hivex does not report any helpful message about this error. (2023-05-09 14:43:13) ogClient: [ERROR] - Unexpected error Traceback (most recent call last): [...] RuntimeError: Success Before this patch, image creation *might* fail because it cannot create the software inventory associated with the image due to the previously described error. The software inventory is part of the response payload of the image creation command (see src/ogRest:image_create). Fixes: 04bb35bd86b5 (live: rewrite software inventory)
* utils: add umount_all functionJose M. Guisado2023-05-091-0/+9
| | | | | | | | Add utility function to unmount any mountpoint present in the /mnt folder. This function is a simplified version of the legacy bash function ogUnmountAll used in several operations.
* utils: rewrite ogExtendFsJose M. Guisado2023-05-091-7/+28
| | | | | | | | | | | | | | | | | | | | Drop subprocess call to bash function ogExtendFs. Use a native python solution with subprocess calls to the required underlying tools. Use get_filesystem_type to get the present filesystem from a partition and call the corresponding filesystem grow function. Filesystem specific functions are declared "_extend_{filesystem}" and should not be imported elsewhere. Each filesystem specific function wraps a subprocess call to the required underlying program: - NTFS filesystems: "ntfsresize -f [partition]" - ext4 filesystems: "resize2fs -f [partition]" Set NTFS related subprocess stdin to 'y' because human input cannot be unset with other ntfsresize parameters.