summaryrefslogtreecommitdiffstats
path: root/src/utils
Commit message (Collapse)AuthorAgeFilesLines
* grub: move get_grub_boot_params() into grub.pyAlejandro Sirgo Rica2024-10-223-16/+19
| | | | | Move get_grub_boot_params() into the file related to all the grub configuration.
* grub: replace legacy grub install scriptsAlejandro Sirgo Rica2024-10-224-31/+401
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Translate old legacy grub scripts into grub.py Implement ogGrubInstallMbr as install_main_grub() and ogGrubInstallPartition as install_linux_grub(). Add grub configuration file generator through the classes GrubConfig and MenuEntry. Ensure EFI tree structure compatibility with legacy code. The structure of the created folders in the ESP is non-standard, efi binaries are usually located in the folder below the EFI/ directory. Structure used by ogClient: EFI/ ├── grub/ │ └── Boot/ │ ├── BOOTX64.CSV │ ├── grub.cfg │ ├── mmx64.efi │ ├── shimx64.efi │ ├── BOOTX64.EFI │ ├── grubx64.efi │ └── ogloader.efi ... The function _mangle_efi_folder handles the folder structure after grub-install to comply with the location expected by ogLive. install_linux_grub() installs a grub local to each Linux install to enable chainloading, each grub is located in EFI/Part-xx-yy/ in UEFI. The local linux BIOS grub in legacy scripts is unreliable, grub-install reports a failure during the install process. install_main_grub() installs a global grub in EFI/grub/ to show a grub menu when the pxe boot fails. The global grub contains entries to every installed os. No global grub is installed for BIOS systems, a Boot partition would be required to store the grub configuration.
* disk: use cxt.label.name to dtect disk type in get_efi_partitionAlejandro Sirgo Rica2024-10-211-1/+1
| | | | | Use cxt.label.name instead of cxt.label to identify if the disk is GPT or MBR. This way is more used in other parts of the codebase.
* bios: fix use of undefined initrd_dir variable in get_vmlinuz_pathAlejandro Sirgo Rica2024-10-211-1/+1
| | | | | Use the intended linuz_dir instead of initrd_dir in the function get_vmlinuz_path.
* uefi: fix error message when no EFI loaderv1.3.2-21Alejandro Sirgo Rica2024-10-111-1/+1
| | | | | Fix log error message when _find_efi_loader does not find any EFI loader in the ESP.
* bios.py: fix typo in GRUB_CMDLINE_LINUX_DEFAULT stringAlejandro Sirgo Rica2024-10-091-1/+1
| | | | | Check against GRUB_CMDLINE_LINUX_DEFAULT instead of GRUB_CMDLINE_LINUE_DEFAULT.
* src: isolate libfdisk operations to enable mount operationsv1.3.2-20Alejandro Sirgo Rica2024-10-082-3/+44
| | | | | | | | | | python-libfdisk does not close file descriptor until the cxt object goes out of scope. Define get_partition_data and get_disk_data functions to isolate the python-libfdisk logic and return the data as an object. Improve error handling of libfdisk operaions in refresh.
* src: fix cache mount in new ogLiveAlejandro Sirgo Rica2024-10-081-19/+4
| | | | | | | Mount cache in image_create() image_restore() and cache_fetch(). Remove init_cache() and ensure /opt/opengnsys/images/ exists within the cache mountpoint if it does not exists in cache_mount().
* src: consolidate compute_md5 functionsAlejandro Sirgo Rica2024-10-022-12/+19
| | | | | | | | | | Add compute_md5 function in src/utils/fs.py Remove identical md5 functions from src/live/ogOperations.py and src/utils/tiptorrent.py Move error checks from ogOperations.py into compute_md5 function in src/utils/fs.py
* src: check if the system is hibernated before /image/createAlejandro Sirgo Rica2024-10-022-4/+14
| | | | | | | Mount the system partition in readonly mode and check for the hiberfil.sys file if the target system is a Windows. Fail the image creation process if the target system is hibernated.
* tiptorrent: missing f-string in errorOpenGnSys Support Team2024-10-011-1/+1
| | | | tip_client_get() needs f-string to display error
* src: replace DEVICE env variable with get_ethernet_interface()Alejandro Sirgo Rica2024-10-013-4/+71
| | | | | | Use a python function to obtain the main net interface. Detect the first ethernet inferface in use. Stop using the DEVICE environment variable.
* src: report used and free partition data in bytesAlejandro Sirgo Rica2024-09-251-11/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | Add "used_size" and "free_size" to the partition data and the cache data. Old response from ogClient for /cache/delete, /cache/fetch and /image/restore: { 'cache': [ {'name': 'windows.img', 'size': 2432370213, checksum: '5d4dcc677bc19f40a647d0002f4ade90'}, {'name': 'linux.img', 'size': 243234534213, checksum: '3eb22f888f88a55ad954f55644e1192e'} ] } New response: { 'cache': { 'used_size': 4520232322423, 'free_size': 48273465287452945, 'images': [ {'name': 'windows.img', 'size': 2432370213, checksum: '5d4dcc677bc19f40a647d0002f4ade90'}, {'name': 'linux.img', 'size': 243234534213, checksum: '3eb22f888f88a55ad954f55644e1192e'} ] } }
* sw_inventory: consolidate os_probe logicAlejandro Sirgo Rica2024-09-252-7/+3
| | | | | | | | Reuse os_probe() function from probe.py and change the fallback case to "unknown" to prevent OS id mismatch in ogserver. The OS id mismatch causes the images to stop being associated with partitions after the /refresh command.
* src: verify the fields of the efibootmgr json in /refreshAlejandro Sirgo Rica2024-09-251-4/+27
| | | | | | | Don't send the efi data in the /refresh payload if efibootmgr is missing any of the json keys. Log the missing keys in case of missing some.
* utils: Add windows_register_c_drive implementationAlejandro Sirgo Rica2024-09-123-9/+92
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Add python implementation of the legacy ogWindowsRegisterPartition function. This function configures the system partition to be considered the new C drive after a Windows image restore. The drive letter configuration is stored in the SYSTEM hive of the windows registry. The node MountedDevices contains all the configuration as key-value pairs. The C drive key \DosDevices\C: contains a different value based on the partitioning type. GPT the value is DMIO:ID: followed by the 16 byte GUID of the partition. Example: DMIO:ID:\xc9\xfc\x1c\x86\x13\x11O\t\xa2\x9c{/\xf1\xdf\xe4) MBR The value is a little endian byte sequence with 2 parts. The first 4 bytes correspond to the disk id. The second part is the byte offset start of the partition in that disk. Example: \xe1\\\x9cP\x00\x00\x10j\x18\x00\x00\x00 If we format the MBR value in a more readable way we get e1 5c 9c 50 00 00 10 6a 18 00 00 00 In this case the disk ID is 509c5ce1. The partition offset is 186a100000. This patch adds the following helper functions to: - get_disk_id_bytes(): to obtain the disk identifier. - get_part_id_bytes(): to obtain a partition identifier as UUID for MBR or DMIO:ID format for GPT. - get_sector_size(): to query the sector size of a specific disk. Read /sys/class/block/{device_name}/queue/hw_sector_size to obtain the value. This is MBR specific. - get_partition_start_offset(): to query the start sector of a specific partition and disk. Use sfdisk with the -J argument to get fdisk data in json format. This is MBR specific.
* bcd: make recovery modifications optionalv1.3.2-19Alejandro Sirgo Rica2024-09-101-1/+5
| | | | Ignore recovery disable if no Recovery node is found in the BCD.
* utils: replace the legacy function ogConfigureFstabAlejandro Sirgo Rica2024-08-303-9/+235
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implement configure_fstab() as a replacement of ogConfigureFstab. Create src/utils/fstab.py to implement the main fstab configuration functions. Define two fstab helper classes, FstabBuilder and FstabEntry. FstabEntry Represents each line in the fstab file. Has the values: device, mountpoint, fstype, options, dump_code and pass_code. FstabBuilder Contains a list of FstabEntry. Handles loading of a preexisting fstab file and the serialization of multiple FstabEntry into a file. The fstab configuration has 3 main steps: Root partition: - Update the device field with the device where the new system is installed. Swap partition: - Preserve all the swapfile entries in every case. - If the filesystem has a swap partition: update the device field in the first fstab swap entry and remove the rest swap entries pointing to a swap partition. Only one swap partition is supported. Create a new fstab entry if no preexisting swap entry exists. - If the system has no swap partition remove every swap partition entry. EFI partition: - Update the device field of the EFI fstab entry if it exists. Create a new fstab entry if no preexisting EFI entry exists. Add get_filesystem_id to disk.py to obtain the UUID. Define every device field as a UUID. That method is more robust than a plain device path as it works after disks being added or removed.
* postinstall: ignore legacy scripts non-zero return codeOpenGnSys Support Team2024-08-301-54/+47
| | | | | legacy scripts are not reliable, legacy scripts continue processing on errors, warn on errors until they are converted to native code instead.
* src: append .img to log that report image fetchingOpenGnSys Support Team2024-08-291-1/+1
|
* src: use logging.warning()OpenGnSys Support Team2024-08-271-2/+2
| | | | | | | logging.warn() is deprecated since 3.3. And use .error() instead when command is unsupported or client is busy, that should not ever happen.
* utils: bcd: update logging related to BCD updates with MBR partitionOpenGnSys Support Team2024-08-271-1/+1
| | | | | MBR requires manual bcdedit invocation to generalize image before creation. Report via logging that no BCD update occurs.
* utils: return False instead of rising exceptionOpenGnSys Support Team2024-08-211-2/+4
| | | | | if image file and checksum are not found in the cache, then report checksum validation has failed so it can proceed to fetch new files from server.
* src: remove call to restoreImageCustom scriptOpenGnSys Support Team2024-08-211-19/+0
| | | | remove call to custom version of legacy script, display warning.
* postinstall: remove shell=TrueAlejandro Sirgo Rica2024-08-091-4/+5
| | | | Remove the use of shell=True.
* postinstall: ignore output when invoking legacy scriptsOpenGnSys Support Team2024-08-081-6/+12
| | | | output is never consumed.
* postinstall: add logging to report this stageOpenGnSys Support Team2024-08-081-0/+2
| | | | log that image restoration has entered OS configuration stage.
* live: move filesystem expansion out of OS configurationOpenGnSys Support Team2024-08-061-2/+0
| | | | just a clean up.
* postinstall: add logging to report postconfiguration script invocationOpenGnSys Support Team2024-08-061-0/+2
|
* postinstall: typo in maximum win hostname errorOpenGnSys Support Team2024-08-061-1/+1
| | | | s/nor/not
* postinstall: linux does not allow more than 64 bytes long hostnamesOpenGnSys Support Team2024-08-061-0/+4
| | | | | | | | | As per: $ getconf HOST_NAME_MAX 64 truncate it to the maximum.
* utils: disk: revisit logging to find EFI partitionOpenGnSys Support Team2024-07-301-4/+5
| | | | | | | | | | | | | | instead of Checking partition "Microsoft Basic Data"... Checking partition "EFI System"... show Searching EFI partition... EFI partition found at /dev/sda1 and refer to EFI partition consistently in logs.
* utils: postinstall: show hostname in logsOpenGnSys Support Team2024-07-301-4/+4
| | | | Display the hostname that is set in the logs.
* live: remove old image and checksum file on downloadOpenGnSys Support Team2024-07-301-0/+6
| | | | | | | Restoring an updated image file (with different checksum) could fail while the old checksum file could remain in place. Remove image and checksum file before fetching the new files.
* utils: tip_check_csum() uses local checksum fileOpenGnSys Support Team2024-07-301-4/+13
| | | | | | | | | | | | | | | | | | | .full.sum file in the local cache contains the local checksum for this file, this checksum is calculated by tip_write_csum(), therefore, there is no need to calculate this checksum again from the image file in the cache, use this checksum content instead. if .full.sum is not available, then cache is inconsistent, tip_check_csum() is called after checking if image file exists, raise an exception. checksum could mismatch in two situations: a) new image version (checksum is different) b) image is corrupted in both cases, a new fresh image needs to be retrieved. this speeds up checksum validation.
* utils: enhance logging for checksum mismatchOpenGnSys Support Team2024-07-301-0/+1
| | | | display checksum for server and local file when checksum mismatches.
* utils: add set_linux_hostnameAlejandro Sirgo Rica2024-07-291-0/+22
| | | | | Add set_linux_hostname function to redefine the hostname of a Linux install by overwriting the contents of /etc/hostname
* utils: add set_windows_hostnameAlejandro Sirgo Rica2024-07-291-0/+47
| | | | | | | | | | | | | | | | | | | | | | | Add function to redefine the hostname of a Windows install. Windows hostnames can't be larger than 15 characters due to legacy heritage. Hostname modification is done by modifying 3 registry values in the ControlSetXXX entry of the SYSTEM hive. ControlSet001 is generally the entry to be edited but one must query the value of the 'Current' key in the 'Select' entry of the SYSTEM hive to retrieve the active ControlSet. The hostname has to be introduced in the following entries: path = 'ControlSetXX/Control/ComputerName/ComputerName' key = 'ComputerName' path = 'ControlSetXXX/Services/Tcpip/Parameters' key = 'HostName' key = 'NV Hostname' The value to store in those keys is of tipe SZ and has to be encoded in UCS-2 Little Endian (utf-16-le).
* utils: add postinstall.pyAlejandro Sirgo Rica2024-07-292-21/+170
| | | | | | | | | | | | | | | Replace ConfigureOs script by native calls to: - update BCD via hivex using bcd.py and winreg.py infrastructure. - restore efi bootloader restore_windows_efi_bootloader(). Call legacy scripts for remaining postinstall procedures to replace them incrementally. Define variable CONFIGUREOS_LEGACY_ENABLED as False by default. Run legacy configureOs when CONFIGUREOS_LEGACY_ENABLED = True. This serves as a auxiliar method to keep the restoration functional in case of problems with the new configure_os logic.
* utils: add BCD native supportAlejandro Sirgo Rica2024-07-291-0/+198
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implement update_bcd() as replacement of ogWindowsBootParameters. The legacy function modified the BCD of a freshly restored system invoking the privative binary "spartlnx". The script edits a set of BCD entries needed for a proper system boot. Each main BCD entry is identified by an UUID and contain a set of subnodes, these hold the configuration and entry information. Each node contains data in the form of key-value. Common BCD structure: 'Objects' ... The Boot Manager entry is always identified by the UUID 9dea862c-5cdd-4e70-acc1-f32b344d4795. Some entries always have the same UUID as identifier such the Boot Manager while other have different UUID depending on the system. To identify these entries with a not known UUID we query the value 'Type' of the node 'Description'. This contains a 32 bit value whose bytes codify the nature of the entry. We obtain 3 different values as a tuple, each value is the result of applying a bitmask to the Type value. These masks are 0xf0000000, 0x00f00000 and 0x000fffff. The resulting tuple of 3 values is then used to obtain the corresponding entry from a map. The data we modify from the BCD are disk and partition references to point to the new disk and partition in the system where the images are restored. Partition and disk information is stored as UUID in specific offsets inside binary data in the BCD. To update these we need to obtain the disk and partition UUIDs, then convert it to bytes as follows: original UUID: C4C61C51-3456-4733-96AD-AE760A41C392 UUID as bytes: 51 1C C6 C4 56 34 33 47 96 AD AE 76 0A 41 C3 92 The modified entries are: Resume from Hibernation, Windows Boot Loader OS, Windows Boot Loader Recovery, Windows Recovery, Boot Loader Settings, Windows Boot Manager and Windows Memory Diagnostic. Some of these options could be omited as the system restoration does not include a recovery partition so in this case all the recovery related entries just point to the main system partition. Most entries are edited modifying the value corresponding to the key 'Element' in 2 subnodes of the 'Elements' node. These subnodes are '11000001' and '21000001'. 'Objects' The 2 values stored in these 2 entries is binary data where we store the partition and disk bytes. We simply replace the byte representation of out UUIDS in the binary data. Partition is stored in offset 32 and the disk in offset 56. The exception is the Bootloader Recovery entry, in which the partition offset is 84 and the disk offset 108. Note that the legacy function only does a proper BCD edit in UEFI systems. The new implementation follows the same behavior with the possibility of implementing BCD modification under MBR partitions in the future. Set the field 16000009 (RECOVERY_ENABLED) to the value x00 to disable recovery in both Resume from Hibernation and Windows Boot Loader OS entries in the BCD. The system install does not include any recovery partition so it makes no sense to have it enabled. This commit is preparatory work for the new native postinstall code.
* utils: add win edit registry utilitiesAlejandro Sirgo Rica2024-07-291-0/+114
| | | | | | | | | | | | | | | | | Add winreg.py to the utils folder. Implement hive enum types, hive handler validation and validated get_* functions for nodes and registry values. Implement the utility hive operations through the Hivex library. This serves as preparatory work for BCD manipulation but it also has potential to improve registry usage in previous code. UCS-2 Little Endian is the prefered windows registry text encoding for binary content. Define a WINDOWS_HIVE_ENCODING global variable to use when encoding string to write in the win registry. This commit is preparatory work for the new native postinstall code.
* utils: add restore_windows_efi_bootloaderAlejandro Sirgo Rica2024-07-291-0/+31
| | | | | | | Add restore_windows_efi_bootloader to copy the EFI loader from the filesystem in the restored system into the EFI partition. This commit is preparatory work for the new native postinstall code.
* utils: add functions to obtain disk and partition UUIDsAlejandro Sirgo Rica2024-07-291-0/+25
| | | | | | | | Add get_partition_id and get_disk_id to obtain the UUID of a disk or partition as a string. These values are obtained invoking the program blkid as a subprocess. This commit is preparatory work for the new native postinstall code.
* utils: consolidate code to find efi loaderAlejandro Sirgo Rica2024-07-292-30/+29
| | | | | Add find_windows_efi_loader and find_linux_efi_loader to reduce code duplication and to centralize efi loader path modifications.
* utils: rename ogExtendFs to extend_filesystemAlejandro Sirgo Rica2024-07-291-1/+1
| | | | Use a more consistent name format for the ogExtendFs function.
* utils: fs: error out if mkfs failsOpenGnSys Support Team2024-07-291-5/+17
| | | | | | | | | | If formatting fails, log shows: mkfs.ntfs reports return code 1 for /dev/sda2 but ogclient reports success to the ogserver. Raise an exception so front-end gets an indication that formatting has failed.
* utils: consolidate logging to write checksum file to cacheOpenGnSys Support Team2024-07-211-2/+3
| | | | Move log message to function that is called both by unicast and tiptorrent.
* utils: missing f-string with filesystem resize errorv1.3.2-13OpenGnSys Support Team2024-07-151-1/+1
| | | | ... failed to resize {partdev} with ext4
* utils: clean up error reporting related to checksumOpenGnSys Support Team2024-07-151-2/+2
| | | | | | | | When checksum is not available, it displays: (2024-07-15 09:04:14) ogClient: [ERROR] - URL error when fetching checksum: Not Found "URL error" is leaking an internal implementation details, reword this report.
* live: add checksum field to image/create responseOpenGnSys Support Team2024-07-151-0/+1
| | | | Report image checksum to ogserver through HTTP response.