summaryrefslogtreecommitdiffstats
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
* live: hw_inventory: fix typoJose M. Guisado2023-10-181-1/+1
| | | | | | Add missing underscore to _bytes_to_human call. Fixes: 39c13287c53bd8 ("live: hw_inventory: fix empty memory bank bug")
* live: hw_inventory: fix empty memory bank bugJose M. Guisado2023-10-051-6/+9
| | | | | | | | | | | | When a client's hardware presents an empty memory bank and invalid call to _bytes_to_human is performed because None is passed as a parameter. size = _bytes_to_human(obj.get('size', None)) Fix this by checking if 'size' is present in the JSON output from lshw. If size is present then map the bytes to a human readable string using _bytes_to_human, if no size is present then use 'Empty slot' to indicate that the memory bank is not being used.
* src: improve logging messagesJose M. Guisado2023-08-012-9/+15
| | | | | | | | | | | | | | | | | | | | | | | | | Some users have mistakenly reported tiptorrent problems when the process takes a long time. Specifically by rebooting or powering off the client in the middle of the md5sum computation stage, just after the tiptorrent transfer. Same problem occurs when image creation command takes a long period of time. In order to help the user understand the different stages of commands such as image creation or image restore using tiptorrent, the following changes have been made to the current logging solution: - Add log messages to warn users not to reboot or shut down the client during a tiptorrent transfer, and also during the md5sum computation stage. - Add a log message telling the user that the image creation processes have started. - Use logging.exception inside "except:" blocks to print a traceback with the log messsage. (https://docs.python.org/3/library/logging.html#logging.exception)
* hw_inventory: use dict.getJose M. Guisado2023-07-211-23/+27
| | | | | | | | | The first stage of parsing the "lshw -json" command output is to load the json string into a Python dictionary. lshw output is large and varies from machine to machine, so it's not safe to assume that different keys will be present in the dictionary. Use dict.get() instead of dict[key] to avoid KeyError exceptions.
* live: add image backup option in image creationv1.3.1Jose M. Guisado2023-07-061-0/+6
| | | | | | | | | | | | Backup image file if image creation request included "backup": true This only applies when the target image is already present in the repository folder before running the partclone subprocess. This parameter is ignored if the target image is not present in the repository.
* rest: add backup parameterJose M. Guisado2023-07-061-0/+7
| | | | | | | Support parsing of "backup" parameter inside incoming requests. Prepare ogClient support for backup image creation in "image create" operation.
* ogclient: add support for X-Sequence headerv1.3.0Jose M. Guisado2023-06-143-22/+35
| | | | | | | | | | | | | | | Enable parsing of "X-Sequence" HTTP headers from incoming requests. Add "seq" field in restRequest class. Enable adding "X-Sequence" to outgoing responses. Add "seq" field inside restResponse class. Store current client sequence number inside ogClient class. Ideally, the restRequest object should be used to retrieve the sequence number but not all processing functions inside ogRest.py receive the request as parameter (eg: process_refresh). In the other hand, all processing functions receive the ogClient object.
* fs: fix subprocess input inside _extend_resize2fsJose M. Guisado2023-06-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | The subprocess module expects bytes-like object for "input" parameter by default. Passing a string object result in the following error: (2023-06-13 14:44:43) ogClient: [ERROR] - Exception when running "image create" subprocess (2023-06-13 14:44:43) ogClient: [ERROR] - Unexpected error Traceback (most recent call last): File "/opt/opengnsys/ogClient/src/live/ogOperations.py", line 465, in image_create ogExtendFs(disk, partition) File "/opt/opengnsys/ogClient/src/utils/fs.py", line 124, in ogExtendFs _extend_ntfsresize(partdev) File "/opt/opengnsys/ogClient/src/utils/fs.py", line 250, in _extend_ntfsresize proc = subprocess.run(cmd, input='y') File "/usr/lib/python3.8/subprocess.py", line 495, in run stdout, stderr = process.communicate(input, timeout=timeout) File "/usr/lib/python3.8/subprocess.py", line 1013, in communicate self._stdin_write(input) File "/usr/lib/python3.8/subprocess.py", line 962, in _stdin_write self.stdin.write(input) TypeError: a bytes-like object is required, not 'str' Fixes: dd999bfe34e7 ("utils: rewrite ogReduceFs")
* fs: fix bug when ntfsresize reports nothing to doJose M. Guisado2023-06-131-0/+3
| | | | | | | | There is a corner case in which a target NTFS filesystem is already shrunken. When this happens ntfsresize text output parsing breaks. Check when ntfsresize reports nothing to do, warn the user about this and stop the dry-run ntfsresize loop.
* fs: fix typo inside _extend_resize2fsJose M. Guisado2023-06-131-4/+4
| | | | | | | | | | _extend_ntfsresize contains an incorrect variable name inside subprocess.run referring the resize command value. Simplify this variable name inside each specific _extend_* function: s/cmd_resize2fs/cmd s/cmd_ntfsresize/cmd
* src: remove unused legacy software inventory codeJose M. Guisado2023-05-172-26/+9
| | | | | | | | | | | | | Remove unnecessary InventarioSoftware invocation inside image_create operation. Software inventory is executed after image creation (see ogRest.py). Remove legacy 'path' parameter. This parameter was used to specify the path of a text file in which legacy bash scripts wrote the software inventory of the client (something like "Csft-{ip}..."). Fixes: 04bb35bd86b58c ("live: rewrite software inventory") Fixes: 2e3d47b7b8db69 ("Avoid writting /software output to a file")
* sw_inventory: ignore invalid windows programsJose M. Guisado2023-05-091-5/+10
| | | | | | | | | | | | | | | | | | | | | | | Don't raise exception if any windows program is missing DisplayName node in the windows registry. This attribute/node should contain the program's name. This name is used as the package's name in the software set (software inventory). This patch should be considered a hotfix, python-hivex does not report any helpful message about this error. (2023-05-09 14:43:13) ogClient: [ERROR] - Unexpected error Traceback (most recent call last): [...] RuntimeError: Success Before this patch, image creation *might* fail because it cannot create the software inventory associated with the image due to the previously described error. The software inventory is part of the response payload of the image creation command (see src/ogRest:image_create). Fixes: 04bb35bd86b5 (live: rewrite software inventory)
* live: rewrite reboot operationv1.2.9Jose M. Guisado2023-05-091-5/+3
| | | | | | Add optional 'operation' parameter to _poweroff_oglive function. Reuse _poweroff_oglive code before the busybox subprocess when rebooting an ogLive client.
* live: rewrite poweroff operationJose M. Guisado2023-05-091-3/+20
| | | | | | | | | | | | | Replace legacy bash script /opt/opengnsys/client/scripts/poweroff with a Python native solution. Use subprocess module for any required external program when shutting down a client. ethtool is used to ensure WoL setting is correct before shutting down. ogLive does not properly use a init system so busybox is used when shutting down the system. In other live environments poweroff operation just calls /sbin/poweroff.
* utils: add umount_all functionJose M. Guisado2023-05-091-0/+9
| | | | | | | | Add utility function to unmount any mountpoint present in the /mnt folder. This function is a simplified version of the legacy bash function ogUnmountAll used in several operations.
* utils: rewrite ogExtendFsJose M. Guisado2023-05-091-7/+28
| | | | | | | | | | | | | | | | | | | | Drop subprocess call to bash function ogExtendFs. Use a native python solution with subprocess calls to the required underlying tools. Use get_filesystem_type to get the present filesystem from a partition and call the corresponding filesystem grow function. Filesystem specific functions are declared "_extend_{filesystem}" and should not be imported elsewhere. Each filesystem specific function wraps a subprocess call to the required underlying program: - NTFS filesystems: "ntfsresize -f [partition]" - ext4 filesystems: "resize2fs -f [partition]" Set NTFS related subprocess stdin to 'y' because human input cannot be unset with other ntfsresize parameters.
* utils: rewrite ogReduceFsJose M. Guisado2023-05-021-9/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | Drop subprocess call to bash function ogReduceFs. Use a native python solution with subprocess calls to the required underlying tools. Use get_filesystem_type to get the filesystem from a partition and call the corresponding supported filesystem shrink function. Filesystem specific functions are declared "_reduce_{filesystem}" and should not be imported elsewhere. In case of NTFS filesystems, the output of 'ntfsresize' is processed directly. This is dirty, but we can expect no changes to the output strings if we read the following comment in the nftsresize.c source code: https://github.com/tuxera/ntfs-3g/blob/edge/ntfsprogs/ntfsresize.c#L12 ntfsresize requires to do previous dry-run executions to confirm that the resizing is possible. If a dry-run fails but a 10% increase in size is still smaller than original filesystem then retry the operation until dry-run reports sucess or the size increase is bigger than original. If resizing to a smaller ntfs filesystem is not possible then ogReduceFs will do nothing.
* utils: add get_filesystem_type functionJose M. Guisado2023-04-251-0/+12
| | | | | Retrieve filesystem type from a partition using get_filesystem_type. Encapsulates a subprocess call to blkid.
* src: add missing copyright noticesv1.2.8Jose M. Guisado2023-04-184-0/+32
|
* live: rewrite hardware inventory commandJose M. Guisado2023-04-182-20/+16
| | | | | | | | | | | | | | | | Replace legacy shell script InventarioHardware for helper functions from hw_inventory.py Use get_hardware_inventory to obtain a HardwareInventory object with the hardware information. Map the HardwareInventory object to a legacy response string with the legacy_list_hardware_inventory function. Remove "Chrd-*" file reading logic, it's no longer needed. Legacy shell script InventarioHardware uses that file. Expect a change in the structure of hardware inventory response payload in the future. This patch does not address the HTTP response containing the hardware inventory as a '\n' separated string of hardware elements.
* utils: add hw_inventory.pyJose M. Guisado2023-04-181-0/+309
| | | | | | | | | | | | | | | | | | hw_inventory.py defines classes and helpers functions enabling fetching of hardware inventory from a running client. Uses a subprocess call to the command 'lshw -json' to obtain hardware information. Relevant public functions: > get_hardware_inventory() Main function encapsulating subprocess and output processing logic. Returns a HardwareInventory object. > legacy_list_hardware_inventory(inventory) Legacy string representation of parameter HardwareInventory object
* utils: rename inventory.py to sw_inventory.pyJose M. Guisado2023-04-182-1/+1
| | | | | | | | Rename software inventory file to sw_inventory to better distinguish it from a future hardware inventory code. In the future sw_inventory and hw_inventory might be merged together once each file is tidied up.
* live: rewrite software inventoryv1.2.7Jose M. Guisado2023-03-272-19/+134
| | | | | | | | | | | | | | | | | | | | | | | | | | | Replace legacy bash script in favor of Python code. Improves error traceability and further development. The software inventory operation mounts the target partition and it fetches the list of installed software (package set). Once the operation is complete, it unmounts the target partition. For Windows, introduce hivex library python bindings for accessing Windows registry hive files (https://libguestfs.org/hivex.3.html). This operation is still processed by legacy code in the server side (ogAdmServer.c in ogServer). Legacy backend process expects the software inventory like the following example: "software": "Windows 10 Enterprise Evaluation 2004 \nIntel(R) Network Connections 24.0.0.11 24.0.0.11 ..." The os name is inserted first in this list followed by a '\n' separated string of the software packages. The legacy server code can be found in function actualizaSoftware at ogServer/src/ogAdmServer.c It is expected for software inventory payload to change in the future to a simpler solution using just a json array of strings.
* live: fix ogChangeRepo invocationJose M. Guisado2023-03-271-1/+1
| | | | | | | | | | | | | Add missing samba credentials parameter in ogChangeRepo invocation. Credentials are loaded from ogClient config file. Any production deployment should use its own samba user and password. ogChangeRepo fails when using default samba credentials in a production environment. Fixes: a1edbe904b30 ("legacy: rewrite ogChangeRepo") Fixes: 3703fd60634f ("live: support native unicast cache image restore")
* live: fix unicast cache error log messageJose M. Guisado2023-03-271-1/+1
| | | | | | | | | Removes undefined 'repo' variable from error logging message. This caused the traceback to be polluted with an unhelpful message about this variable being undefined. Fixes: 3703fd606 ("live: support native unicast cache image restore")
* live: add linux swap parttypeJose M. Guisado2023-03-231-10/+12
| | | | | | | Adds linux swap partition type, mapped to the 'LINUX-SWAP' string in web interfaces like ogCP or webconsole. Fixes: 29c53e54e945 ("live: add parttypes.py")
* ogRest: improve error logging when executing operationsv1.2.6Jose M. Guisado2023-03-101-32/+23
| | | | | | | | | | | | | Capture all possible Python exceptions in the try/except block of every opengnsys operation. Create an error handling function to deduplicate code in the except block. The error handling function resets the ogRest state to IDLE and sends the corresponding 500 Internal Server Error. This *does not cover* every possible error. There are functions inside ogThread which contain code that may raise errors that are not covered by any try/except block.
* ogRest: remove root logger constantJose M. Guisado2023-03-101-7/+6
| | | | | | | | | | | | | | | | Remove unnecesary root logger constant: LOGGER The root logger is used by default when executing: logging.debug() logging.info() logging.warning() ... There is no point in doing: LOGGER = logging.getLogger() # Get root logger LOGGER.debug() # Use root logger
* legacy: improve readability of ogGetImageInfo helper functionsJose M. Guisado2023-03-101-14/+13
| | | | | | | | | | | | | | | | | | | | Change the name of the helper functions used when getting opengnsys image information (legacy ogGetImageInfo bash script). As of now the process consist of decompressing the image file with lzop and feeding that output to partclone.info. Prefer a more explicit function name rather than "process_image_*" Add comment about skipping the first two lines of partclone.info output. Usually, partclone.info starts printing out these two lines that are not related to the partclone image information: Partclone v0.3.23 http://partclone.org Showing info of image (-) As long as partclone.info output doesn't change we'll be fine, but we should not depend on human readable output. This might change in the future (i.e. adding json output format to partclone.info).
* legacy: rewrite ogGetImageInfoJose M. Guisado2023-03-022-18/+108
| | | | | | | | | | | | | | | | | | | | Rewrites this legacy script behavior using native Python code, using subprocess module when executing programs like partclone.info or lzop ogGetImageInfo is a bash script that retrieves information regarding an OpenGnsys partition image, specifically: - clonator - compressor - filesystem - datasize (size of the partition image) This rewrite only supports partclone and lzop compressed images. This is standard behavior, we have no reports of other programs or compression algorithms in use. Keep this legacy function with hungarian notation to emphasize this is still a legacy component that may be replaced in the future.
* legacy: rewrite ogChangeRepoJose M. Guisado2023-03-022-7/+53
| | | | | | | | | | | | | | | | Drop ogChangeRepo Bash script in favor of a native Python approach. Use only necessary subprocess calls instead of bringing all the logic of this function into a Bash script black box. ogChangeRepo unmounts the current OpenGnsys image samba folder (/opt/opengnsys/images) and mounts (connects to) a new directory using the new provided ip address. Keeping access mode from previous mount. If anything goes wrong when mounting the new directory, it will fallback to mounting the previous directory. If no previous OpenGnsys image samba directory is detected, this functions tries to mount the new directory anyway. In this case, it will raise CalledProcessError if something goes wrong.
* legacy: improve readability of cambiar_accessoJose M. Guisado2023-03-021-10/+10
| | | | | Expand function docstring and do not use CalledProcessError handling to return True or False. Just checking for returncode value is simpler.
* live: rewrite setup operationv1.2.5Jose M. Guisado2023-02-091-16/+44
| | | | | | | | | | | | | | | | Rewrites the setup operation using python-libfdisk module instead of an external bash script. Consolidating the operation into Python's code, limiting external subprocesses to well known programs and small concrete tasks that are difficult to fully integrate into Python. Use parttypes.py to fetch partition types from python-libfdisk module. Use fs.py to create any specified supported filesystem. OpenGnsys cache partitions are created labelling the partition as "CACHE". Stops setting non-standard MBR hexcode (0xca) to the cache partition in addition to the filesystem label. Any partition specified as type EMPTY will be ignored.
* utils: add init_cache functionJose M. Guisado2023-02-081-0/+12
| | | | | | | | | | | | | | | init_cache() creates the default directory in which OpenGnsys stores images when using any cache enabled transfer method. As of this commit this folder must exist for tiptorrent.py to work properly. Subprocess Popen object inside tiptorrent.py use 'cwd' optional parameter like: cwd='/opt/opengnsys/cache/opt/opengnsys/images/' This folder convention might change in the future.
* utils: add fs.pyJose M. Guisado2023-02-081-1/+66
| | | | | | | | | | | Adds utility module which wraps several mkfs.* calls as a subprocess. The main utility function is mkfs(fs, disk, partition, label), which subsequently calls the corresponding mkfs_*(partition_device) function. mkfs() supports specifying a drive label where supported. Other modules using fs.py should call mkfs() only.
* live: add parttypes.pyJose M. Guisado2023-02-081-0/+55
| | | | | | | | | | | | | Adds parttypes.py module with utility functions to get partition types (parttypes) from python-libfdisk. Supports standard partition types, either DOS or GPT. DOS labels use a hex code to define partition types, python-libfdisk exposes get_parttype_from_code to look up for DOS partition types from a given hexcode. GPT label uses a string (UUID) for each supported partition type, python-libfdisk exposes get_parttype_from_string to look up for GPT partition types from a given string.
* live: clear ogbrowser log before image_createv1.2.4Jose M. Guisado2022-12-051-0/+1
| | | | | Remove any previous unrelated log message shown in the "real time log" html page before executing this command.
* log: add file handler for ogLive "real time log"Jose M. Guisado2022-12-051-0/+9
| | | | | | | | | | | Clients running in ogLive can show log messages via a lighttp server. Particularly, a html page named "real time log" consists of <text-area> tags with the contents of two particular text files /tmp/session.log and /tmp/command.log Adds a Python logging handler in order to write ogClient log messages into /tmp/session.log. This way ogClient logs are show in the "real time log" html page too.
* live: clear ogbrowser logs before image_restoreJose M. Guisado2022-12-011-1/+7
| | | | | | | | | | Clears content of blue text areas in the real time log view before executing a restore image operation. Adds private function _ogbrowser_clear_logs, this function writes to a couple of text files present in the ogLive environment. The contents of this file are printed out to the blue text areas in the "real time log" view.
* live: improve error pathsv1.2.3Jose M. Guisado2022-11-172-5/+22
| | | | | | | | | | | | | Fix error paths in live operations which do not reset the "browser" to the main page (one with the menu). Add error logging messages when: * _restartBrowser fails. * ogChangeRepo fails. Improve checksum fetch error handling. For example, when an invalid repository IP is specified.
* log: declare samba handler only in live modeJavier Sánchez Parra2022-11-021-9/+20
| | | | | | ogClient in linux mode only write its logs to syslog and the console. Also, raise exception if mode its not supported.
* log: import fcntl only on linuxJavier Sánchez Parra2022-11-021-67/+73
| | | | | Otherwise, ogClient do not work on Windows because fcntl module do not exists on Windows.
* live: support native unicast cache image restoreJose M. Guisado2022-11-021-3/+32
| | | | | | | | | | | | | UNICAST-CACHE consist of: 1. Checking if the target image is already present at the opengnsys cache partition. If so, check for integrity (local and remote checksum). If the image is not present in the cache partition, download the target image into it. 2. Restore the image from cache partition. This commit add support for this operation natively from ogClient Python's code.
* tiptorrent: improve raised exceptions for tip_client_getJose M. Guisado2022-09-291-4/+2
| | | | | | Raise exception when tiptorrent-client subprocess runs normally but exits with non-zero code. (For example, if download file allocation failed)
* live: don't use python open() when reading imageJose M. Guisado2022-09-231-5/+3
| | | | | | | | Specifies the image path in the lzop subprocess string. It might be interesting to study efficient mechanisms to read large binary files in python before using open() with default parameters for buffered binary reading.
* live: rewrite image_restoreJose M. Guisado2022-09-143-21/+99
| | | | | | | | Integrates image restore command into native ogClient code. Further reduces the need for external Bash scripts. After a succesful image restore, OS configuration is still using external Bash script "osConfigure/osConfigureCustom".
* utils: add tiptorrent.pyJose M. Guisado2022-09-141-0/+90
| | | | | Utility and wrapper functions related to the usage of tiptorrent. To be used by the image restore command.
* utils: add get_partition_deviceJose M. Guisado2022-09-141-0/+17
| | | | | Maps a disk and partition number to the corresponding device path ('/dev/sdXY', '/dev/nvmeXnYpZ', ...)
* image_restore: fix ogCopyEfiBootLoaderJose M. Guisado2022-09-142-4/+5
| | | | | | | | | ogCopyEfiBootloader is an invalid legacy bash function name. Rename to the correct function name 'ogCopyEfiBootLoader' and rename utility python wrapper too. Fixes: 0bd037c1a409c65fbcb01355ee0dd6dca770330e
* utils: minor fix for ogReduceFs and ogExtendFsJose M. Guisado2022-09-142-12/+11
| | | | | | | | | | Do not return the subprocess result for ogReduceFs/ogExtendFs. ogReduceFs works with or without the target filesystem mounted. ogExtendFs requires the target filesystem to be mounted. 'ogMount' legacy script invocation should be replaced by a better mount/umount wrapper.