tinyapps.org · 1mo · discuss
Increasing a UTM virtual disk's size does not automatically expand the guest's APFS container. If a Recovery partition lies between the container's partition and the added free space, it blocks direct expansion. This Codex/Astra-crafted script uses Python 3.9+ standard library and macOS diskutil, cp, and lsof; no pip packages, Go, Homebrew, or QEMU are required. It targets macOS 26+ with diskutil image support and an APFS host filesystem for copy-on-write backups: #!/usr/bin/env python3 # Grow an offline Apple Silicon UTM image. Python stdlib + macOS diskutil/cp. import argparse import hashlib import json import os from pathlib import Path import plistlib import re import struct import subprocess import sys import uuid import zlib RECOVERY = uuid.UUID('52637672-7900-11aa-aa11-00306543ecac').bytes_le APFS = uuid.UUID('7c3457ef-0000-11aa-aa11-00306543ecac').bytes_le ISC = uuid.UUID('69646961-6700-11aa-aa11-00306543ecac').bytes_le CHUNK = 8 * 1024 * 1024 def require(ok, message): if not ok: raise ValueError(message) def read_at(fd, length, offset): data = os.pread(fd, length, offset) require(len(data) == length, 'Short read at %d' % offset) return data def write_at(fd, data, offset): while data: n = os.pwrite(fd, data, offset) require(n > 0, 'Short write at %d' % offset) offset += n data = data[n:] def header(fd, sector, lba, total): raw = bytearray(read_at(fd, sector, lba * sector)) require(raw[:8] == b'EFI PART', 'Missing GPT signature') revision, length, checksum, reserved = struct.unpack_from('<4I', raw, 8) require(revision == 0x10000 and 92 <= length <= sector and reserved == 0, 'Unsupported GPT header') struct.pack_into('<I', raw, 16, 0) require(zlib.crc32(raw[:length]) == checksum, 'GPT header CRC mismatch') struct.pack_into('<I', raw, 16, checksum) current, backup, first, last = struct.unpack_from('<4Q', raw, 24) table, count, entry_size, crc = struct.unpack_from('<Q3I', raw, 72) require(current == lba and 1 <= backup < total and backup != current, 'Invalid GPT header locations') require(2 <= first <= last < total, 'Invalid usable GPT bounds') require(0 < count <= 16384 and entry_size >= 128 and entry_size % 128 == 0 and count * entry_size <= 16 * 1024 * 1024, 'Invalid GPT array dimensions') sectors = (count * entry_size + sector - 1) // sector require(table >= 2 and table + sectors <= total, 'GPT array outside disk') padded = read_at(fd, sectors * sector, table * sector) entries = padded[:count * entry_size] require(zlib.crc32(entries) == crc, 'GPT partition array CRC mismatch') return dict(raw=raw, length=length, current=current, backup=backup, first=first, last=last, table=table, count=count, entry_size=entry_size, entries=entries, sectors=sectors) def inspect(fd, sector, size): require(sector in (512, 4096) and size % sector == 0, 'Unsupported disk geometry') total = size // sector mbr = read_at(fd, sector, 0) require(mbr[510:512] == b'\x55\xaa', 'Missing protective MBR') require(mbr[450] == 0xee and mbr[446+16:510] == bytes(48), 'Unsupported hybrid or non-protective MBR') h = header(fd, sector, 1, total) b = header(fd, sector, h['backup'], total) require(b['backup'] == 1 and h['raw'][56:72] == b['raw'][56:72] and h['first'] == b['first'] and h['last'] == b['last'] and h['count'] == b['count'] and h['entry_size'] == b['entry_size'] and h['entries'] == b['entries'], 'GPT copies disagree; restore backup before retrying') require(h['table'] == 2 and h['table'] + h['sectors'] <= h['first'] and b['table'] > h['last'] and b['table'] + b['sectors'] == b['current'], 'Unsupported GPT metadata layout') parts = [] for i in range(h['count']): e = h['entries'][i*h['entry_size']:(i+1)*h['entry_size']] if e[:16] == bytes(16): continue start, end = struct.unpack_from('<QQ', e, 32) require(h['first'] <= start <= end <= h['last'], 'Partition outside usable bounds') parts.append(dict(index=i+1, type=e[:16], start=start, end=end, name=e[56:128].decode('utf-16-le').rstrip('\0'))) ordered = sorted(parts, key=lambda p: p['start']) require(all(a['end'] < b['start'] for a,b in zip(ordered, ordered[1:])), 'Overlapping GPT partitions') # Deliberately narrow: no ambiguous targets, extra disks, or guessed recovery sources. require([p['type'] for p in ordered] == [ISC, APFS, RECOVERY], 'Expected exactly ISC, main APFS, and Recovery partitions in that order') recovery = ordered[-1] sb = read_at(fd, max(sector, 4096), recovery['start'] * sector) require(sb[32:36] == b'NXSB', 'Recovery has no APFS superblock; refusing guessed repair') blocksize, blocks = struct.unpack_from('<IQ', sb, 36) require(blocksize in (4096, 8192, 16384, 32768, 65536) and blocks * blocksize == (recovery['end']-recovery['start']+1)*sector, 'Recovery APFS size does not match its GPT partition') last = total - h['sectors'] - 2 length = recovery['end'] - recovery['start'] + 1 # Preserve APFS block alignment, leaving at most one APFS block at the end. align = max(blocksize, sector) // sector start = ((last - length + 1) // align) * align require(start >= recovery['start'], 'Shrinking is not supported') require(start == recovery['start'] or start > recovery['end'], 'Growth smaller than Recovery size is unsupported (copy would overlap)') return h, parts, dict(source=recovery['start'], destination=start, sectors=length, last=last, index=recovery['index'], main_index=ordered[1]['index'], block_size=blocksize) def digest(fd, offset, length): result = hashlib.sha256() for off in range(0, length, CHUNK): result.update(read_at(fd, min(CHUNK, length-off), offset+off)) return result.hexdigest() def make_header(h, current, backup, table, last, entries): raw = bytearray(h['raw']) struct.pack_into('<QQ', raw, 24, current, backup) struct.pack_into('<Q', raw, 48, last) struct.pack_into('<Q', raw, 72, table) struct.pack_into('<I', raw, 88, zlib.crc32(entries)) struct.pack_into('<I', raw, 16, 0) struct.pack_into('<I', raw, 16, zlib.crc32(raw[:h['length']])) return raw def relocate(fd, sector, size): h, parts, plan = inspect(fd, sector, size) print('Relocation:', json.dumps(plan), flush=True) src, dst, length = (plan[k]*sector for k in ('source', 'destination', 'sectors')) if src == dst: print('Recovery already at the aligned end of disk.', flush=True) return plan source_hash = digest(fd, src, length) for off in range(0, length, CHUNK): write_at(fd, read_at(fd, min(CHUNK, length-off), src+off), dst+off) os.fsync(fd) require(digest(fd, dst, length) == source_hash, 'Recovery copy SHA-256 mismatch; GPT untouched') print('Recovery SHA-256 verified:', source_hash, flush=True) entries = bytearray(h['entries']) struct.pack_into('<QQ', entries, (plan['index']-1)*h['entry_size']+32, plan['destination'], plan['destination']+plan['sectors']-1) padded = entries + bytes(h['sectors']*sector-len(entries)) end = size//sector-1 backup_table = end-h['sectors'] # Copy first, then backup GPT, then primary GPT. An interrupted metadata commit # fails validation on rerun; the mandatory image backup is the rollback path. write_at(fd, padded, backup_table*sector) write_at(fd, make_header(h, end, 1, backup_table, plan['last'], entries), end*sector) os.fsync(fd) write_at(fd, padded, h['table']*sector) write_at(fd, make_header(h, 1, end, h['table'], plan['last'], entries), sector) mbr = bytearray(read_at(fd, sector, 0)) struct.pack_into('<I', mbr, 458, min(end, 0xffffffff)) write_at(fd, mbr, 0) os.fsync(fd) inspect(fd, sector, size) return plan def run(*args, plist=False): print('+', ' '.join(str(a) for a in args), flush=True) result = subprocess.run([str(a) for a in args], check=True, stdout=subprocess.PIPE) if plist: return plistlib.loads(result.stdout) print(result.stdout.decode(), end='', flush=True) return result.stdout def attach(path, readonly=False): args = ['diskutil', 'image', 'attach', '--noMount', '--plist'] if readonly: args.append('--readOnly') data = run(*args, path, plist=True) nodes = [e['dev-entry'] for e in data['system-entities'] if e.get('content-hint') == 'GUID_partition_scheme'] require(len(nodes) == 1, 'Expected one GPT device from image attachment') return '/dev/' + nodes[0].removeprefix('/dev/') def device_info(dev): info = run('diskutil', 'info', '-plist', dev, plist=True) require(info.get('WholeDisk') and info.get('BusProtocol') == 'Disk Image' and not info.get('Internal'), 'Refusing a non-image or internal device') return info['DeviceBlockSize'], info['TotalSize'] def image_path(path): path = path.resolve(strict=True) if path.suffix == '.utm' and path.is_dir(): with open(path/'config.plist', 'rb') as f: config = plistlib.load(f) require(config.get('Backend') == 'Apple', 'Only Apple Virtualization VMs supported') drives = [d for d in config['Drive'] if not d.get('ReadOnly') and d.get('ImageName')] require(len(drives) == 1, 'Select the image explicitly for a multi-drive VM') image = (path/'Data'/drives[0]['ImageName']).resolve(strict=True) require(image.parent == path/'Data', 'Image path escapes VM bundle') return image require(path.is_file(), 'Pass a .utm bundle or disk image file, never a device') return path def check_idle(path): p = subprocess.run(['/usr/sbin/lsof', '-t', str(path)], capture_output=True) require(p.returncode in (0, 1), 'Cannot check image users with lsof') require(not p.stdout.strip(), 'Image is open: shut down VM and detach image first') def main(): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('image', type=Path) parser.add_argument('--size-gib', type=int, required=True, help='Target virtual capacity in GiB') parser.add_argument('--apply', action='store_true', help='Otherwise inspect only') parser.add_argument('--backup', type=Path, help='Required new backup image path with --apply') args = parser.parse_args() path = image_path(args.image) require(args.size_gib > 0, 'Size must be positive') require(not args.apply or args.backup is not None, '--apply requires --backup PATH') check_idle(path) dev = attach(path, readonly=True) try: sector, size = device_info(dev) with open(dev.replace('/dev/disk', '/dev/rdisk'), 'rb', buffering=0) as f: h, parts, plan = inspect(f.fileno(), sector, size) recovery_hash = digest(f.fileno(), plan['source']*sector, plan['sectors']*sector) if args.apply else None target = args.size_gib * 1024**3 require(target >= size, 'Refusing to shrink image') recovery_bytes = plan['sectors']*sector require(target == size or target-size > recovery_bytes, 'Growth must exceed recovery size for non-overlapping relocation') print(json.dumps(dict(image=str(path), old_bytes=size, target_bytes=target, recovery_bytes=recovery_bytes), indent=2), flush=True) finally: run('diskutil', 'eject', dev) if not args.apply: return backup = args.backup.absolute() require(not backup.exists() and not backup.is_symlink(), 'Backup path must not exist') backup.parent.mkdir(parents=True, exist_ok=True) check_idle(path) # Native APFS clone; -n prevents accidental overwrite. Fail if cloning unavailable. run('/bin/cp', '-c', '-p', '-n', path, backup) require(backup.is_file() and backup.stat().st_size == path.stat().st_size, 'Backup verification failed') print('Rollback image:', backup, flush=True) if target > size: run('diskutil', 'image', 'resize', '--image-only', '--size', str(target), path) dev = attach(path) try: sector, actual = device_info(dev) require(actual == target, 'Image capacity differs from target') run('diskutil', 'unmountDisk', dev) with open(dev.replace('/dev/disk', '/dev/rdisk'), 'r+b', buffering=0) as f: plan = relocate(f.fileno(), sector, actual) require(digest(f.fileno(), plan['destination']*sector, plan['sectors']*sector) == recovery_hash, 'Recovery changed during image resize; restore backup before proceeding') print('Original Recovery SHA-256 preserved:', recovery_hash, flush=True) finally: run('diskutil', 'eject', dev) # Reattach so the kernel re-reads GPT; no Linux ioctl. dev = attach(path) try: run('diskutil', 'apfs', 'resizeContainer', dev+'s'+str(plan['main_index']), '0') run('diskutil', 'verifyDisk', dev) with open(dev.replace('/dev/disk', '/dev/rdisk'), 'rb', buffering=0) as f: _, final_parts, final_plan = inspect(f.fileno(), sector, actual) main_part = next(p for p in final_parts if p['type'] == APFS) require(main_part['end']+1 == final_plan['source'], 'Main APFS did not fill available space') run('diskutil', 'list', dev) finally: run('diskutil', 'eject', dev) print('Completed. Backup retained at', backup, flush=True) if __name__ == '__main__': try: main() except (ValueError, OSError, subprocess.CalledProcessError) as exc: print('ERROR:', exc, file=sys.stderr) sys.exit(1) Usage Shut down the VM and do not start it while the script runs. Pass either the .utm bundle or its writable disk image. Inspection is the default: python3 resize_utm.py '/path/macOS.utm' --size-gib 200 Apply the change, creating a new mandatory image backup first: python3 resize_utm.py '/path/macOS.utm' --size-gib 200 --apply --backup '/path/macOS-before.img' The size is GiB, so 200 means 214,748,364,800 bytes (Disk Utility displays about 214.7 GB). An initial 120 GiB image therefore gains 80 GiB. The main APFS container is smaller than the disk because ISC and outer Recovery retain their space. What it does Resolves the single writable image from an Apple-backend UTM bundle, or accepts an image directly. Rejects physical devices, images currently open by another process, shrinking, and unexpected partition layouts. Attaches read-only for inspection. Validates both GPT headers and partition arrays, their CRCs, partition bounds/order, and Recovery's APFS superblock geometry. With --apply, hashes the entire Recovery partition and creates a native APFS clone backup of the image. Uses diskutil image resize --image-only to grow the image. On a tested macOS 27 build, this already relocates Recovery for ASIF images. If needed, relocates Recovery itself through the attached raw device. It derives usable space from actual device capacity and GPT array dimensions, preserves APFS block alignment, copies and SHA-256-verifies Recovery before updating GPT, and writes both GPT copies with new CRCs. Partition IDs, names, attributes, and unrelated entries are preserved. Checks Recovery against its original full SHA-256, detaches/reattaches to refresh the kernel's partition map, and expands the main APFS container using diskutil apfs resizeContainer ... 0. Verifies the partition map and checks the final GPT layout fills the space up to Recovery. APFS's resize operation also performs a filesystem consistency check. Only exactly three outer partitions, in physical order ISC / APFS / Apple_APFS_Recovery, are supported. Matching uses the Recovery type GUID, not a guessed name. Raw and ASIF image interpretation is delegated to macOS. The relocation core supports 512- and 4096-byte logical sectors. Growth smaller than or equal to Recovery's size is deliberately refused to avoid an overlapping copy. Already-grown images can be completed by requesting their existing capacity. Failure and rollback This is not a crash-atomic disk transaction. A crash during GPT commit can leave its copies inconsistent; validation then fails closed. The script does not infer missing Recovery data from an NXSB signature, and does not attempt to repair a partially executed third-party resizer. The retained image backup is the rollback source. A copy interruption before GPT commit leaves the original Recovery and GPT intact, although unused destination space may have changed. If anything fails, leave the VM stopped. Eject any attachment listed in the log before restoring. Replace the modified image with a copy of the retained backup at the exact original image path. Do not restore only the old GPT after APFS growth. An encrypted/locked APFS guest may require unlocking its Data volume before APFS expansion; the script reports the native error rather than asking for or recording passwords. Do not launch UTM or attach the image in another app during the run; the initial open-file check is not an exclusive lock against other applications. References The workflow is based on the UTM discussion and yunyang088's Go resizer. The discussion documents the stale-GPT-end bug, fixed here by computing geometry from actual device capacity. This implementation takes a different interruption strategy: verified nonoverlapping copy before GPT publication, with a mandatory whole-image backup for rollback.