Task
A compiled, statically-linked, stripped, PIE Linux ELF service listened on 127.0.0.1:5555. It executed custom 64-bit virtual machine bytecode: a 9-opcode ISA operating over 8 registers (R0–R7) and a 512-byte (64-qword) VM stack buffer. Before execution, a static range verifier (verify_bytecode) checked register value bounds. The service was hardened with ASLR, PIE, NX, stack canaries, and full RELRO. Goal: craft bytecode that bypasses the verifier and triggers SYS_UNLOCK_VAULT with the correct vault key.
ISA summary: OP_MOV_IMM, OP_MOV_REG, OP_ADD, OP_SUB, OP_MUL, OP_XOR, OP_LOAD_STACK, OP_STORE_STACK, OP_SYSCALL — with SYS_WRITE_VAL and SYS_UNLOCK_VAULT as system calls.
Approach
1. Binary Disassembly
Disassembled the stripped ELF to recover the VM dispatch loop, opcode handlers, verifier logic, and vault key derivation routine — all without symbols.
2. Verifier Vulnerability
The static range verifier tracked register value bounds through the instruction sequence to reject programs that could cause out-of-bounds stack access. Identified a case where the verifier's bound tracking failed to account for a specific arithmetic sequence — allowing crafted instructions to pass verification while producing a runtime stack index that exceeded the 64-qword buffer.
3. Vault Key Derivation
Traced the SYS_UNLOCK_VAULT handler to understand the runtime key derivation — computed from the service's execution context. Derived the expected key value and constructed bytecode that loads it into the correct register before invoking the syscall.
4. Exploit Construction
Crafted a bytecode payload that: (1) passes the static verifier, (2) loads the derived vault key into R[src], (3) calls SYS_UNLOCK_VAULT — triggering flag decryption and delivery.
Key Techniques
ELF reverse engineering, custom VM ISA analysis, static analysis bypass, vault key derivation from binary, exploit construction under modern mitigations (ASLR/PIE/NX/canaries/RELRO)
Environment
Isolated Docker environment. Binary bound to loopback only. Exploit validated by service returning the decrypted flag.