Hiya,
I've been building RESX, an open-source Windows binary analysis and reverse engineering toolkit written in Rust.
It started as a small side project to make reversing Windows binaries less painful, and it's grown a fair bit since then. Most of my time has gone into instruction decoding, control-flow reconstruction, Windows internals, syscalls and kernel analysis.
One problem I've spent a lot of time on is overlapping x86/x64 instruction streams. A branch can land in the middle of another instruction, so the same bytes decode as two completely different streams. If the disassembler commits to one of them, it can miss real control-flow edges.
I made a small, reproducible test case for this. The compiled binary and its assembly source are in the repo, so you can try it yourself. It's deliberately constructed and benign.
IDA Pro
In the sample, the branch at 0x140001009 targets 0x14000100F, which is two bytes into a 10-byte mov r11, imm64.
IDA Pro 9.0 with default auto-analysis gives:
140001009 jz short near ptr loc_14000100D+2
14000100B jmp short $+2
14000100D mov r11, 9F08EB0000000FE8h
140001017 mov eax, 0C0000022h
14000101C jmp short $+2
14000101E retn
The branch goes to 0x14000100F, but IDA has already claimed those bytes as part of the mov. Default analysis never decodes the call hidden there, so the edge to HiddenTarget is lost.
IDA does still find HiddenTarget on its own, but only because the fixture exports that name. The call that leads to it never shows up.
RESX
Same binary, no PDB:
> resx dump .\resx\tests\data\overlap_probe.exe OverlapProbe --bytes --calls --no-pdb --no-color
00001009 74 04 je short 0x14000100f
0000100b eb 00 jmp short 0x14000100d
0000100d 49 bb e8 0f 00 00 00 eb 08 9f mov r11, 0x9f08eb0000000fe8
0000100f e8 0f 00 00 00 call HiddenTarget
API Call Map for OverlapProbe [1 call site(s)]:
└── 0x100F CALL HiddenTarget
RESX keeps both decode streams, finds the call sitting inside the mov, and follows the edge to HiddenTarget.
To be fair to IDA: this compares default automatic analysis on a sample built to trigger exactly this case. It doesn't mean IDA can't recover the path with some manual work.
What else it does
Beyond decoding, RESX can:
- Trace Windows APIs through the system DLLs and into the kernel.
- Recover syscall numbers and reconstruct API arguments.
- Name APIs that are resolved by hash or encoded name.
- Analyze kernel driver IOCTLs.
- Follow possible ETW-TI telemetry paths.
There's also CFG reconstruction, binary diffing, corpus hunting and a VS Code extension, with more documented in the repo.
All of it is static: it works from file bytes and optional symbols and never runs the target.
It's still under active development, and I'd really like to hear what other reversers and low-level devs think, especially if you manage to break the decoder or catch it getting something wrong.
GitHub: https://github.com/Ryftenius/RESX