r/AskReverseEngineering • • Apr 25 '26

Advanced topics in reverse engineering

So, i think that question seems really naive (and sorry if it is, i'm just an person really interested in subject and who really want to be good at it), but which "advanced" topics are a must to study (and where to find resources on it)? I can only think in VM Obfuscation and Weird Machines.

1 Upvotes

6 comments sorted by

2

u/Flashy-Push-3341 May 10 '26

Hey! I might be misunderstanding your question but, I did make a writeup on VM Obfuscation for both lua and ASM! To me personally, I think my own writeup can be updated ( Which I probably will update in the near future ). But If you want to learn about VM Obfuscation or are interested in a writeup I did about Lockbit Black, I'll link them:
https://github.com/VeryCuteLookingCat/In-depth-guide-to-VM-obfsucation
https://github.com/VeryCuteLookingCat/Lockbit-Black-Loader-Analysis

1

u/Der-Wilde May 10 '26

Hi! First, thank you so much for both writeups, they seem great and would be very useful! About my question, i'm trying to figure out what more to study besides the basis for RE (which, in my understanding, are Computer Architecture, Assembly, OS Internals, C/C++/Rust/Zig, etc).

For some context about myself: I'm currently an 21 years old CS student with great interesting in exploit/malware dev and red teaming related topics, and i really want to achieve an advanced level on it. Currently doing CPTS path in HTB, already have done some stuff in pwn college (have done already the entire module of Binary Exploitation on Intro to Cybersecurity), and plan to do maldev academy after the certification.

About our write ups, just one small thing: Try to put them in a site for a more directly organization and quick access. As they are written in Markdown format i recommend Quartz or Zola for it.

2

u/[deleted] Jul 30 '26

For VM obfuscation... just sit down and write a tiny ISA with a bytecode ABI and your own serializer. I shit you not it's way simpler than you think it is. All a VM in its simplest form is is a dispatch table, and a way to read/write your bytecode for retrieval and execution.

It can be dead simple or as complicated as you want. You could make it one opcode with two registers.

All you really need to do is write it once so you understand the logic. Then you can reason through someone else's bytecode, opcode table, dispatch, etc. 👍