Prologue: This is just my brief discussion about this challenge. I was not able to solve this challenge during the contest (I upsolved it later after doing some research and reading other player writeups)
Basically the solution was about to analyze the logic of the serial checker. There are several techniques implemented in the binary such as Nanomites, Dynamic API Resolution, API Hashing, and many other small anti-debugging and anti-disassembly tricks. I will not dive into how to reverse this binary. My main focus is the obfuscation itself and other interesting anti reverse engineering techniques
Before dive into this challenge, I really appreciate Fatmike for creating such an amazing challenge
First of all the binary is stripped, so it is hard to spot the correct entry of the encryption routine
This program implements a minimalist GUI with an OK button. Moreover, there is a dialog for typing input. So we can place a breakpoint at GetDlgItemTextA or GetDlgItemTextW, and by inspecting the call stack we can find the checker function
By using HashDB to look up the CRC32 hash value, we can determine that it is VirtualProtect
So the function at 04046C8h changes the memory protection attribute of the pc section to PAGE_EXECUTE_READWRITE
All of the important APIs are dynamically resolved. Fortunately the list is tiny, so we can manually patch it.
There is a struct used throughout execution which is initialized at 403DD2h
The function at 4046C8h is used to raise a breakpoint exception
Alright, that is all we can explore while navigating around the binary. Nothing is special to inspect anymore
But if you try to recover the hashed API, you will find something really interesting. The challenge uses KiUserExceptionDispatcher which is an NTDLL native API, to implement some stuff
Let’s talk more about this API. What I have known is that this function will resolve and dispatch the exception based on the priority, for example from Windows VEH (Vector Exception Handler through AddVectoredExceptionHandler) to SEH (Structured Exception Handler through __try __catch stuff). If none of them are registered, the program will crash.
So back to the binary, by examining all the cross-reference assosiated with the KiUserExceptionDispatcher function located at 040187A, you can find this interesting function
There is an anti-debugging technique in the overwrite_NtQueryInformationProcess function using the ProcessDebugPort option. We can manually patch this to bypass.
So if no debugger exists, the program will trigger the overwrite_the_API function which is also a really interesting one
First of all, this function uses VirtualProtect to change the memory protection of ntdll text section to PAGE_EXECUTE_READWRITE. After that, the program uses a technique called inline hooking to create an indirect trampoline led to the custom structured exception handler.
The author uses an assembly trick which is PUSH-RET to craft the trampoline. The byte 0x68 and 0x3C can be translated to PUSH and RET respectively, so the epilouge of the function looks like
Sorry for the inconvenience that I would not dive into how I’m able to reverse this part, but in general, I was using x32dbg to debug, watching the memmory at runtime and guessing the function variables properties and rename them. Although, there is still some part that I didn’t understand, the challenge is totally solvable
I would analyze this first, those other exception handlers are not much different
int__thiscallexception_breakpoint_handler(obj*this,_CONTEXT*context){_CONTEXT*context_1;// edi
_CONTEXT*context_2;// [esp-Ch] [ebp-10h]
char*Eip;// [esp-8h] [ebp-Ch]
if(check_something((_BYTE*)this->sixth)){inc_rip_n_flush((_CONTEXT*)&context);set_ctxflag((_CONTEXT*)&context);}else{context_1=context;if(check_eip_in_pc(this,context->Eip)){calc_next_eip(this,context_1);Eip=(char*)context->Eip;context_2=context;this->EIP=(int)Eip;decrypt_next_eip(this,context_2,Eip);}}return-1;}
So in the set_ctxflag function, it resets some register and activates the Trap Flag for the single step exception.
calc_next_eip looks up the hard-coded mapping table to determine which jcc instruction each int 3 instruction should be translated to (the nanomites github has a deeper explaination)
The decrypt_next_eip function decrypts the next 0x10 bytes from the next EIP in the .pc section. It uses a custom ChaCha20 constant. The key is located at 0x0040D264 + 4, it is a SHA256 hash value of the whole .text section. This can be called an anti-tampering technique so that any software breakpoints or modifications like patching to the binary will break the accuracy of the key. Therefore, we can attach the program to the debugger later to extract the key safely
bool__stdcallcheck(inta1,_CONTEXT*context){DWORDEFlags;// ecx
boolresult;// al
charv4;// dl
DWORDv5;// eax
EFlags=context->EFlags;switch(*(_DWORD*)(a1+4)){case1:EFlags>>=6;gotoLABEL_12;case2:return1;case3:EFlags>>=7;gotoLABEL_3;case4:gotoLABEL_12;case5:v4=1;if((EFlags&0x40)==0&&(((unsigned__int8)(EFlags>>7)^(unsigned__int8)(context->EFlags>>11))&1)==0){return0;}returnv4;case6:return(EFlags&0x41)==0;case7:LOBYTE(v5)=~(unsigned__int8)(EFlags>>11);return((EFlags>>7)^v5)&1;case8:EFlags>>=2;gotoLABEL_3;case9:EFlags>>=11;gotoLABEL_3;case0xA:return(EFlags&0x41)!=0;case0xB:returncontext->Ecx==0;case0xC:EFlags>>=2;gotoLABEL_12;case0xD:EFlags>>=6;gotoLABEL_3;case0xE:v5=EFlags>>11;return((EFlags>>7)^v5)&1;case0xF:EFlags>>=7;gotoLABEL_12;case0x10:EFlags>>=11;LABEL_12:LOBYTE(EFlags)=~(_BYTE)EFlags;gotoLABEL_3;case0x11:v4=1;if((EFlags&0x40)!=0||(((unsigned__int8)(EFlags>>7)^(unsigned__int8)(context->EFlags>>11))&1)!=0){return0;}returnv4;case0x12:LABEL_3:result=EFlags&1;break;default:result=0;break;}returnresult;}
The logic is not that hard, it is a little bit lengthy. For example, the first one simulates the JNE/JNZ instruction
You do not have to remember these signatures, just googling them
This is what you get after reversing the function, given in the format (name, short jump, near jump)
So int 3 will be replaced by one of these jcc instructions. So how does it change? Well remember the
hard-coded mapping table that I mentioned? You can dump these values by looking into the mapping table initialization located at 4045A7h.
The call instruction at 4045DAh is the STL map insert function. Basically, it uses the STL map to store and query data, but we just need data. So the solution was first set up a breakpoint at 4045DAh and then follow in dump the value stored in EDI
So the data is given in the (address offset, type, jump offset, instruction size) format which is
Another small technique used in this binary is anti-disassembly, it looks like this
1
2
3
jmploc+1loc:// some really meaningless instruction go here
The idea is simple, disassemblers often translate bytecode into assembly in order. So if we insert a junk byte between two instructions and somehow make the runtime skip it (or else we can get crashed). The disassemblers like IDA still translate and get confused. And the solution to make CPU skip that junk byte is the jump instruction. In order to fix dodge this, we can modify all junk bytes into nop opcode
So basically challenge’s execution flow is:
By chaining all the pieces, you can deobfuscate this amazing challenge. Now is the play of hashing algorithm
This is where the 5 hard-coded constant be initialized
This function divides the 19 bytes of the serial into 5 different chunks, each chunk has length of 4 bytes. Then it calculates the hash of each character consecutively based on the calc_next_hash function. Then if all chunks are matched, the serial checker is valid, our flag will be displayed
We don’t have to understand what the calc_next_hash does, we can just manually simulate it in the script by looking real quick through the implementation. Then start attacking the hash, this is my solve script
#!/home/ryou/.venvs/rev/bin/pythonimportstructimportstringdefcalc_next_hash(hash_val,character):ifnotisinstance(character,int):raiseValueError("Character is not an integer value")hash_val&=0xFFFFFFFFcharacter&=0xFFifhash_val&1:if(character^hash_val)<=0x80000000:ifnot(97<=character<=122):ifnot(65<=character<=90):value=(9*hash_val)&0xFFFFFFFFelse:value=(character+hash_val)&0xFFFFFFFFifvalue&0x100:value^=0x13371337else:value=(hash_val-character)&0xFFFFFFFFelse:value=(hash_val-0x21524111)&0xFFFFFFFFifcharacter>0x60:value^=(33*character)&0xFFFFFFFFelifcharacter>=64:ifcharacter%2:value=(((hash_val<<27)&0xFFFFFFFF)|(hash_val>>5))^2271560481else:value=((hash_val>>29)|((hash_val<<3)&0xFFFFFFFF))+0x12345678value&=0xFFFFFFFFelifcharacter&2:value=(character+(hash_val^0x55AA55AA))&0xFFFFFFFFelse:value=(hash_val-16*character)&0xFFFFFFFFifcharacter==48:value|=0xF0F0F0F0foriinrange((character%5)+2):ifnot(value&0x80000000):hash_valueb=(value<<1)&0xFFFFFFFFelse:hash_valueb=((value<<1)&0xFFFFFFFF)^0x04C11DB7ifi%2:value=(hash_valueb+10*i)&0xFFFFFFFFelse:value=character^hash_valuebif((value^(value>>8)^(value>>16)^(value>>24))&0xF)>7:return(~value)&0xFFFFFFFFifvalue==0:return0xBADF00Dreturnvaluecharset=b'0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz-_'defhash_crack(hash_init,target,serial_length):foraincharset:forbincharset:forcincharset:hash=hash_inithash=calc_next_hash(hash,a)hash=calc_next_hash(hash,b)hash=calc_next_hash(hash,c)ifserial_length==3:ifhash==target:returnchr(a)+chr(b)+chr(c)else:fordincharset:ifcalc_next_hash(hash,d)==target:returnchr(a)+chr(b)+chr(c)+chr(d)return"Error"result_set=[0x865DBB47,0xA6EB190,0x20476C33,0x1C8A7693,0x59FEBDFB]hash=0xCAFEBABEfinal_serial=""forchunkinrange(5):serial=hash_crack(hash,result_set[chunk],3ifchunk==4else4)print(f"Founded {chunk}'th serial part {serial}!")final_serial+=serialhash=(0x112233*(chunk+1)*4-0x35014542)&0xFFFFFFFFprint(final_serial)
#!/home/ryou/.venvs/rev/bin/python# key = bytearray(open("keydump.bin", "rb").read())# print(len(key))# print(' '.join(hex(i) for i in key))importstructjcc={1:('JNE',b'\x75',b'\x0F\x85'),2:('JMP',b'\xEB',b'\xE9'),3:('JS',b'\x78',b'\x0F\x88'),4:('JNC',b'\x73',b'\x0F\x83'),5:('JLE',b'\x7E',b'\x0F\x8E'),6:('JA',b'\x77',b'\x0F\x87'),7:('JGE',b'\x7D',b'\x0F\x8D'),8:('JP',b'\x7A',b'\x0F\x8A'),9:('JO',b'\x70',b'\x0F\x80'),10:('JBE',b'\x76',b'\x0F\x86'),11:('JECXZ',b'\xE3',None),12:('JNP',b'\x7B',b'\x0F\x8B'),13:('JE',b'\x74',b'\x0F\x84'),14:('JL',b'\x7C',b'\x0F\x8C'),15:('JNS',b'\x79',b'\x0F\x89'),16:('JNO',b'\x71',b'\x0F\x81'),17:('JG',b'\x7F',b'\x0F\x8F'),18:('JC',b'\x72',b'\x0F\x82')}raw_map_data=bytearray(open("mapdump.bin","rb").read())map_data=[struct.unpack("<IIII",raw_map_data[i:i+16])foriinrange(0,len(raw_map_data),16)]eip=0decrypted_pc=bytearray.fromhex(open("decrypted_pc",'r').read())size=len(decrypted_pc)patched_byte=[]fori,opcodeinenumerate(decrypted_pc):ifi<eip:continueifopcode!=0xCC:patched_byte.append(opcode)continuebase,cond,off,sz=map_data[i]mnem,short,near=jcc[cond]ifsz==0x2:insn=short+struct.pack("<B",off&0xFF)else:insn=near+struct.pack("<L",off&0xFFFFFFFF)patched_byte+=list(insn)eip=i+sz
There is a problem in this nanomites deobfuscator. There is a special case that cause an incorrect translation, for example
So basically the issue is exactly similar to the anti-assembly I mentioned above. It is not always a good choice to decrypt a 0xcc because it could not even be executed during the actual runtime. So if we decrypt every 0xcc it can affect the nearby bytecode and mess up so many things. For example, if the 0xcc located at 0000 is decoded into jmp 0x3, continuing to decode the following 0xcc could break the byte at 0003 and ruin the program.
I really appreciate the contribution of Fatmike in making this such an amazing challenge, it has a very educational meaning for reverse engineer in particular and all binary analyst in general
I’m also give an enormous respect to the community as well as some individuals because they have provided a great explanation and writeup about this challenge. I learnt a lots while reading your guys’ blog. Thanks again!
If you notice any misleading informations in my blog, you could contact me! Have a good day while reversing ^_^