Applied Cryptanalysis

August 31, 2020 by Howard Poston

Making use of simple cryptanalysis

Most modern encryption algorithms are secure, meaning that no known attacks exist that can enable a ciphertext to be decrypted without knowledge of the secret key. However, this doesn’t mean that knowledge of simple cryptanalytic techniques are useless.

While cryptography “done right” is largely unbreakable, not everyone uses cryptography properly. If a broken encryption algorithm is used or encoding is used in place of encryption, then an understanding of cryptography and cryptanalysis can be very useful in the real world.

Applications of cryptanalysis in cybersecurity

Cryptography is only secure if implemented properly. If loopholes exist or an insecure algorithm is used, then it is only good for obfuscating data, not protecting it against unauthorized exposure.

TLS decryption

Transport Layer Security (TLS) is the protocol used to protect data traveling over the network. It is capable of using a wide variety of different combinations of asymmetric and symmetric encryption algorithms and hash algorithms to accomplish this task.

In the past, TLS/SSL relied on RSA keys for encryption, making it easy to decrypt traffic if the RSA private key was compromised. Modern implementations use algorithms that offer perfect forward secrecy, meaning that they incorporate unique, random values into the encryption of each session.

However, this does not make it impossible to decrypt TLS traffic in Wireshark or similar tools. It just makes it necessary to have access to the client’s browser or the server at the time of communication. By setting the SSLKEYLOGFILE environment variable on the client, browsers can be configured to store the ephemeral data used to secure each connection. Access to this file enables decryption of the TLS traffic.

Malware command and control

Malware commonly uses encryption and encoding to protect its command-and-control (C2) traffic from being detected and inspected. Access to malware’s C2 traffic could enable incident responders to identify the existence and scope of a malware infection and potentially force the malware to uninstall itself from the affected computer.

While malware can use strong encryption to protect its C2 traffic, not all malware variants do. It is not uncommon for malware to use encoding or broken encryption algorithms to obfuscate their traffic and make reverse engineering more difficult and time-consuming.

The image above shows a traffic capture from a malware variant that is an example of this. The body of the response (in blue) shows data that has been obfuscated using Base64 encoding (as indicated by the character set and equal sign padding).

Decoding this data produces the results shown above. While this is not human-readable text, certain attributes of it (such as the use of an @ followed by two capital letters throughout) suggest that this is the malware’s actual C2 protocol, rather than a properly encrypted ciphertext.

The use of encoding for obfuscation can slow analysis and defeat simple signature and string matching. However, it does not use a secret key, making it easily reversible if the algorithm is identified.

Encrypted files

Simple encryption algorithms are also used to hide the contents of malicious files. For example, downloader or dropper malware may request an encrypted file over the network or carry one embedded as data within it. Once it reaches its destination — and has successfully passed the organization’s cybersecurity scanners — the malicious file is downloaded and extracted.

The image above is an example of a network traffic capture of the C2 traffic of a downloader. Note that the majority of the response contains the sequence mlvr.

This is an example of the use of a one-time pad or XOR cipher with a very short, repeated string. The repeated sequences of mlvr exist because some files have large sections of NULL padding, which reveal the encryption key.

After extracting the file from the traffic and decrypting it – by XORing it with the repeated key mlvr — the file above is created. As shown, this is a Windows executable file that the malware would then execute on the target machine.

In this case, the malware used weak encryption to download an executable file. However, the same technique could be used for downloading configuration files or exfiltrating sensitive data. Unless the corporate security solutions know how to identify and decrypt XOR encryption, this would enable the file to bypass simple string matching and signature detection.

Applying cryptanalytic techniques

Done properly, cryptography is secure, but cryptography isn’t always implemented correctly. The large number of free decryptors on the No More Ransom web page demonstrates that, in some cases, mistakes can be made that undermine the security of an encryption algorithm.

Additionally, some cybercriminals will use weak encryption or encoding algorithms for obfuscation in their malware. In this case, it may be possible to deobfuscate and read the original data.



  1. Hybrid Analysis
  2. Malware Traffic Analysis
  3. The No More Ransom Project
Posted: August 31, 2020
Howard Poston
View Profile

Howard Poston is a cybersecurity researcher with a background in blockchain, cryptography and malware analysis. He has a master's degree in Cyber Operations from the Air Force Institute of Technology and two years of experience in cybersecurity research and development at Sandia National Labs. He currently works as a freelance consultant providing training and content creation for cyber and blockchain security.