To study you should try challenges yourself as well as read writeups about how people approached solving these types of problems.
Write ups from previous competitions:
Defcon CTF: https://www.defcon.org/html/links/dc-ctf.html
Brute force hash collision:
PHP and Linux
Now some XXS:
Shellcode and Linux.
ssh email@example.com -p 7771
Exploit a programs memory
ssh firstname.lastname@example.org port 22
I found there is mail running and stuff in log files. Cypher has some mail.