ProsVJoes: Failure at The Warped Wall

This post is part 2 of a two part series. The 1st part is here.

In American Ninja Warrior, based on the original Japanese show Sasuke, there’s an obstacle called the warped wall, and if you’ve ever seen an episode of these shows, it’s an extremely familiar challenge. A wall that goes vertical and then curves slightly back toward itself, like a wave. The challenger must run up the wall, get a hold on the lip, and pull themselves up. It’s not an easy challenge, and thus it has become one of the most widely recognized parts of the Ninja Warrior Challenge. It was the final challenge of the first stage, representing the one last thing you had to do to earn a spot in the next round.

I am no Ninja; I legit tried to hang on a monkey bar a few days ago and immediately and persistently regretted it for the next few hours. “We used to just zip across these when we were kids!” I exclaimed to my wife as she made the same sound for me that she does when she sees a really old and sad dog on Instagram.

Pictured: My Wife watching me realize we are getting old and that time's arrow neither stands still nor reverses; it merely marches forward.

Anyway, if you read my previous blog, you know that I really hyped myself up for ProsVJoes by talking about leveling up and not talking myself out of attempting something beyond my current ability. Well I’m here to tell you that I was completely right.

This was way above my ability level, and I really did not do very well.

Overview

There were four Joes teams. Each team started with about five linux boxes and 5 windows boxes. There were a variety of versions, but most of the Linux machines were Ubuntu 16 with something running on it. Our job was to ensure the boxes stayed open and available. They would check for availability using dns and through two ports per machine, usually 22 and 80. Throughout the competition, more boxes would be added to our responsibilities, and we would need to go in and ensure they could stay “up”.

Day 1 was all defense. I encountered a few times where a random script would run saying “I think it’s time we take a break” and then in 420 seconds, the machine would restart. Another time, someone was typing while I was trying to change a password. Essentially, the red team was in our business, and we wanted them out. Day 2 was a purple day; we could attack other Blue teams as well as defend against the red team. There were clear rules about what was in and out of scope, and we got fed, too! Hooray!

Notice how everyone else is working and I'm posing for the camera. Notice, also, the red red red boxes in the third column on the screen. Yup, that's us.

With that quick overview out of the way, here’s some more detailed notes about my experience.

Day 1

So, when I first applied to play, I understood the term Joes to mean, “Regular folks with little to no experience trying something new/unfamiliar.” Turns out, Joes means “Not an active professional hacker,” as many of the people on my team were working in some capacity with security/computers. I was, by a long distance, the least experienced person there. Reminder: I learned Linux in March, and opened my first HackTheBox module in April. It seemed everyone knew what to do and how to find where the red team had placed beacons/ inserted malicious scripts/ messed around in our boxes. I did not know what to do. In fact, I spent a lot of time asking what I should do, and had a hard time understanding the responses.

"Just kill the process and then reboot the old kernel with a new sudo command you found in the user's logs. But don't kill the 2nd kernal and make sure you check in etc/bin/norm for any leftover users that don't have permissions for your task. Got it?"

There was a massive gap between me and the rest of the team. Or rather, a huge wall in front of me, warping back and threatening to collapse right on top of me and suck me into the undertow of competition. I felt lost the whole first day, drowning in misunderstood commands from others and confusion about who was working where. Eventually I settled into trying to change default passwords and attempting to upgrade Ubuntu to at least v20 from where they started at v16 (which if I remember correctly, is the version of Ubuntu they used in the American Revolutionary War).

The above machine is optimized for Ubuntu 16. I'm talking about the telephone.

That was my job at that point. Change and document the new password and then update the box. However, I started running into problems with the second part. The update process takes a long long long long time. And during that process, the red team had placed some things that would stop the process, pause it, or override it, and we’d lose all our progress. I was trying to update several boxes all at once, keep an eye on it to make sure to respond to installation prompts, and once in a while they will just stop being able to be upgraded. And I don’t even know where to start to try and Google the problem.

So I’d just say “I couldn’t upgrade here because of the red team” and kind of move on. The rest of the team was busy doing their work they’d settled into, and sometimes a teammate would try to help me, but it was chaotic, and multiple people needed help, so I settled back and just kept trying to find a way to contribute. I had to leave early for a work event and meeting around 4, but I was there for around 7 hours the first day.

Since the boxes were pre-compromised, you have to assume that even the usernames and passwords you were given are compromised. Eventually, I settled on a process that seemed to be working better than just updating first.

Change default password. Document new password.
Add new user `julian` (Our team mascot was King Julian from Madagascar)
Add julian to the sudo user group.
Switch user to julian
Apt upgrade and apt update and do-release-upgrade.
Keep an eye on it while updating.

At first, I was only working in our given user account. But later, making a new sudo account seemed to be helping us a little by keeping our work in an account the red team didn’t presumably have access to.

Day 2

Day 2 was largely the same, I took some machines that hadn’t been touched yet by our team and would apply the previous day’s process to it. But I started seeing new error messages, new issues, and none of them were easily Googleable. I kept getting stuck with no ability to move forward, and ended up at another stalemate with my brain. I had to leave early again for work stuff, this time at 2 pm. The work stuff I left for ended up taking way longer than I knew it would, and I was gone for the rest of the day. So, unfortunately, day 2 was very short for me and I did very little. I was able to get one of our boxes online at one point with an update, but it went down shortly after. Still, that was a nice minor victory in what was an incredibly frustrating experience.

"Hey team! The machine is up oh wait never mind..."

Now, to be extremely clear, I am very happy I did this event. I met some very cool people, and got to do something pretty unique. Regardless of my ability level, I am glad I did it. But my frustration simply cannot be overstated in this article; I was dead stuck and floundering for any sort of success but found very little. Of course, this was all by design and means the event was crafted correctly by its designers.

“Every day it gets a little easier. But you gotta do it every day. That's the hard part."

I love learning. The ability to do something you were previously unable to do is an astounding and wonderful part of being alive. The feeling of accomplishment when completing something new is absolutely unparalleled in anyone’s life. The issue is, it’s not always sudden. Often, you move in such small degrees that you can’t see the full breadth of what you’ve just done until after you’ve done it.

Throughout the event, I found myself whispering under my breath about not understanding, or not knowing what to do, or even just kind of cursing at myself for not knowing something. But looking back now a week later, I’m thinking about the things I was able to do that I didn’t know before.

That process earlier? Changing passwords and updating usernames and adding them to the sudo list? I didn’t know how to do that before. And the concept and reason behind doing it was even more of a foreign a concept to me before last week. I had gotten better. But in a kind of small way. On top of that, I was able to automatically pull out Naabu to scan and see if a port was open or not and to possibly see what was running on that port blocking it.. And when someone asked, “What’s usually on port 80?” I knew it was HTTP. And that port 22 was SSH.

I knew these things, and didn’t know I knew them. Until ProsVJoes.

And I know these are obvious things, and that they are basic pieces of knowledge. And it’s not anything I’m bragging about. It’s more that this event was the moment that I REALIZED that I’d learned them. This was “wax on, wax off” turning into a block right in front of me. I know now that I can build off of this.

I know you were hoping for an original Karate Kid gif. Deal with it.

That’s what’s hard. These moments of realization don’t come often, and most days you feel like nothing has changed. Going into the competition, and during most of it, I felt like I knew nothing. And luckily, that turned out to be untrue. I knew some, and there is much more to learn, but I am able to learn; so it’s not an impossible task.

My Bad, guys.

One of the things that made this harder was I missed two meetings with my Joes team. They were organized a little last minute (usually 1 or 2 days before the meeting happened) but nevertheless, I did miss those meetings. I overestimated just how much headspace moving from Florida to California would take (weird how uprooting your home of 6 years and moving it all 3,000 miles to a city where you do not yet have an apartment takes a mental toll. Who knew?) and thus was a bit out of it. I was learning online, but not like I used to, and not with any sort of true fidelity, and I put that responsibility on my own shoulders. Had I been in a more stable environment, I might have been able to do a bit better and feel more prepared.

Ok, so I didn't prepare well. But even at the event, I was distracted and overly busy. I left early twice, I was worried about a demo at DEF CON later in the week and an event ProjectDiscovery were putting on as well. My brain was cluttered and I was physically and mentally distracted. Knowing that, and knowing the format of the CTF, means that I can better prepare for next year. I can throw myself into the competition more completely. Plus, I’ll be more prepared as I continue to learn more about hacking.

Final Thoughts

I started learning about hacking in March; there’s not many people who would go to their first CTF 5 months after installing Linux for the first time and come out feeling like they were amazing. And I'm no exception.

I don’t feel amazing, I feel frustrated by my lack of knowledge, but also inspired by the success I realized I did have. It was an exposure to a world that I knew very little about and now know a little more about. Small steps lead to big changes. I might not have gotten up the wall this time, and the gap might seem insurmountable from where I stand now, but I know for a fact that when I do this next year, I will still feel nervous, but I won’t feel helpless.

There will always be a warped wall in my way and there will be times I fail to climb. BsidesLV PRosVJoes was one such time; I am sitting on the edge after failure watching others succeed and move on. But if I’m earnest, ambitious, and honest with myself, then there will come a time when I can scale the warped wall and find a new challenge waiting beyond it. And maybe I'll fail at that.

But I'll still be on top of that damn wall.

Thanks

Thanks to SpikeRoche for leading my Blue team, and thanks to all my other team members for your patience and kindness as an absolute rookie attempted something brand new. I’ll be back and will be a better teammate and more experienced player.

Thank you Brendan O’Leary for telling me to give it a shot and consistently being a kind and earnest supporter of my education/career change since you first came into my Twitch chat back in 2021. Thank you to the entire ProjectDiscovery team who never once doubted me; for all your encouragement and excitement I am extremely grateful.

Thanks to BsidesLV for hosting the event and for the excellent vibes and organization. The CTF was fun and I enjoyed my time inside the hall. It’s no small feat to organize an event, and to have had a successful one at that.

Thank you to the Gold Team for overseeing the entire CTF. Red Team… thank you, too. Without your absolute soul crushing attack, there would be nothing to defend against.

Thank you, you, for reading this! If you want to watch me learn more you can check me out on Twitch. You can also join our Discord where you can ask questions I am slowly learning the answers to.