Hello, I'm 8ayac 🐝

This post looks back at preparing and running ISCCTF 2020, held on Saturday, October 24, 2020.

ISCCTF 2020 at a glance

  • Event schedule
    • October 24, 2020 (Sat), 10:00–23:00 JST
  • Participants
    • Registered users: 135
    • Users who scored at least one point: 84
  • Intended audience: Students new to CTFs or at a beginner level
  • Organizer: IPFactory
  • Challenge archive and official writeups: https://github.com/IPFactory/ISCCTF2020

How the event came about

ISCCTF 2020 was one of the events in the online school festival at 情報科学専門学校.

Schedule

The CTF was held on October 24, 2020 (Sat), to fit the school festival's schedule. There were about six weeks between deciding to hold it and the event itself. The time available for the work was shorter, though, and some concerns remained unresolved when the event began.

Intended difficulty

We aimed the CTF at students at the school who were new to CTFs or still at a beginner level. Someone who could solve B-level problems in ABC and use Linux commands should have been able to solve at least two or three challenges. For comparison, we aimed slightly below the difficulty of SECCON beginners CTF 2020.

Although the intended audience was beginners at the school, we planned from the start to allow participants from outside as well. There were two main reasons:

  • We expected a small turnout if we kept the event private
  • We wanted feedback from experienced participants outside the school

We thought a very small turnout would make the event feel quiet. We also expected more participants to give us more insight into issues that arose and the load on the servers.

Organizers and responsibilities

The following members of the IPFactory club shared the work of organizing the CTF. Thank you all for your help.

  • @8ayac: Writing Web challenges, reviewing Pwn/Rev/Misc challenges, infrastructure other than the challenge servers, writing rules and tweets, project management, coordination with the school, and other work
  • @a_zara_n: Reviewing Web challenges and infrastructure for the challenge servers
  • @Drumato: Reviewing Rev challenges
  • @iria_piyo: Writing Forensics challenges
  • @scgajge12: Reviewing Web challenges
  • @n01e0: Writing all Pwn and Rev challenges, and Misc challenges
  • @odgrso: Reviewing Web challenges
  • @ry0kvn: Writing and reviewing Forensics challenges
  • @wa1m3im: Writing Web challenges
  • @y0d3n: Writing Web challenges
  • @yky4n4g1: Writing Misc challenges

CTFs we referred to

We drew in particular on these CTFs:

  • TSGCTF
  • InterKosenCTF

Among recently held CTFs, they had good reputations for both their challenges and infrastructure.

Their retrospectives and archives were also available, which was another reason to refer to them. We looked at participant writeups and archives from earlier years as well as 2020.

We also referred to SECCON beginners CTF 2020 to get a sense of the difficulty appropriate for beginners.

Creating the challenges

Resources for challenge authors and reviewers

I recommended that the challenge authors and reviewers read the following resources:

Review checklist

Every challenge in ISCCTF 2020 was reviewed at least once.

We used the following as a minimum review checklist:

  • Does the challenge require guessing what the author had in mind?
  • Are there unintended solutions?
  • Can participants break the challenge?
  • Does it place an excessive load on the server?
  • Are there obstacles unrelated to the substance of the challenge?
  • Have uninteresting elements been removed as far as possible?

The checklist was based on icchy's CTFの問題を作るときに気をつけること and hakatashi's TSG CTF 開催記. We selected the items to keep the checklist from becoming too broad.

The PPC challenge

At first, we planned to offer the PPC challenge maze in the Misc category. As the event drew closer, I reconsidered whether it fit the purpose of a CTF. With the agreement of its author, @yky4n4g1, we offered it as a zero-point bonus challenge.

Feedback on the challenges

The survey we opened near the end of the competition received the following ratings:

Challenge ratings, with higher numbers indicating better ratings

We also received this response to the question, "Do you have any comments on the challenges?"

None of the challenges required mind-reading or felt poorly designed. Every category had varied, interesting challenges, and I enjoyed them. Although the event was advertised as a CTF for newcomers, I think both beginners and experienced players could enjoy it.

Infrastructure

We used GCP for ISCCTF 2020's infrastructure. There were no particular selection criteria, but being able to refer to TSGCTF's hosting retrospective was useful, and my familiarity with GCP compared with other cloud services was a major factor. In the end, the infrastructure ran without major trouble.

Score server

The score server ran CTFd on an Ubuntu VM in GCE (Google Compute Engine). I used the official docker-compose.yml almost as it was. For Nginx, I used the installation on Ubuntu rather than running it in a Docker container, so I could set up HTTPS before starting CTFd.

Load balancing across multiple servers (which didn't happen)

The score server ran on a single machine throughout. I hadn't studied the setup enough and assumed I could get load balancing done quickly once the challenge work had settled down.

When that work began to settle down, I looked into load balancing and learned that it would involve creating instance templates, consolidating CTFd's database, and other changes.

There were only two days left before the event, so I abandoned the move to a multi-server setup rather than risk losing data that had already been registered.

Keeping the server up anyway

I gave up on load balancing across multiple servers, but took the following measures to keep the score server up:

  • Changed the instance's machine type to e2-standard-16
  • Set the number of worker processes started by Nginx/gunicorn to eight

With little experience running Web services or CTFs, I gave up early on trying to find a neat solution and settled for throwing resources at keeping the server up. It's not much of a success story.

Load testing

Before settling on the measures above, I ran several load tests. I used Apache JMeter for these tests. Here too, I was learning by following examples.

The following image shows the final load test against the score server. It indicates a capacity of around 400 requests per second for simple GET requests to the home page.

Load test results with requests per second (rps) outlined in red

Monitoring

Monitoring server access and load

I used GCP's Monitoring features for all server-access and load monitoring. I added graphs like these to a dashboard and monitored it.

A dashboard created with GCP's Monitoring features

For detecting unusual traffic resembling a DoS attack, I followed TSGCTF's hosting retrospective. During the CTF, I observed the unusual traffic shown below.

A spike in the graph where unusually high request traffic resembled a DoS attack

This traffic continued for about an hour. We had a way to ban users, but I was on monitoring duty and missed it while working on something else. The server had plenty of spare capacity, so response times should have been unaffected. Still, I should have caught it.

Monitoring whether challenges could still be solved

I also automated some of the checks for whether challenges were still solvable. The code was put together three days before the event, so it was a rush job, but I've made it available.

https://github.com/IPFactory/ISCCTF2020_SolvabilityInspector

Below is an overview of SolvabilityInspector's structure.

SolvabilityInspector's class diagram

A solver for each challenge can be added to the scheduled checks. The base Inspector provides the shared logging and Discord message-sending functions.

I also added a feature to send the monitoring logs to a specified Discord channel. I remembered the idea while rereading InterKosenCTF's hosting retrospective the day before the event, and implemented it then.

The following image shows logs posted on Discord during the competition.

Logs from the challenge solvability checks

When a challenge cannot be solved as expected, the monitor mentions a designated Discord role. The notification also includes information to help distinguish a connection failure from a challenge that is reachable but cannot be solved. None of the challenges became unsolvable during the event, so this notification was never needed. The image below is from a test run.

Example logs when a challenge could not be solved

For one challenge, I checked only connectivity rather than solvability. Solving this challenge required creating files on the challenge server, with no way to delete those files. I therefore limited monitoring to connectivity checks so that files created by the monitoring solver would not give participants hints. When monitoring whether a challenge is solvable, it's worth checking that the solver doesn't leave hints for participants.

Making the work more efficient

GitHub

We wrote and reviewed the challenges on GitHub.

Challenge review on GitHub, first example Challenge review on GitHub, second example

GitHub let us develop the challenges in a team workflow:

  1. Create a branch named <category>-<challenge-name>
  2. Once the challenge is ready, open a pull request
  3. Assign the challenge reviewer to the pull request
  4. The reviewer reviews the challenge
    1. If no changes are needed, the author merges it into master, and the work is done
    2. If changes are needed, push the revisions, request another review, and return to step 4

For most of the process, this kept master aligned with the challenges we intended to use in the event, which made the work easier to follow.

GitHub's per-user repository permissions also helped keep the challenge content from leaking. We considered sharing through HackMD or secret Gists, but chose GitHub to avoid exposing the challenges if a URL became visible. Some organizers often shared their screens in online classes or study groups, so I thought that protection was necessary.

After the event, it was also easy to show what we'd been doing1 to club members who hadn't taken part in writing the challenges.

Google Sheets

We used Google Sheets to track each challenge's progress and the ports it used. Each challenge had a separate host, though, so we ultimately didn't need to manage the port assignments.

A spreadsheet for tracking challenge development

Google Apps Script recorded the time of each status change automatically.

For each challenge's flag, we used data validation to check that it matched the format specified in the rules.

A warning displayed by the data validation feature

We didn't get as far as automating progress updates in the spreadsheet, so it often fell out of sync with the actual work.

Making the CTF enjoyable

Sharing solve logs

Following InterKosenCTF 2020's example, we posted updates on Discord showing who had solved each challenge.

A bot congratulating participants on Discord for solving challenges

I made the following changes to CTFd's source code for this feature:

diff --git a/CTFd/api/v1/challenges.py b/CTFd/api/v1/challenges.py
index f24e38d..adfa315 100644
--- a/CTFd/api/v1/challenges.py
+++ b/CTFd/api/v1/challenges.py
@@ -1,6 +1,8 @@
+import os
 import datetime
 from typing import List

+import requests
 from flask import abort, render_template, request, url_for
 from flask_restx import Namespace, Resource
 from sqlalchemy.sql import and_
@@ -579,6 +581,10 @@ class ChallengeAttempt(Resource):
                     )
                     clear_standings()

+                discord_webhook_url = os.getenv('DISCORD_WEBHOOK_URL')
+                discord_log_msg = f'Conguratulations! `{user.name}` solved `[{challenge.category}]{challenge.name}` :tada:'
+                requests.post(discord_webhook_url, data={'content': discord_log_msg})
+
                 log(
                     "submissions",
                     "[{date}] {name} submitted {submission} on {challenge_id} with kpm {kpm} [CORRECT]",

Liking participants' tweets on Twitter

On Twitter, I searched for tweets about ISCCTF 2020 and liked them from IPFactory's account. The idea was that a response from the organizers could help people feel involved in the event and enjoy it a little more.

This part was manual. I created a search-results column in TweetDeck and liked tweets that appeared there. Putting those users into a list would also let me find tweets that didn't match the search query, so that's what I'd do next time.

Other reflections

Forgetting CTFd's notifications at first

I forgot until partway through the event that CTFd's Notifications feature let me send announcements to all participants through the score server. For a while, announcements therefore went out only on Discord. I should have posted them on the score server, which all participants had to use, and then also announced them on Discord.

If announcements go out on multiple platforms, automating them might be worthwhile.

Looking back

Apart from a mini CTF2 I had previously run as part of a class for students at the school, this was my first time organizing a CTF, and I enjoyed it.

I wrote about the challenges I created in a separate writeup post3.

Footnotes

1 Until the competition ended, IPFactory members could view the CTF development repository only if they were part of the ISCCTF 2020 organizing team.