Designing Nice Challenges for Cybersecurity Competitions

Not too long ago, the Division of Homeland Safety (DHS) recognized the necessity to encourage hands-on studying by way of cybersecurity competitions to deal with a scarcity of expert cyber defenders. Likewise, in 2019, Govt Order 13870 addressed the necessity to determine, problem, and reward the US authorities’s greatest cybersecurity practitioners and groups throughout offensive and defensive cybersecurity disciplines. Nicely-developed cybersecurity competitions supply a method for presidency organizations to meet that order.

The Software program Engineering Institute (SEI) has been working with the DHS Cybersecurity & Infrastructure Safety Company (CISA) to deliver distinctive cybersecurity challenges to the federal cyber workforce. This weblog put up highlights the SEI’s expertise creating cybersecurity challenges for the President’s Cup Cybersecurity Competitors and general-purpose tips and greatest practices for creating efficient challenges. It additionally discusses instruments the SEI has developed and made freely accessible to assist the event of cybersecurity challenges. The SEI technical report Problem Growth Pointers for Cybersecurity Competitions explores these concepts in larger element.

The Goal and Worth of Cybersecurity Challenges

Cybersecurity challenges are the center of cybersecurity competitions. They supply the hands-on duties rivals carry out as a part of the competitors. Cybersecurity challenges can take a number of varieties and may contain totally different responses, reminiscent of performing actions on one or many digital machines (VM), analyzing numerous varieties of information or information, or writing code. A single cybersecurity competitors may comprise a number of totally different challenges.

The purpose of those cybersecurity challenges is to show or assess cybersecurity abilities by way of hands-on workouts. Consequently, when constructing challenges, builders choose mission-critical work roles and duties from the Nationwide Initiative for Cybersecurity Schooling Workforce Framework for Cybersecurity (NICE Framework), a doc revealed by the Nationwide Institute of Requirements in Know-how (NIST) and the Nationwide Initiative for Cybersecurity Careers and Research (NICCS). The NICE Framework defines 52 work roles and offers detailed details about the precise information, abilities, and skills (KSAs) required to carry out duties in every.

Utilizing the NICE Framework helps builders focus challenges on important abilities that greatest characterize the cybersecurity workforce. Every problem clearly states which NICE work position and duties it targets. By figuring out the information and abilities every problem targets, rivals can simply give attention to challenges that handle their strengths throughout the competitors and isolate studying alternatives when utilizing challenges for coaching.

Problem Planning

Creating profitable cybersecurity challenges begins with complete planning to find out the extent of issue for every problem, assessing the factors accessible for every problem, and figuring out the instruments required to unravel the challenges. When it comes to issue, competitors organizers need members to really feel engaged and challenged. Challenges which might be too simple will make extra superior members lose curiosity, and challenges which might be too onerous will frustrate rivals. Competitions usually ought to embody challenges which might be appropriate for all ranges—newbie, intermediate, and superior.


Factors techniques are used to reward rivals for the effort and time they spend fixing every problem. Furthermore, competitors organizers can use factors to find out competitor placement—rivals with larger scores can advance to future rounds, and organizers can acknowledge these with the best factors as winners. Factors ought to be commensurate with the issue posed by the problem and energy required to unravel it. Level allocation is usually a subjective course of, a matter we are going to return to within the part Problem Testing and Evaluation part under.

Problem Tooling

Figuring out the instruments required to unravel a problem is a crucial step within the growth course of for 2 causes:

  • It ensures that problem builders set up all required instruments within the problem atmosphere.
  • It’s good follow to offer rivals an inventory instruments accessible within the problem atmosphere, particularly for competitions by which organizers present rivals with the evaluation atmosphere.

Builders ought to be cautious to construct challenges that don’t require using paid or licensed software program. Open supply or free instruments, functions, and working techniques are important as a result of some rivals may not have entry to sure software program licenses, which might put them at a drawback and even forestall them from finishing altogether.

Problem Growth

Builders should be well-versed in cybersecurity subject material to plot modern approaches to check rivals. Not solely should builders determine the abilities the problem will goal and the situation it would simulate, they have to additionally develop the technical facets of the problem, implement an automatic and auditable grading system, incorporate variability, and write documentation for each the testers and the rivals.

Pre-Growth Concerns

Builders ought to start by figuring out the work roles and abilities their problem goals to evaluate. By so doing, they’ll construct extra exact challenges and keep away from together with duties that don’t assess relevant abilities or that check too vast an array of abilities. After they’ve outlined the work position related to a given problem, builders can kind a problem concept.

The problem concept contains the technical duties rivals should full and the placement by which the problem situation will happen. All problem duties ought to resemble the duties that professionals undertake as a part of their jobs. Builders are free to be as inventive as they want when constructing the situation. Topical challenges primarily based on real-world cybersecurity occasions supply one other method so as to add distinctive and inventive situations to challenges.

Technical Element Concerns

The technical elements of problem growth usually contain VM, community, and repair configuration. This configuration ensures the problem atmosphere deploys accurately when rivals try the problem. Growth of technical elements may embody:

  • Configuring VMs or providers to include identified vulnerabilities
  • Configuring routers, firewalls, providers, and many others., to the state builders need
  • Staging assault artifacts or proof all through networks or logs
  • Finishing different actions that put together the atmosphere for the problem

Builders may also purposefully misconfigure facets of the atmosphere if the problem targets figuring out and fixing misconfigurations.

Finest Practices for Creating Challenges

Every problem targets totally different abilities, so there isn’t a commonplace course of for creating a cybersecurity problem. Nevertheless, builders ought to apply the next greatest practices:

  • Make sure the technical abilities assessed by the problem are relevant in the true world.
  • Make sure the instruments required to unravel the problem are free to make use of and accessible to the rivals.
  • Make an inventory of the instruments accessible to rivals within the hosted atmosphere.
  • Guarantee challenges don’t power rivals down a single answer path. Opponents ought to be capable to clear up challenges in any lifelike method.
  • Take away pointless hints or shortcuts from the problem, together with command historical past, searching information, and different information that would permit rivals a shortcut to fixing the problem.

Problem Grading

Basically, builders ought to automate grading by way of an authoritative server that receives solutions from the rivals and determines what number of factors to award the submission. The submission system ought to usually ignore variations in capitalization, white house, particular characters, and different variations which might be finally irrelevant to correctness. Doing so ensures rivals aren’t unfairly penalized for immaterial errors.

Ignoring these errors may appear to contradict an evaluation of operational readiness in circumstances the place precise precision is required. Nevertheless, cybersecurity competitions have objectives and concerns past evaluating operational proficiency, reminiscent of making certain a good competitors and inspiring broad participation.

Builders could apply totally different grading strategies, together with the next:

  • Token discovery. In token-discovery grading, rivals should discover a string or token that follows an outlined format (these tokens may also be referred to as “flags”). Builders can place the token in any a part of the problem the place the competitor will discover it by finishing the problem duties.
  • Query-and-answer issues. For question-and-answer issues, the competitor should discover the right reply to a number of questions by performing problem duties. The solutions to the problem questions can take a number of varieties, reminiscent of coming into file paths, IP addresses, hostnames, usernames, or different fields and codecs which might be clearly outlined.
  • Setting verification. In atmosphere verification grading, the system grades rivals primarily based on adjustments they make to the problem atmosphere. Challenges can activity rivals with fixing a misconfiguration, mitigating a vulnerability, attacking a service, or some other exercise the place success will be measured dynamically. When the grading system verifies adjustments to the atmosphere state, it offers rivals with successful token.

Problem Variation

Builders ought to embody some degree of variation between totally different deployments of a problem to permit for various appropriate solutions to the identical problem. Doing so is vital for 2 causes. First, it helps promote a good competitors by discouraging rivals from sharing solutions. Second, it permits competitors organizers to reuse challenges with out dropping instructional worth. Challenges that may be accomplished quite a few instances with out leading to the identical reply allow rivals to be taught and hone their abilities by way of repeated follow of the identical problem.

Builders can introduce variation into challenges in a number of methods, relying on the kind of grading that they use:

  • Token-based variation. Challenges utilizing token-discovery or environment-verification grading can randomly generate distinctive tokens for every competitor when the problem is deployed. Builders can insert dynamically generated submission tokens into the problem atmosphere (e.g., inserting guestinfo variables into VMs), and so they can copy them to the areas the place they anticipate rivals to obtain the problem solutions.
  • Query-and-answer variation. In question-and-answer challenges, builders can introduce variation by configuring totally different solutions to the identical questions or by asking totally different questions.

Problem Documentation

The 2 key paperwork builders should create in assist of their problem are the problem information and the answer information.

The problem information, which is seen to the rivals, offers a brief description of the problem, the abilities and duties the problem assesses, the situation and any background info that’s required to grasp the atmosphere, machine credentials, and the submission space or areas.

The problem doc ought to describe the situation in a method that rivals can simply comply with and perceive. The problem situation and background info ought to keep away from logical leaps and the issue degree mustn’t hinge on info internationally overlooked of the information.

The answer information offers a walk-through of 1 method to full the problem. Throughout testing, builders use the answer information to make sure the problem will be solved. Builders may also launch the answer information to the general public after the conclusion of the competitors to function a group studying useful resource.

The supposed viewers for this information is the overall cybersecurity group. Consequently, builders ought to assume the reader is accustomed to fundamental IT and cybersecurity abilities, however shouldn’t be an skilled within the discipline. Screenshots and different pictures are useful additions to those guides.

Problem Testing and Evaluation

After builders construct a problem, it ought to undergo a number of rounds of testing and evaluate. Builders check challenges to make sure high quality, and so they evaluate them to estimate the problem’s issue.

Builders ought to carry out an preliminary spherical of testing to catch any errors that come up throughout the problem deployment and initialization course of. They need to additionally be sure that rivals can absolutely clear up the problem in a minimum of a technique. A second spherical of testing ought to be carried out by certified technical workers unfamiliar with the problem. Testers ought to be inspired to aim fixing the problem on their very own however could also be supplied the developer’s answer information for assist.

The testers ought to guarantee every problem meets the next high quality assurance standards:

  • The problem deploys as anticipated and with out errors.
  • The problem VMs are accessible.
  • The problem is solvable.
  • There are not any unintentional shortcuts to fixing the problem.
  • Problem directions and questions are correctly formatted and provides a transparent indication of what rivals should do.

Of their evaluate of the problem, testers ought to take notes in regards to the content material, together with estimates of issue and size of time it could take rivals to unravel. After testers full their evaluate, competitors organizers can look at the issue assessments and examine every problem with others. This comparability ensures that simpler challenges stay in earlier rounds and are value fewer factors than challenges judged as harder.

When deciding problem level allocations, organizers can use a base or commonplace rating allotment as a place to begin (e.g., all challenges are value 1,000 factors firstly of the method). Organizers can then improve or lower level allocations primarily based on the accessible issue information, protecting in thoughts that the primary purpose is for the variety of factors they allocate to a problem to immediately correspond with the hassle required for fixing it. Level allocations ought to think about each the issue and the time it takes to unravel the problem.

SEI Open Supply Purposes for Cybersecurity Problem Competitions

Builders can use a number of open supply functions to develop challenges and to orchestrate cybersecurity competitions. The SEI has developed the next two functions for operating cybersecurity competitions:

  • TopoMojo is an open supply lab builder and participant software that builders can use to develop cybersecurity challenges. It offers digital workspaces by which problem growth can happen. The workspaces permit builders so as to add VMs, digital networks, and some other assets which might be required for creating or fixing a single problem.
  • Gameboard is an open supply software that organizers can use for orchestrating cybersecurity competitions. It permits organizers to create competitions that may both be staff or particular person primarily based and that include both single or a number of rounds. Challenges are organized into rounds and rivals try to unravel as many challenges as they’ll to maximise their rating. Gameboard makes use of the TopoMojo API to deploy the rivals’ recreation house for every problem.

Gameboard additionally serves because the authoritative location for rivals to submit solutions or tokens. Furthermore, as a part of dealing with reply and token submissions, Gameboard has logging, brute power protections, and different options to make sure the integrity of the competitors.

Determine 1 reveals how the TopoMojo and Gameboard functions work together. Builders use TopoMojo workspaces to develop challenges. Opponents then use Gameboard to deploy and in- teract with challenges. When a participant deploys a problem, Gameboard will work together with the To- poMojo API to request a brand new recreation house for the competitor. TopoMojo creates and returns the participant’s problem recreation house.

Finest Practices Help Higher Cybersecurity Competitions

The event practices now we have highlighted on this put up are the results of the SEI’s expertise creating cybersecurity challenges for the President’s Cup Cybersecurity Competitors. Cybersecurity competitions present a enjoyable and attention-grabbing method to train technical abilities, determine and acknowledge cybersecurity expertise, and interact college students and professionals within the discipline. They will additionally function training and coaching alternatives. With the US authorities, and the nation as an entire, dealing with a big scarcity within the cybersecurity workforce, cybersecurity competitions play an vital position in creating and increasing the workforce pipeline.

There is no such thing as a single method to run a contest, and there’s no one method to develop cybersecurity challenges. Nevertheless, these greatest practices might help builders make sure the challenges they create are efficient and interesting. Problem growth is the one most vital and time-consuming side of operating a cybersecurity competitors. It requires meticulous planning, technical growth, and a rigorous quality-assurance course of. In our expertise, these practices guarantee efficiently executed competitions and enduring, hands-on cybersecurity belongings that competitors organizers and others can reuse many instances over.

If you want to be taught extra in regards to the work we do to strengthen the cybersecurity workforce and the instruments now we have developed to assist this mission, contact us at

Related Articles


Please enter your comment!
Please enter your name here

Latest Articles