Hackathons in computer science education: monitoring and evaluation of programming projects

There are several ways inmaking learning activities more engaging and interesting to computer science students. In addition to traditional higher education (curriculum, project-based approaches, lectures, labs), we consider competitive approaches such as hackathons to develop hard and soft skills. Computer science education needs to change requirements for hard and soft skills. In order hard skills university CS education has to capture the changes behind the trends, such as big data, artificial intelligence, cloud computing, etc. Developing soft skills is important teamwork, end-user awareness, collaboration, etc. This study aims to present the various ways to implement hackathons in the context of CS education. We also present a taxonomy of hackathons based on our experiences and observations from 2016 to the present. We aim to share our lessons learned on the following issues: (1) How can hackathons be designed in CS education to teach students necessary skills and competencies; (2) what kinds of programming projects monitoring and evaluation we need during hackathons.


Introduction
Nowadays time-based events, such as hackathons, code camps, or code fests, are used to learn and teach industrial software development using intensive collaboration to complete a project. Like teaching programming and software development courses, time-based events (particularly, hackathons) can be considered as successful learning tools. Hackathons have been adopted not only in computer science, software engineering education [2,12,18,22,[27][28][29] but in enterprises [9,30], corporations [23], virtual communities [3,16,25] and others. Most hackathons are organized with specific goals to create some innovative technology or product.
Briscoe and Mulligan [4] describes a hackathon as a digital innovation that can be used to teach computer science concepts. Falk, Kannabiran and Hansen [7], Kollwitz and Dinter [17], Medina Angarita and Nolte [19], Nolte et al. [24] discuss the important aspects of hackathon organization, an approach to the business environment involving experts from companies.
The topic of hackathons is reflected in the papers of Falk Olesen and Halskov [8], Frey and Luks [10], Gama et al. [11], Medina Angarita and Nolte [20], Pe-Than et al. [26]. Nolte, Chounta and Herbsleb [23] note the importance of hackathons as an educational environment for the formation of students' skills in developing software products as a course project. Cobham et al. [6], Imam and Dey [15] describe a methodology for involving stakeholders (universities, IT companies, students, volunteers, etc.) to participate in the hackathon. Byrne, Sullivan and O'Sullivan [5], Guerrero et al. [13] present possible forms of organizing hackathons, and also describe the advantages of hackathons. Abduramanov, Ibraimov and Seydametov [1]investigated the automation of tools for tracking the results of hackathon participants, as well as checking the work of students. Mtsweni and Abdullah [21] analyze the factors influencing the motivation and professional interest of computer science students. Horton et al. [14], Uys [32] describe teamwork methods, as well as important aspects of agile application development methodology that need to be taught to students.
Despite the popularity of hackathons, there are no more researches and studies that have the answers to the following questions: what kind of approach to organizing hackathons has a greater pedagogical effect (for example, short-term or long-term event), what stakeholders should be involved, what are the practical differences in organizing hackathons. To study this problem, we use such methods as a literature review, own experience in organizing time-based events, surveys of hackathon participants (students), mentors, and representatives of the IT industry.

Objectives and taxonomy of hackathons
The universities' computer science and software engineering departments need to redevelop their teaching approaches because IT industries are rapidly changing and developing the requirements for both hard and soft skills [31,33]. In terms of skills, curriculum guidelines such as IT professional and educational standards, ACM, and IEEE Curricula (both in Computer Science and Software Engineering) aim to accommodate the increasingly interdisciplinary nature of computer science and software development.
Hard skills are the skills that are easy to observe, measure, and demonstrate -for example, these are programming skills, calculating skills, or swimming skills. Without hard skills, people cannot effectively engage in a certain type of activity. Soft skills are more about communication skills. Soft skills are difficult to measure, but these are the skills that are most effective in demonstrating and applying hard skills. Soft skills are essential in any kind of activity. In terms of soft skills, the emphasis is shifting towards teamwork, end-user awareness, and understanding of the software application development context.
Often, universities and IT industry representatives see the importance of hard and soft skills differently. Learning is becoming more interactive, that is why the importance of collaborative learning and teamwork is increasing. At the same time, hackathon-like activities that teach rapid team-based software development are becoming increasingly popular for teaching computer science and software development. Such approaches can help to teach students to collaborate in teams, to help each other in achieving learning goals. Communication and collaboration have become important factors to consider when teaching students and are essential skills in the 21st century.
Hackathons are project-based and work-integrated events that address real-world problems or challenges posed by IT industry representatives.
Events (hackathons) aimed at learning through interaction in a common project may differ in duration, forms of implementation, and participants coverage. Table 1 shows the classification (taxonomy) of Hackathons. Hackathons can be different in duration, objectives, goals, stakeholders, and achieved results. Understanding own strengths and weaknesses in programming.

Specialization in IT
Acquaintance with software development technologies.

Universities, educators, students
Knowledge of the technology and the acquisition of relevant skills. Hackathon as an exam or a capstone project Testing and evaluating the skills using a real project environment.

Universities, educators, students
Knowledge and skills that every student should have.

Competition
Innovative approaches and solutions to the given problem and the implementation.

Universities, educators, students, mentors
Ability to work with a real problem, innovative opportunities, implementation, participation in competitions.

Industry hackathon
Results presented: product and solutions on a company-defined topic using special tools and methodologies. Idea generation and diffusion Students, IT industry Ability to work in real conditions and the company's environment.

Hackathon model
One of the hackathon goals is to deepen and expand the professional competencies of computer science or software engineering students. The approach to solving this problem is to apply the existing theoretical knowledge and practical skills of students in real professional activity. This approach can be realized through the flow of students in the framework of the educational process of various types of practices and the involvement of professional software developers to contact students outside of the educational process.
Hackathons were designed as time-based events that help programmers and software developers collaborate on dome software ideas or projects. As we showed in the previous section, hackathons can last from one day up to a month. Mostly the motivation behind hackathons can be connected with educational or social goals.
Hackathon as a time-based and work-integrated event includes multiple desirable features, for example, stakeholders and students can face-to-face communicate and interact, a better understanding of how to work together in different teams.
We asked during the survey about the advantages and disadvantages of hackathons. The answers you can see in table 2. Table 2 describes some of the advantages and disadvantages that hackathon participants can expect.   We got the following answers to the question "An important quality of a Hackathon participant?": knowledge and skills in software development -61,9 % of participants, knowledge of domain area -28,6 %, teamwork skills -81 %, ability to work in stressful conditions -71,4 %, extraordinary thinking -50 %, other (flexibility, testing skills, etc.) -2,4 %. Results on answering the question "Do you plan to further participate in Hackathons (in different roles) or assist in hackathons organization?" were "yes, of course" -69,1 %, "maybe" -23,8 %, "no" -7,1 %.
From the results of the survey described below, the hackathon model is presented in figure 3. The purpose of the model is to capture key elements that are essential toward stimulating and maintaining students' interest in computer science and software engineering hackathons.

Guidelines for organizing of the hackathon
After questionary and based on the experience of hackathons organizers it is important to consider the following guidelines for organizing the hackathons: allocate a specific location for the event and allow participants to fully focus on their tasks during the hackathon with minimal distraction due to external factors. Moreover, areas of relaxation and stress relief are desirable, this is useful for maintaining a creative atmosphere. It is also advisable to ensure that all key players: participants, mentors, organizers, experts, judges are available to check consistency throughout the process. Physical space and resources are essential to a successful collaboration.

Timely information and support of the participants.
Hackathons are usually very tedious, so organizers should take steps to support the participants so that they remain focused, active, and motivated. Discussions and presentations should be limited in time. It is useful to have a well-planned and uncomplicated schedule so that the work on the project develops gradually and progressively. A presentation on the project at the end of each stage and a final presentation at the end of the event will provide an opportunity for a wider group of people (stakeholders, experts, participants) to see the results of the work. events. This creates difficulties in organizing the hackathon but allows you to extract real value from the event.

Teamwork.
Meeting and working closely with other team members involved in the project is beneficial for all involved and is a very productive way to achieve results. At the end of the event, all participants have a better understanding of the problems and tasks they were trying to solve. During the hackathon, there is also a significant exchange of knowledge and technical information between participants, team members. In addition, the hackathon provides an opportunity to work through complex technical problems with mentors, experts, software development industry with diverse backgrounds and technical skills.

5.
Flexibility. When organizing the hackathon and during the hackathon, it is important to find a balance between setting rigid goals and methods and fairly open and poorly defined objectives. This balance is difficult to achieve during a hackathon as it requires active collaboration. However, flexible formulation of recommendations and methodologies allows for an appropriate degree of flexibility. This approach allows you to combine goal achievement, allows to emerge new ideas, and also develops innovative thinking.
6. Size of a team. It has been found that the optimal size of the team is between 5 and 6 team members. If the team is too large, then subgroups naturally from within the team, and there is a danger that the subgroups will start to work separately.

Experts, judges.
At the end of each stage of the hackathon, a meeting is held in which non-team members participate (remotely or in-person) in a session to review and discuss project progress and possible next steps. Experts and the judges are allowed to ask questions or ask about specific points of interest in the projects.
8. Duration. It was found that the duration should be close to the maximum limit of effective intensive work. Long hackathons allow for more ambitious tasks than those considered during short-time hackathons (for example, a 24-hour hackathon or weekend). It is difficult to schedule and motivate a large group of participants for a long-time hackathon when they have other education activities or work responsibilities. Preparation for the hackathon should start 6 months before the event is scheduled. Figure 4 provides an overview of the timeline: Step 1 -planning documents, mentors' selection, software engineers working in the context of the event.
Step 2 -registration of the students, formation of the teams.
Step 3 -introduction to the event context, event's domain areas.
Step 4 -kick off the hackathon.
Hackathon activities should include the following stages of software development:

Preparation
• The study of the task and the subject area. Definition of the concept of the developed software product. Finding the answer to the question: "What should the product do?" • Market research and search for similar products. Highlight the killer feature of the product.
• Interviewing potential users to identify competitive advantages. Creating a project description.

Planning
• Forming and adding tasks to Project Management services.
• Study of the technology stack for the project and the selection of optimal solutions.
• Score tasks by the hour. Highlighting tasks for the MVP version.
• Forming tasks for increments and sprints using the Scrum methodology. Distribution of tasks between team members.

Programming and testing
• Initializing the project and creating a Git repository.

User Acceptance Testing
• Verification of the developed product by end-users for efficiency and convenience.
• Defining the list of tasks that require execution in the next versions of the product.

Presentation
• Presentation creation: project description (in one sentence); product functionality and killer features; problems solved by the product; potential users of the product; the stack of used technologies; plans for further product development.
• Preliminary (closed) presentation for mentors and jury members. Live demonstration of a workable project.
Completed projects (software products) of Hackathon are assessed by the jury according to the following criteria: • The quality of the development process is controlled during the project development process; • Design -estimated the beauty and style of the project; • Functionality -a set of the declared functions of the software product is checked, satisfying the requirements of the customer's technical specification and the needs of potential users; • Reliability -estimated completeness of the project, its fault tolerance, and recoverability.
• Security -properties ensuring data confidentiality, their integrity, and availability are checked; • Convenience -features of the software product are taken into account, which allows the user to easily work with data; • Efficiency -the ratio of the level of services provided by the software product to the user under specified conditions, to the volume of resources used is assessed; • Accompanying -estimated ease of analysis of the software, the ability to make changes to the software in connection with changing user needs; • Portability -adaptability to different hardware environments, installation flexibility is checked; • Q-factor -the value of the software product, reasonableness, and rationality of the used solutions and technologies are taken into account; • Presentation -the quality of the performance, demonstrating the developed software product, is evaluated.

Technical features of the hackathon and evaluation criteria
To monitor and evaluate the development of the Hackathon's assignments we developed a web-based accounting system. The architecture of this system is presented on the figure 5.
A web-based accounting system provides access (limited by category) in desktop and mobile versions to Hackathon's participants, organizers, judges, and mentors.
The system provides the following capabilities: 1) registration and authorization for participants, mentors, judges, etc.; 2) choice of roles in the system: participant, team mentor, administrator, etc.; 3) a description of the terms of reference for the participating team; 4) control and assessment of the passing of the stages of product development; 5) iterative the team and mentor cooperation (receiving and making edits at each stage of development); 6) tracking the status of teams (passed stages, the number of points received).
Discussing the quantitative and qualitative results, it should be noted that the final rating of evaluation by the jury members of the developed software products of the Hackathon was the sum of two components: 1. Rating evaluation of the development process of a software product by a team, taking into account in the web-based accounting system (the highest possible rating is 3000 points) (table 3).
2. Rating evaluation of the developed software jury (the maximum possible rating is 3000 points) (table 4).

Conclusions
Hackathons are growing in popularity in computer science education and encourage the practical ingenuity of putting together some inherent elements that are universally required, often via programming. The value of hackathons is in providing an opportunity for students to meet and create new links in the IT-industry medium. Hackathon is not only a prototyping exercise technically speaking, but it is similarly a prototyping exercise of new working and personal collaborations for the participants.
There are a lot of approaches to the implementation of hackathons in the context of computer science education. We presented the taxonomy of hackathons, the advantages and possible disadvantages of hackathons. Hackathons as pedagogical technology can be used to teach students the necessary skills and competencies for developing programming projects and software applications. Demonstration of the project 100 Table 4 Criteria for assessing the quality of the development process (jury).