diff --git a/Koli_2017/Koli_2017_Stanger.tex b/Koli_2017/Koli_2017_Stanger.tex index 28de681..8148457 100644 --- a/Koli_2017/Koli_2017_Stanger.tex +++ b/Koli_2017/Koli_2017_Stanger.tex @@ -111,7 +111,7 @@ Second, does the schema meet the requirements of the problem being solved? A database schema is normally designed and implemented within the context of a specific set of requirements, so verifying that the implemented SQL schema fulfills these requirements is an effective way to grade the implementation, and also provides a useful framework for providing feedback to students. The requirements for a database schema can be loosely divided into \emph{structure} (e.g., tables, columns, data types), \emph{integrity} (e.g., keys, constraints), and \emph{behavior} (e.g., sequences, triggers). -In this paper we describe a novel system that semi-automates the grading of students' SQL schema implementations. The system takes as input a machine-readable specification of the assessment requirements and a live instance of a submitted student schema, and verifies whether the schema conforms to the specification. Rather than attempt to parse and check the \texttt{CREATE} statements directly, the system verifies the structure of the schema by issuing queries against the schema's metadata (catalog) to look for expected values such as table and column names, etc. It verifies integrity constraints by attempting to insert known legal and illegal values. The system currently does not check behavioral aspects. The results of the tests are compared against the machine-readable specification. This process is effectively one of unit testing the schema using the specification as a framework. We use the PHPUnit database unit testing framework to carry out this process, albeit in a somewhat unorthodox way (see \cref{sec-design}). +In this paper we describe a novel system that semi-automates the grading of students' SQL schema implementations. The system takes as input a machine-readable specification of the assessment requirements and a live instance of a submitted student schema, and verifies whether the schema conforms to the specification. Rather than attempt to parse and check the \texttt{CREATE} statements directly, the system verifies the structure of the schema by issuing queries against the schema's metadata (catalog) to look for expected values such as table names, column names, etc. It verifies integrity constraints by attempting to insert known legal and illegal values. The system currently does not check behavioral aspects. The results of the tests are compared against the machine-readable specification. This process is effectively one of unit testing the schema using the specification as a framework. We use the PHPUnit database unit testing framework to carry out this process, albeit in a somewhat unorthodox way (see \cref{sec-design}). The remainder of the paper is structured as follows. In the next section we discuss related work, while \cref{sec-motivation} discusses the motivation and context for our approach. \Cref{sec-design} discusses the design of our system, and \cref{sec-evaluation} evaluates its impact. We discuss known issues and future work in \cref{sec-issues}, and conclude in \cref{sec-conclusion}. @@ -119,11 +119,11 @@ \section{Related work} \label{sec-literature} -There have been many prior efforts to build systems to support students learning SQL. Early examples such as RDBI \cite{Dietrich.S-1993a-An-educational} and esql \cite{Kearns.R-1997a-A-teaching} were essentially simple query execution environments that students could use to practice writing SQL queries. RDBI provided relatively little feedback about the correctness of a query, whereas esql could visualize the intermediate tables generated by each step of a query, enabling students to better understand the steps in its execution. +There have been many prior efforts to build systems to support students who are learning SQL. Early examples such as RDBI \cite{Dietrich.S-1993a-An-educational} and esql \cite{Kearns.R-1997a-A-teaching} were essentially just query execution environments that students could use to practice writing SQL queries. RDBI provided relatively little feedback about the correctness of a query, whereas esql could visualize the intermediate tables generated by each step of a query, enabling students to better understand the steps in its execution. -Later systems became more responsive in terms of the feedback they provided to students. Systems like SQL-Tutor \cite{Mitrovic.A-1998a-Learning}, SQLator \cite{Sadiq.S-2004a-SQLator}, AsseSQL \cite{Prior.J-2004a-Backwash}, ActiveSQL\footnote{\url{https://db.grussell.org/}} \cite{Russell.G-2004a-Improving,Russell.G-2005a-Online}, SQLify \cite{Dekeyser.S-2007a-Computer}, and aSQLg \cite{Kleiner.C-2013a-Automated} all provided varying levels of syntactic and semantic feedback. Many of these took a more ``functional'' approach to checking SQL query code, i.e., verifying that the code written fulfilled the requirements of the problem, rather than focusing on the code itself. This was most commonly done by checking how much the result set of a student's query differed from the correct result set (e.g., SQLator's ``equivalence engine''). ActiveSQL could also detect ``hard-coded'' queries that produced the desired result, but would fail if the data set changed \cite{Russell.G-2005a-Online}, while the SQL Exploratorium \cite{Brusilovsky.P-2010a-Learning} took an interesting alternative approach, using parameterized query templates to generate the questions given to students. +Later systems became more responsive in terms of the feedback they provided to students. Systems like SQL-Tutor \cite{Mitrovic.A-1998a-Learning}, SQLator \cite{Sadiq.S-2004a-SQLator}, AsseSQL \cite{Prior.J-2004a-Backwash}, ActiveSQL\footnote{\url{https://db.grussell.org/}} \cite{Russell.G-2004a-Improving,Russell.G-2005a-Online}, SQLify \cite{Dekeyser.S-2007a-Computer}, and aSQLg \cite{Kleiner.C-2013a-Automated} all provided varying levels of syntactic and semantic feedback. Many of these took a more ``functional'' approach to checking SQL query code, i.e., verifying that the code written fulfilled the requirements of the problem, rather than focusing on the code itself. This was most commonly done by checking how much the result set of a student's query differed from the correct result set (e.g., SQLator's ``equivalence engine'' \cite{Sadiq.S-2004a-SQLator}). ActiveSQL could also detect ``hard-coded'' queries that produced the desired result, but which would fail if the data set changed \cite{Russell.G-2005a-Online}. The SQL Exploratorium \cite{Brusilovsky.P-2010a-Learning} took an interesting alternative approach, using parameterized query templates to generate the questions given to students. -Given the relatively static nature of an SQL schema, we feel this style of verification approach is the most appropriate way of approaching an automated grading system for SQL DDL. As an aside, the concept of verifying a schema against its specification sounds like it should be a useful application of formal methods \cite{Spivey.J-1989a-An-introduction}, but work with formal methods and databases has historically focused either on \emph{generating} a valid schema from a specification (e.g., \cite{Vatanawood.W-2004a-Formal,Lukovic.I-2003a-Proceedings,Choppella.V-2006a-Constructing}), or on verifying schema transformation and evolution \cite{Bench-Capon.T-1998a-Report}. Certainly formal methods approaches could be used to generate the model answer or rubric for grading an assessment, but this is effectively the inverse of what is needed to grade a student-written schema. +Given the relatively static nature of an SQL schema, we feel this verification style of approach is the most appropriate way of constructing an automated grading system for SQL DDL. As an aside, the concept of verifying a schema against its specification sounds like it should be a useful application of formal methods \cite{Spivey.J-1989a-An-introduction}, but work with formal methods and databases has historically focused either on \emph{generating} a valid schema from a specification (e.g., \cite{Vatanawood.W-2004a-Formal,Lukovic.I-2003a-Proceedings,Choppella.V-2006a-Constructing}), or on verifying schema transformation and evolution \cite{Bench-Capon.T-1998a-Report}. A formal methods approach could certainly be used to generate the model answer or rubric for grading an assessment, but this is effectively the inverse of what is needed to grade a student-written schema. Previous systems focused almost exclusively on SQL \emph{queries} using the \texttt{SELECT} statement (i.e., DML) rather than SQL \emph{schema definitions} (DDL). This focus is unsurprising given that \texttt{SELECT} is probably the most frequently used of any SQL statement, and also because it is relatively complex compared to most other SQL statements. Only a few of systems we reviewed even mentioned schema definition. RDBI \cite{Dietrich.S-1993a-An-educational} supported a DDL, but only its own non-SQL language, while esql \cite{Kearns.R-1997a-A-teaching} simply passed anything that was not a \texttt{SELECT} statement through to the DBMS. \citeauthor{Gong.A-2015a-CS-121-Automation}'s ``CS 121 Automation Tool'' \cite{Gong.A-2015a-CS-121-Automation} focuses primarily on SQL DML statements, but appears to be extensible and could potentially be modified to support SQL DDL statements. @@ -133,13 +133,13 @@ Another branch of related work is systems that actively aid students in learning SQL, e.g., SQL-Tutor \cite{Mitrovic.A-1998a-Learning}, SQLator \cite{Sadiq.S-2004a-SQLator}, ActiveSQL \cite{Russell.G-2004a-Improving}, and aSQLg \cite{Kleiner.C-2013a-Automated}. SQL-Tutor is a typical example of this category. It was an intelligent teaching system that provided students with a guided discovery learning environment for SQL queries, and used constraint-based modeling \cite{Ohlsson.S-1992a-Constraint-based,Ohlsson.S-2016a-Constraint-based} to provide feedback to students. \citeauthor{Kenny.C-2005a-Automated} \cite{Kenny.C-2005a-Automated} described a similar SQL learning system that incorporated an assessment of a student's previous progress. This enabled a more personalized and adaptive approach to student learning, where feedback was tailored according to student progress. -There is relatively little prior work on unit testing of databases. Most authors working in this area have focused on testing database \emph{applications} rather than the database itself (e.g., \cite{Binnig.C-2008a-Multi-RQP,Chays.D-2008a-Query-based,Marcozzi.M-2012a-Test,Haller.K-2010a-Test}). \citeauthor{Ambler.S-2006a-Database} discusses how to test the functionality of a database \cite{Ambler.S-2006a-Database}, while \citeauthor{Farre.C-2008a-SVTe} test the ``correctness'' of a schema \cite{Farre.C-2008a-SVTe}, focusing mainly on consistency of constraints. Neither consider whether a database schema conforms to the requirements of the original specification. +There is relatively little prior work on unit testing of databases. Most authors working in this area have focused on testing database \emph{applications} rather than the database itself (e.g., \cite{Binnig.C-2008a-Multi-RQP,Chays.D-2008a-Query-based,Marcozzi.M-2012a-Test,Haller.K-2010a-Test}). \citeauthor{Ambler.S-2006a-Database} discusses how to test the functionality of a database \cite{Ambler.S-2006a-Database}, while \citeauthor{Farre.C-2008a-SVTe} describe how to test the ``correctness'' of a schema \cite{Farre.C-2008a-SVTe}, focusing mainly on consistency of constraints. Neither consider how to verify that a database schema conforms to the requirements of the original specification. \section{Motivation} \label{sec-motivation} -Since 1989, our department has offered some form of mandatory database concepts course, typically one semester during the second year\footnote{New Zealand Bachelor's degrees normally comprise three years of study.} of study, building on a short introduction to basic data management concepts in the first year. Typical of courses of this nature, it covered core topics such as the relational model, relational algebra, data integrity, SQL (DDL and DML), and a varying mixture of other database topics such as transactions, concurrency control, triggers, and security. Assessment of SQL skills was typically carried out using a mixture of assignments and tests. +Since 1989, our department has offered some form of mandatory database concepts course, typically one semester during the second year\footnote{New Zealand Bachelor's degrees normally comprise three years of study.} of study, building on a short introduction to basic data management concepts in the first year. Typical of courses of this nature, it covered core topics such as the relational model, relational algebra, data integrity, SQL DDL and DML, and a varying mixture of other database topics such as transactions, concurrency control, triggers, and security. Assessment of SQL skills was typically carried out using a mixture of assignments and tests. From 2001 to 2003, we assessed students' SQL DDL skills with an in-lab practical examination. Students were given a fictional scenario specification, and had 100 minutes in which to modify a provided schema template with additional tables, constraints, etc. The test was generally easier to grade than the more ``realistic'' practical assignment that we had used in earlier years, as the scenario specification tended to be quite tightly specified and thus less open to (mis)interpretation. However, the test experience was quite stressful to students due to the limited timeframe and limited access to online references. We did not attempt to automate the grading of these tests. @@ -156,26 +156,26 @@ Prior to 2001, the specifications for the scenarios used in this assignment were deliberately somewhat loosely defined and often contained areas that were under-specified or ambiguous. At the time the course also included some data modeling content, so this approach enabled students to explore the different ways that a conceptual model could be converted into an implementation. This of course led to variation across student submissions, due to differing interpretations of the under-specified elements. Automation of grading was problematic, especially when students chose different database structures, or different names for tables and columns, than what was ``expected''. We therefore made no significant attempt to automate grading under this approach. -By 2004 the data modeling content was being migrated into other courses, so we took the opportunity to tighten up the scenario specifications to reduce ambiguity. In 2013 we took the further step of ``freezing'' the specification, i.e., students were not permitted to make arbitrary changes to the specification without strong justification, and even then only if the changes did not impact how client programs interacted with the database. For example, if a column needed to be restricted to a discrete set of values, a student could---if they wished---choose to go beyond the specification and enforce this requirement using a separate lookup table, as long as this did not change the structure of the original table. This meant that students could still exercise some flexibility in their schema implementations where appropriate, and ensured that the assignment was not just a mechanical exercise. +By 2004 the data modeling content was being migrated into other courses, so we took the opportunity to tighten up the scenario specifications to reduce ambiguity. In 2013 we took the further step of ``freezing'' the specification, i.e., students were not permitted to make arbitrary changes to the specification without strong justification, and even then only if the changes did not impact how client programs interacted with the database. For example, if a column needed to be restricted to a discrete set of values, a student could---if they wished---choose to go beyond the specification and enforce this requirement using a separate lookup table rather than a \texttt{CHECK} constraint, as long as this did not change the originally specified structure of the table. This meant that students could still exercise some flexibility in their schema implementations where appropriate, and ensured that the assignment was not just a mechanical translation exercise. The in-scenario rationale for freezing the specification was that other developers were independently using the same specification to code end-user applications. Any significant variation from the specification would therefore break those applications. This approach tested not only students' ability to write SQL DDL, but also their ability to correctly interpret and convert a natural language database specification into a corresponding SQL schema. -This approach seemed effective, but maintaining consistent grading standards across a large number of submissions (on the order of 50--70) was difficult, due the often large number of distinct gradable elements implied by the specification. This required a complex and highly-detailed rubric to be constructed so that no element was missed, and the grading process consequently took a significant amount of time, especially if feedback to students was required. In 2012 a significant change to the structure of the course resulted in higher grading workloads and increased time pressure, prompting greater interest in the possibility of at least semi-automating the grading of this assignment. This was now more feasible than in earlier years due to the more tightly constrained nature of the project specification. +This approach seemed effective, but maintaining consistent grading standards across a large number of submissions (on the order of 50--70) was difficult, due the often large number of distinct gradable elements implied by the specification. This required a complex and highly-detailed rubric to be constructed so that no element was missed, and the grading process consequently took a significant amount of time, especially when feedback to students was required. In 2012 a significant change to the structure of the course resulted in higher grading workloads and increased time pressure, prompting greater interest in the possibility of at least semi-automating the grading of this assignment. This was now more feasible than in earlier years due to the more tightly constrained nature of the project specification. Another motivation for automated grading was that it can sometimes be difficult for database novices to know whether they are on the right track while implementing a specification. If a limited version of the grading tool was available to students before the submission deadline, it could be used to provide feedback on their progress. -We decided to define a minimum set of requirements for the assignment: students' SQL code should be syntactically correct, and it should include all tables and columns---with correct names and appropriate data types---as detailed in the specification. If a student satisfied these minimum requirements, they were guaranteed to score at least 50\%. They could check whether their schema met these requirements prior to the deadline by submitting their schema for checking through a web application. That way they (and we) could be more certain that at least the basic structure of their schema was correct. After the deadline, teaching staff used a console application to check for correct implementation of other aspects of the schema such as integrity constraints. +We decided to define a minimum set of requirements for the assignment: students' SQL code should be syntactically correct, and it should include all tables and columns---with correct names and appropriate data types---as detailed in the specification. If a student satisfied these minimum requirements, they were guaranteed to score at least 50\%. They could check whether their schema met these requirements prior to the deadline by submitting their schema for checking through a web application. That way they (and we) could be more certain that at least the core structure of their schema was correct. After the deadline, teaching staff used a console application to check for correct implementation of other aspects of the schema, such as integrity constraints. -We trialed a prototype of the staff-facing application in 2012, and rolled out the student-facing application in 2013. +We trialled a prototype of the staff-facing application in 2012, and rolled out the student-facing application in 2013. \section{System design} \label{sec-design} -The architecture of our system is shown in \cref{fig-architecture}. Its core function is to check whether a student has conformed to the assignment specification, by automatically comparing their schema submission against a machine-readable version of the specification. This is essentially a unit testing approach, so we decided to build the system around an existing unit testing framework. +The architecture of our system is shown in \cref{fig-architecture}. Its core function is to check whether a student's schema conforms to the assignment specification, by automatically comparing their schema submission against a machine-readable version of the specification. This is essentially a unit testing approach, so we decided to build the system around an existing unit testing framework. -\begin{figure} +\begin{figure}[h] \sffamily \begin{tikzpicture}[every node/.style={draw, minimum height=7.5mm, inner sep=1em}] \node (console) {\shortstack{Console app \\ \footnotesize(staff mode)}}; @@ -298,7 +298,7 @@ Each table specification also requires the definition of two distinct sets of tests that will be run on the database. The first set verifies the structural elements of the table (columns, data types, etc.), thus verifying that a schema meets the minimum requirements of the assignment. This is done by issuing queries against the metadata (catalog) of the schema for elements like tables, columns, and data types. When in student mode, only this set of tests is run. An empty data fixture (specified using a separate XML document) is also required to support this set of tests. -The second set of tests verifies the integrity elements of the table, i.e., its constraints. The only integrity constraints that are explicitly tested are primary and foreign keys, which are again verified using queries against the schema's metadata, and nullability (\texttt{NOT NULL}), which is tested by attempting insert a null. The remaining constraints are tested by attempting to insert lists of known legal and illegal values, an approach consistent with normal unit testing practice. When in staff mode, both this set of tests and the set of structural tests are run. A known-legal data fixture is also required to support this set of tests. +The second set of tests verifies the integrity elements of the table, i.e., its constraints. The only integrity constraints that are explicitly tested are primary and foreign keys, which are again verified using queries against the schema's metadata, and nullability (\texttt{NOT NULL}), which is tested by attempting to insert a null. The remaining constraints are tested by attempting to insert lists of known legal and illegal values, an approach consistent with normal unit testing practice. When in staff mode, both this set of tests and the set of structural tests are run. A known-legal data fixture is also required to support this set of tests. \begin{figure} @@ -308,7 +308,7 @@ \end{figure} -The way that the system runs the tests is somewhat unusual, in two ways. First, database unit testing frameworks are generally designed around the concept of testing database-backed \emph{applications}, rather than testing the \emph{database} itself. Second, in typical unit testing, the tests are essentially standalone code units that are automatically executed in an order determined by the unit testing framework. The framework resolves dependencies among tests and handles collation of test results internally, reporting only success, failure, or ``error'' back to the client. In fact, most unit testing frameworks appear to be designed to be called from a specific tool (e.g., PHPUnit's \texttt{phpunit} command line tool), rather than embedded as a library into other programs. +The way that the system runs the tests is somewhat unusual, in two ways. First, database unit testing frameworks are generally designed around the concept of testing database-backed \emph{applications}, rather than testing the database itself. Second, in typical unit testing, the tests are essentially standalone code units that are automatically executed in an order determined by the unit testing framework. The framework resolves dependencies among tests and handles collation of test results internally, reporting only success, failure, or ``error'' back to the client. In fact, most unit testing frameworks appear to be designed to be called from a specific tool (e.g., the \texttt{phpunit} command line tool), rather than embedded as a library into other programs. Our system effectively inverts (or perhaps subverts) this approach. The main driver explicitly creates test suites itself and executes them directly, listening for the results of each test and collating them. This is because we need to be able to control the order in which tests are executed. If the structural tests fail, there is little point in running the integrity tests, as they will only generate a stream of errors. Similarly, if a column is missing, there is no point in running the data type and length tests. (Note that while PHPUnit supports grouping of tests, dependencies can only be defined between individual tests, not between groups.) @@ -397,9 +397,9 @@ Unfortunately, the system was conceived as a practical solution to a perceived teaching issue, rather than as a research project with formal evaluation in mind. We therefore did not carry out any user evaluations with students that used the system. -From the perspective of the teaching staff, we found that the total amount of time taken to grade the relevant assignment was reduced only a little, as the system only semi-automated the grading process, and we still needed to convert the system's output into corresponding grades and meaningful feedback (see \cref{sec-issues} for further discussion of this). There was also a surprising number of submissions that did not meet the minimum requirements and thus had to be manually graded. On a more positive note, the system did automatically ensure that all gradable elements were checked, which improved consistency, and made the subjective experience of grading much less painful. +From the perspective of the teaching staff, we found that the total amount of time taken to grade the relevant assignment was reduced only a little, as the system only semi-automated the grading process, and we still needed to convert the system's output into corresponding grades and meaningful feedback (see \cref{sec-issues} for further discussion of this). There were also a surprising number of submissions that did not meet the minimum requirements and thus had to be manually graded. On a more positive note, the system did automatically ensure that all gradable elements were checked, which improved consistency, and made the subjective experience of grading much less painful. -We were able, however, to analyse whether use of the system had any impact on student performance in the relevant assignment, as we had extensive historical data of student grades. We collated data for the period 2009--2016 (there are no data for 2017 because the course has been discontinued; see \cref{sec-issues}), which encompassed several different permutations of scenario and available system modes, as summarized in \cref{tab-data}. The assignment counted for 15\% of a student's total grade in 2009 and 2010, and 10\% in subsequent years. The grade distributions for the assignment in each year are shown in \cref{fig-distributions}. +However, we were able to analyze whether use of the system had any impact on student performance in the relevant assignment, as we had extensive historical data of student grades. We collated data for the period 2009--2016 (there are no data for 2017 because the course has been discontinued; see \cref{sec-issues}), which encompassed several different permutations of scenario and available system modes, as summarized in \cref{tab-data}. The assignment counted for 15\% of a student's total grade in 2009 and 2010, and 10\% in subsequent years. The grade distributions for the assignment in each year are shown in \cref{fig-distributions}. \begin{table} @@ -430,14 +430,14 @@ \begin{figure} \includegraphics[width=\columnwidth,keepaspectratio]{grades_all_years.pdf} - \caption{Comparison of grade distributions for database assignment, 2009--2016.} + \caption{Comparison of grade distributions for the database implementation assignment, 2009--2016.} \label{fig-distributions} \end{figure} The horizontal rule in \cref{tab-data} between 2011 and 2012 marks a significant reorganization of the course's curriculum, and also a switch from first to second semester.\footnote{First semester at the University of Otago runs from March to June, second semester from July to October.} (2012 was also the year we deployed the first prototype of staff mode.) The horizontal rule between 2013 and 2014 marks a shift from second semester back to first semester. The system was not used at all in 2015 due to different staff teaching the course, and student mode was not made available in 2016 due to technical issues delaying its deployment beyond the point where it would be useful. These differences have provided us with a natural experiment with some interesting points for comparison. -The mean grade for the assignment drifted slowly downwards from 2009 to 2012. This reversed dramatically in 2013, however, the year we first deployed student mode. The grades are not normally distributed (they typically have negative skew), so we used a Mann-Whitney \emph{U} test to determine whether the increase in mean from 2012 to 2013 was statistically significant. This showed that the increase was highly significant (\(p \approx 10^{-9}\)). The 2013 mean was also significantly higher than both 2010 (\(p \approx 0.0002\)) and 2011 (\(p \approx 10^{-6}\)), but not significantly higher than 2009. The mean decreased significantly again in 2014 (\(p \approx 0.0012\)), the second year that the system was used, and even more dramatically in 2015 (\(p \approx 0.0005\)), when the system was not used at all. The change from 2015 to 2016 was not significant. +The mean grade for the assignment drifted slowly downwards from 2009 to 2012. This reversed dramatically in 2013, however, the year we first deployed student mode. As can be seen in \cref{fig-distributions}, the grades are not normally distributed (they typically have negative skew), so we used a Mann-Whitney \emph{U} test to determine whether the increase in mean from 2012 to 2013 was statistically significant. This showed that the increase was highly significant (\(p \approx 10^{-9}\)). The 2013 mean was also significantly higher than both 2010 (\(p \approx 0.0002\)) and 2011 (\(p \approx 10^{-6}\)), but not significantly higher than 2009. The mean decreased significantly again in 2014 (\(p \approx 0.0012\)), the second year that the system was used, and even more dramatically in 2015 (\(p \approx 0.0005\)), when the system was not used at all. The increase from 2015 to 2016 was not significant. Even more interesting, if we compare the performance between the years that student mode was not available (2009--2012 and 2015--2016, mean 71.6\%) and the years it was (2013--2014, mean 81.7\%), there is again a highly statistically significant increase in the mean (\(p \approx 10^{-8}\)). This difference is strikingly illustrated in \cref{fig-distribution-comparison}, which compares the grade distributions for both periods. During the period that student mode was available, the lowest grade awarded was 46\%, contrasted with a much longer tail of lower grades when student mode was not available. This strongly suggests that the introduction of student mode had a positive impact on students' ability to complete the assignment more effectively. @@ -458,7 +458,7 @@ Third, the switch to second semester in 2012--2013 could have negatively impacted students' performance by increasing the length of time between their exposure to basic data management concepts in first year, and their entry into the second year database course. In effect, they had longer to forget relevant material they learned in first year. If so, we could reasonably expect the grades in second semester offerings of the course to be lower. However, mean grades for second semester offerings (76.9\%) were in fact significantly \emph{higher} (\(p \approx 0.015\)) than those for first semester offerings (72.9\%). This should not be surprising, given that 2013 (second semester) had the highest grades overall. This effectively rules out semester changes as a factor. Fourth, perhaps the years with higher grades used less complex---and therefore easier---scenarios. To test this, we computed the following database complexity metrics for each of the four scenarios used: database complexity index (DCI) \cite{Sinha.B-2014a-Estimation}; referential degree (RD), depth of referential tree (DRT), and number of attributes (NA) \cite{Calero.C-2001a-Database,Piattini.M-2001a-Table}; database complexity (DC) \cite{Pavlic.M-2008a-Database}; and ``Software Metric Analyzer for Relational Database Systems'' (SMARtS) -\cite{Jamil.B-2010a-SMARtS}. The results are shown in \cref{tab-metrics}. All but the DRT metric clearly showed the ``BDL'', ``used cars'', and ``student records'' scenarios to be of comparable complexity, while the ``postgrad'' scenario was less complex. It therefore seems unlikely that scenario complexity is a factor in student performance. It is also interesting to note that the ``used cars'' scenario was used in 2014 and 2015, and yet the 2015 grades were significantly \emph{lower} than those for 2014. The only clear difference here is that our system was not used in 2015. +\cite{Jamil.B-2010a-SMARtS}. The results are shown in \cref{tab-metrics}. All but the DRT metric clearly showed the ``BDL'', ``used cars'', and ``student records'' scenarios to be of comparable complexity, while the ``postgrad'' scenario was noticeably less complex. It therefore seems unlikely that scenario complexity is a factor in student performance. It is also interesting to note that the ``used cars'' scenario was used in both 2014 and 2015, and yet the 2015 grades were significantly \emph{lower} than those for 2014. The only clear difference here is that our system was not used in 2015. \begin{table} @@ -479,13 +479,13 @@ Fifth, class size could be a factor. We might plausibly expect a smaller class to have a more collegial atmosphere that promotes better learning. However, if we look at the sizes of the classes in \cref{tab-data}, we can see no discernible pattern between class size and performance. Indeed, both the best (2013) and worst (2012, 2015) performances came from classes of similar size (75, 77, and 71, respectively). -Sixth, it could be that better performance occurred in years where the students were just more capable in general. We obtained annual GPA data for the students enrolled in each year, and computed the median as an indication of the general capability of the class. Looking at \cref{tab-data}, we can immediately see that the year with the best results (2013) was also the year with the second-lowest median GPA (3.2). Contrast this with the poor performance in 2012, where the median GPA was 3.4. In both years that student mode was available, median GPA was lower than or the same as in most other years, yet performance was better than in years with higher median GPA. This argues against the idea that we simply had a class full of very capable students in the better performing years. +Sixth, it could be that better performance occurred in years where the students were just more capable in general. We obtained annual GPA data for the students enrolled in each year, and computed the median as an indication of the general capability of the class. Looking at \cref{tab-data}, we can immediately see that the year with the best results (2013) was also the year with the second-lowest median GPA (3.2). Contrast this with the poorer performance in 2012, where the median GPA was 3.4. In both years that student mode was available, median GPA was lower than or the same as in most other years, yet performance was better than in years with higher median GPA. This argues against the idea that we simply had a class full of very capable students in the better performing years. Seventh, the timing of the assignment varied across the period, occurring either early (2012--2014), halfway (2009-2010), or late (2011, 2015--2016) in the semester, depending on the ordering of course content. The mean grade for early timing (77.0\%) was significantly higher than for both halfway (72.2\%, \(p \approx 0.019\)) and late (71.8\%, \(p \approx 0.013\)), while there was no significant difference between the halfway and late means. This suggests that scheduling the assignment early in the semester may have a positive effect on grades, and the period of early timing does overlap the period that student mode was available. However, as noted earlier there was a highly significant difference in mean between 2012 and 2013. There is also a smaller, but still significant difference in mean between 2012 and 2014 (\(p \approx 0.0015\)). We therefore conclude that while scheduling the assignment early in the semester may have had some positive effect on student performance, it does not explain all of the positive effect seen in 2013 and 2014. Finally, perhaps the different weightings of the assignment (15\% in 2009--2010 vs.\ 10\% in 2011--2016) affected student motivation. It could be argued that the higher weighting in 2009--2010 provided a greater incentive for students to work more, as the potential reward was greater. If so, we should expect better performance in 2009--2010. Indeed, we do find this: the mean for 2009--2010 is 75.1\%, while that for 2011--2016 is 73.9\%, a statistically significant decrease (\(p \approx 0.034\)). However, since this change occurred well before our system was even conceived of, let alone implemented, it cannot be a factor in the improved performance seen in 2013 and 2014. -The sum of this evidence provides a strong argument in favour of student mode being the main factor in improving student grade performance in the database implementation assignment during 2013 and 2014. The most plausible explanation for this effect is that the direct feedback provided by the student mode web application enabled students to more effectively correct the structural aspects of their schema to meet the minimum requirements, giving them more time to work on the integrity aspects of their schema, and thus gain a higher grade. +The sum of this evidence provides a strong argument in favor of student mode being the main factor in improving student grade performance in the database implementation assignment during 2013 and 2014. The most plausible explanation for this effect is that the direct feedback provided by the student mode web application enabled students to more effectively correct the structural aspects of their schema to meet the minimum requirements, giving them more time to work on the integrity aspects of their schema, and thus gain a higher grade. \section{Known issues \& future work} @@ -495,7 +495,7 @@ Ideally, our system would automatically assign appropriate marks and generate meaningful feedback, and write both of these directly into a student management database. Currently, however, it only semi-automates the grading process, and it is still up to the teacher to interpret the test results, assign appropriate marks, and write feedback. -Automatically generating appropriate marks is not particularly difficult, and we have already implemented the core functionality required to assign mark penalties to different kinds of error. It would be a relatively simple extension to write these penalties directly into a student management database. +Automatically generating appropriate marks is not particularly difficult, and we have already implemented the core functionality required to assign mark penalties to different kinds of error. It would be a relatively simple extension to calculate marks based on these penalties and write them directly into a student management database. Automatically generating meaningful feedback is problematic, however, due to the way that PHPUnit reports test failures. There is currently no way to control or suppress the messages generated by PHPUnit test assertions, most likely because PHPUnit is not designed to be used as an embedded library (as discussed in \cref{sec-design}). Individual tests can specify a meaningful message to be displayed, but PHPUnit will still generate somewhat obscure messages like ``Failed asserting that 0 matches expected 1'', which can be confusing for students. @@ -505,11 +505,11 @@ A second issue is that any schema that fails to meet the minimum requirements will result in a large number of errors due to missing tables. This could be alleviated by more careful exception handling in the main driver. -Another issue is how students made use of the student mode web application. In 2013, students tended to misuse it as a ``schema compiler'': they would submit their schema, fix only the first issue reported (rather than attempting to fix as many as possible), immediately re-submit, and so on. We had not expected so high a request rate, so the system was single-threaded and thus not designed to handle concurrent requests. Consequently the web application suffered from waits and timeouts. An initial workaround was to log all student access and warn those who were abusing the system. We later throttled the system by enforcing a short delay between attempts by the same student. Things improved in 2014 with careful management of student expectations, but longer term, the system should be redesigned to handle concurrent loads. +Another issue is how students made use of the student mode web application. In 2013, students tended to misuse it as a ``schema compiler'': they would submit their schema, fix only the first issue reported (rather than attempting to fix as many as possible), immediately re-submit, and so on. We had not expected so high a request rate, so the system was only single-threaded and thus not designed to handle concurrent requests. Consequently the web application suffered from waits and timeouts. An initial workaround was to log all student access and warn those who were abusing the system. We later throttled the system by enforcing a short delay between attempts by the same student. Things improved in 2014 with careful management of student expectations, but longer term, the system should be redesigned to handle concurrent loads. -The system currently does not support behavioral aspects such as sequences, triggers, or stored procedures, as these are more often DBMS-specific in nature. It would not be difficult to add support for these, however, which could be included in the DBMS abstraction layer proposed in \cref{sec-design}. +The system currently does not support behavioral aspects such as sequences, triggers, or stored procedures, as these are usually DBMS-specific in nature. It would not be difficult to add support for these, however, which could be included in the DBMS abstraction layer proposed in \cref{sec-design}. -Looking forward, it would be interesting to extend our system to help facilitate the teaching of SQL DDL, rather than just focusing on assignment grading. This has been done previously with several systems for SQL DML (e.g., \cite{Kenny.C-2005a-Automated,Kleiner.C-2013a-Automated,Mitrovic.A-1998a-Learning,Russell.G-2004a-Improving,Sadiq.S-2004a-SQLator}). +Looking forward, it would be interesting to extend our system to help facilitate the teaching of SQL DDL, rather than just focusing on assignment grading. This has been done previously with several systems for SQL DML, as discussed in \cref{sec-literature} (e.g., \cite{Kenny.C-2005a-Automated,Kleiner.C-2013a-Automated,Mitrovic.A-1998a-Learning,Russell.G-2004a-Improving,Sadiq.S-2004a-SQLator}). From 2017, we no longer have a dedicated second year database course. The main introduction to database concepts and SQL now occurs as part of our new (in 2017) first year ``Foundations of Information Systems'' course. This course attracted about 160 students in first semester, and about 100 students in second semester. Classes of this size further strengthen the argument for automated grading, and we are currently exploring whether our system could be used in this course in 2018. If this is feasible, we intend to carry out user evaluations with the students. @@ -517,9 +517,9 @@ \section{Conclusion} \label{sec-conclusion} -In this paper we have described a novel system that semi-automates the process of grading SQL data definition code. To our knowledge, all prior work in this area has focused only on data manipulation statements such as \texttt{SELECT}, so extending this to data definition code is a key contribution of our system. The system also takes a novel approach to checking an SQL schema: rather than attempting to parse the SQL DDL code directly, it instead verifies that the schema conforms to a machine-readable specification. Finally, our system uses a unit testing approach to carry out the verification, which has not been used before in this context. +In this paper we have described a novel system that semi-automates the process of grading SQL data definition code. To our knowledge, all prior work in this area has focused only on data manipulation statements such as \texttt{SELECT}, so extending this to data definition code is a key contribution of our system. The system also takes a novel approach to checking an SQL schema: rather than attempting to parse the SQL DDL code directly, it instead verifies that the schema conforms to a machine-readable version of the original specification. Finally, our system uses a unit testing approach to carry out the verification, which has not been used before in this context. -While no user evaluations of the system were carried out, analysis of student performance in a relevant database assessment shows a highly statistically significant increase in mean grade in the two years that the system was available to students. While we cannot be absolutely certain, analysis of all the factors involved strongly suggests that use of our system by students had a positive impact on their performance in this assessment. This is an encouraging result, which we will explore further in future work. +While no user evaluations of the system were carried out, analysis of student performance in a relevant database assessment shows a highly statistically significant increase in mean grade in the two years that the system was available for use by students. While we cannot be absolutely certain, analysis of all the factors involved strongly suggests that use of our system by students had a positive impact on their performance in this assessment. This is an encouraging result, which we will explore further in future work. \bibliographystyle{ACM-Reference-Format}