Pull requests - nigel.stanger/Publications

nigel.stanger / Publications

...

54 commits

22 files changed

0 commit comments

1 contributor

Commits on 2017-11-03

Nigel Stanger

Merge branch 'DP_2017' of https://github.com/nstanger/Publications into DP_2017

e3500fb

Commits on 2017-11-02

Nigel Stanger	• Fixed more typos.	7443f49
Nigel Stanger	• Removed unnecessary line break.	50c5e6b
Nigel Stanger	• Fixed en dash.	6878787
Nigel Stanger	• Fixed typo.	ccc4148
Nigel Stanger	• Fixed typo.	b3d5f71

Commits on 2017-11-01

Nigel Stanger

• Fixed some minor inconsistencies.

39e4855

Commits on 2017-10-31

Nigel Stanger

• Fixed typo.

0e8bc7c

Commits on 2017-08-28

Nigel Stanger

• Ignored submitted paper.

9ab3e26

Commits on 2017-08-04

Nigel Stanger	• Anonymised country and institution.	f827128
Nigel Stanger	• Switched to anonymous review mode.	a3b6797
Nigel Stanger	• Final proof-read.	5ba66e8
Nigel Stanger	• Added some missing SQL learning systems.	4ff9cc6
Nigel Stanger	• Rewrote literature review.	a8186d7
Nigel Stanger	• Added grade distribution comparisons.	d159833
Nigel Stanger	• Changed code listing to a figure. • Fixed position of table captions.	1a67d67

Commits on 2017-08-03

Nigel Stanger	• Second proof-reading pass. • Rewrote test execution description. • Added summary to evaluation section. • Restructured issues discussion.	057a4c3
Nigel Stanger	• Rephrased first paragraph of literature review.	925a4fb
Nigel Stanger	• Added results of database complexity metrics.	299149a
Nigel Stanger	• First proof-reading pass. • Added another confounding factor. • More consistent use of “behaviour”. • Added two-way flows to architecture diagram. • Expanded on DBMS-independence. • Cleaned up integrity testing. • Expanded on use of unit testing. • Added note about grouping tests. • Clarified teacher grading process. • Rewrote several parts of evaluation section. • Rewrote marks & feedback generation in issues section.	7f4b457

Commits on 2017-08-02

Nigel Stanger	• Ignored ACM documentation.	5a029f8
Nigel Stanger	• Added GPA data. • Ignored raw data directory.	d93caa4
Nigel Stanger	• Tweaked format of Ambler 2006. • Moved version number in Apache 2017 (CQL).	5158a00
Nigel Stanger	• Wrote abstract. • Switched code to listings and reformatted. • Rearranged positions of some figures.	c3a79db
Nigel Stanger	• Tweaked title slightly. • Added a bunch of ACM boilerplate. • Added CCS concepts and keywords. • Expanded a missing list of citations.	d2367a8
Nigel Stanger	• American spell-checked. • A bunch of minor tweaks in lots of places. • Added missing description of how the tests actually work (e.g., metadata querying) in design section. • Emphasised novelty more. • “Behavioural” tests → “integrity” tests. • Added teaching experience to evaluation. • Wrote issues & future work section. • Wrote conclusion.	b72c4ac
Nigel Stanger	• Added another confounding factor and supporting GPA data. • Added new discussion section. • Added notes to data table.	fa65edb

Commits on 2017-08-01

Nigel Stanger	• Rewrote the motivation section a bit more. • Removed all sub-section headings. • Rewrote system design section. • Added description of how tests are run and the ability to add custom table properties. • Rewrote evaluation section, and added dsicussion of confounding factors.	e7f9678
Nigel Stanger	• Added data files for database complexity metrics, and student marks data.	a453a39
Nigel Stanger	• Added some references to database schema complexity metrics.	04118c2
Nigel Stanger	• Added a bunch of statistical analysis.	ed0eee5

Commits on 2017-07-31

Nigel Stanger	• Added raw data for assignment marks.	a6eda31
Nigel Stanger	• Added section on test specification. • Updated sections on different operation modes and rearranged figures. • Added more notes on evaluation and known issues. • Other minor changes.	5480e58
Nigel Stanger	• Rewrote motivation section. • Wrote first part of system design section.	742f5ba
Nigel Stanger	• Added PHPUnit documentation.	fe8508e

Commits on 2017-07-30

Nigel Stanger

• Rewrote motivation section.

76995cd

Commits on 2017-07-28

Nigel Stanger	• Added Date and Douce references. • Removed unused macro strings. • Added some missing metadata.	92913ca
Nigel Stanger	• Rewrote introduction. • Moved material into new Motivation section. • Started using cleveref package.	4685d03

Commits on 2017-07-24

Nigel Stanger	• Added first draft of literature review. • Added more notes. • Added web output figure. • Changed console output table into a figure.	49dcd90
Nigel Stanger	• Reordered older systems into date order.	545c8b1

Commits on 2017-07-23

Nigel Stanger	• Added more notes. • Added system architecture diagram. • Tweaked layout.	540b790
Nigel Stanger	• Tweaked border thickness.	16a9bbd
Nigel Stanger	• Updated fonts. • Adjusted diagram layout.	8d2a251

Commits on 2017-07-22

Nigel Stanger	• Ignored generated PDFs.	0b90bcb
Nigel Stanger	• Added example of PHP table specification. • Added notes on implementation.	cd3c95a
Nigel Stanger	• Added initial version of BDL ERD.	8ea78cf
Nigel Stanger	• Finalised content, colouring, and spacing of terminal output.	f39303b

Commits on 2017-07-21

Nigel Stanger

• Added example of text output.

ba06acc

Commits on 2017-07-20

Nigel Stanger	• Wrote rough draft of introduction.	6a25075
Nigel Stanger	• Added initial files.	1936d4b
Nigel Stanger	• Added braces around capitalised acronyms.	d05efc6

Commits on 2017-05-30

Nigel Stanger	• Corrected capitalisation in a couple of references. • Added title. • Fiddles with formatting and phrasing. • Added new section titles. • Added description of main goal.	987b600
Nigel Stanger	• Ignored main LaTeX PDF output.	139b5d5
Nigel Stanger	• Added initial files.	d188ab8

Patch

Showing 22 changed files

Ignore Space Show notes View DP_2017/.gitignore 0 → 100644
DP.pdf

Ignore Space Show notes View DP_2017/DP.bib 0 → 100644
@techreport{Brown.A-2006a-Automatic, Address = {London, UK}, Author = {Adrian Brown}, Institution = {The National Archives}, Month = {7~} # mar, Number = {DPTP-01}, Title = {Automatic Format Identification Using {PRONOM} and {DROID}}, Type = {Digital Preservation Technical Paper}, Url = {http://www.nationalarchives.gov.uk/aboutapps/fileformat/pdf/automatic\_format\_identification.pdf}, Year = {2006}} @techreport{Brown.A-2006a-PRONOM, Address = {London, UK}, Author = {Adrian Brown}, Institution = {The National Archives}, Month = {27~} # jul, Number = {DPTP-02}, Title = {The {PRONOM} Unique Identifier Scheme}, Type = {Digital Preservation Technical Paper}, Url = {http://www.nationalarchives.gov.uk/aboutapps/pronom/pdf/pronom\_unique\_identifier\_scheme.pdf}, Year = {2006}} @misc{Cerf.V-2015a-Digital, Author = {Vinton Cerf}, Howpublished = {Presentation at the American Association for the Advancement of Science 2015 Annual Meeting, San Jose, California, USA}, Month = {13~} # feb, Title = {Digital vellum}, Url = {https://aaas.confex.com/aaas/2015/webprogram/Paper14064.html}, Year = {2015}} @techreport{Gillespie.J-2004a-Coping, Address = {Sydney, Australia}, Author = {Julian Gillespie and Patrick Fair and Adrian Lawrence and David Vaile}, Institution = {Cyberspace Law {\&} Policy Centre, University of New South Wales}, Note = {Accessed on 12 May 2015}, Title = {Coping when everything is digital? {D}igital documents and issues in document retention}, Type = {White paper}, Url = {http://www.cyberlawcentre.org/ddr/}, Year = {2004}} @misc{JHOVE-2009a, Author = {{JHOVE}}, Note = {Accessed on 12 May 2015}, Title = {{JSTOR}/{H}arvard {O}bject {V}alidation {E}nvironment}, Url = {http://hul.harvard.edu/jhove/}, Year = {2009}} @article{Kolowich.S-2009a-Archiving, Author = {Steve Kolowich}, Journal = {The Chronicle of Higher Education}, Month = apr # {~10}, Note = {Accessed on 12 May 2015}, Title = {Archiving writers' work in the age of e-mail}, Url = {http://chronicle.com/article/Archiving-Writers-Work-in/22770}, Year = {2009}} @inproceedings{Li.Q-2010a-SVM, Author = {Qiming Li and Alvin Y. Ong and Ponnuthurai N. Suganthan and Vrizlynn L. L. Thing}, Crossref = {WDFIA-2010a-Proceedings}, Title = {A novel support vector machine approach to high entropy data fragment classification}, Url = {http://isis.poly.edu/~qiming/publications/wdfia10.pdf}} @misc{Microsoft-2014a-Word2013, Author = {{Microsoft Corporation}}, Note = {Accessed on 12 May 2015}, Title = {File formats that are supported in {W}ord 2013}, Url = {https://technet.microsoft.com/en-us/library/dd797428(v=office.15).aspx\#section1}, Year = {2014}} @inproceedings{Mokhov.S-2008a-File, Author = {Serguei A. Mokhov and Mourad Debbabi}, Bibsource = {DBLP, http://dblp.uni-trier.de}, Crossref = {Gobel.O-2008a-IMF}, Pages = {73--86}, Title = {File type analysis using signal processing techniques and machine learning vs.\ file {U}nix utility for forensic analysis}, Url = {http://subs.emis.de/LNI/Proceedings/Proceedings140/gi-proc-140-007.pdf}} @article{Mount.R-2009a-Data, Author = {Richard Mount}, Journal = {Ariadne}, Month = jan, Note = {Accessed on 12 May 2015}, Number = {58}, Title = {Data preservation and long-term analysis in high-energy physics}, Url = {http://www.ariadne.ac.uk/issue58/dplta-hep-rpt/}, Year = {2009}} @misc{NatLib.NZ-2007a-Metadata, Author = {{National Library of New Zealand}}, Date-Added = {2017-05-08 09:45:26 +0000}, Date-Modified = {2017-05-08 09:46:27 +0000}, Note = {Accessed on 12 May 2015}, Title = {Metadata {E}xtraction {T}ool}, Url = {http://meta-extractor.sourceforge.net/}, Year = {2007}} @article{Pringle.H-2010a-NASA, Author = {Heather Pringle}, Doi = {10.1126/science.327.5971.1322}, Journal = {Science}, Month = mar, Number = {5971}, Pages = {1322--1323}, Title = {{NASA} dives into its past to retrieve vintage satellite data}, Volume = {327}, Year = {2010}} @inproceedings{Roussev.V-2009a-File, Author = {Vassil Roussev and Simson L. Garfinkel}, Bibsource = {DBLP, http://dblp.uni-trier.de}, Crossref = {SADFE-2009a-Proceedings}, Doi = {10.1109/SADFE.2009.21}, Pages = {3--14}, Title = {File fragment classification---{T}he case for specialized approaches}, Year = {2009}} # Crossrefs @proceedings{Gobel.O-2008a-IMF, Address = {Mannheim, Germany}, Bibsource = {DBLP, http://dblp.uni-trier.de}, Booktitle = {Proceedings of the 2008 Conference on IT-Incidents Management and IT-Forensics (IMF 2008)}, Editor = {Oliver G{\"o}bel and Sandra Frings and Detlef G{\"u}nther and Jens Nedon and Dirk Schadt}, Isbn = {978-3-88579-234-5}, Month = sep # {~23-25}, Publisher = {Gesellschaft f{\"u}r Informatik---GI (German Computer Society)}, Series = {Lecture Notes in Informatics}, Title = {Proceedings of the 2008 Conference on IT-Incidents Management and IT-Forensics (IMF 2008)}, Volume = {140}, Year = {2008}} @proceedings{SADFE-2009a-Proceedings, Address = {Berkeley, California, USA}, Bibsource = {DBLP, http://dblp.uni-trier.de}, Booktitle = {Proceedings of the Fourth International IEEE Workshop on Systematic Approaches to Digital Forensic Engineering (SADFE 2009)}, Isbn = {978-0-7695-3792-4}, Month = may # {~21}, Publisher = {IEEE Computer Society}, Title = {Proceedings of the Fourth International IEEE Workshop on Systematic Approaches to Digital Forensic Engineering (SADFE 2009)}, Year = {2009}} @proceedings{WDFIA-2010a-Proceedings, Address = {Port Elizabeth, South Africa}, Booktitle = {Proceedings of the Fifth Annual Workshop on Digital Forensics and Incident Analysis (WDFIA)}, Date-Added = {2011-05-13 15:54:22 +1200}, Date-Modified = {2011-05-13 15:55:34 +1200}, Month = {17--18~} # may, Title = {Proceedings of the Fifth Annual Workshop on Digital Forensics and Incident Analysis (WDFIA)}, Year = {2010}}

Ignore Space Show notes View

DP_2017/DP.bib 0 → 100644

@techreport{Brown.A-2006a-Automatic,
	Address = {London, UK},
	Author = {Adrian Brown},
	Institution = {The National Archives},
	Month = {7~} # mar,
	Number = {DPTP-01},
	Title = {Automatic Format Identification Using {PRONOM} and {DROID}},
	Type = {Digital Preservation Technical Paper},
	Url = {http://www.nationalarchives.gov.uk/aboutapps/fileformat/pdf/automatic\_format\_identification.pdf},
	Year = {2006}}

@techreport{Brown.A-2006a-PRONOM,
	Address = {London, UK},
	Author = {Adrian Brown},
	Institution = {The National Archives},
	Month = {27~} # jul,
	Number = {DPTP-02},
	Title = {The {PRONOM} Unique Identifier Scheme},
	Type = {Digital Preservation Technical Paper},
	Url = {http://www.nationalarchives.gov.uk/aboutapps/pronom/pdf/pronom\_unique\_identifier\_scheme.pdf},
	Year = {2006}}

@misc{Cerf.V-2015a-Digital,
	Author = {Vinton Cerf},
	Howpublished = {Presentation at the American Association for the Advancement of Science 2015 Annual Meeting, San Jose, California, USA},
	Month = {13~} # feb,
	Title = {Digital vellum},
	Url = {https://aaas.confex.com/aaas/2015/webprogram/Paper14064.html},
	Year = {2015}}

@techreport{Gillespie.J-2004a-Coping,
	Address = {Sydney, Australia},
	Author = {Julian Gillespie and Patrick Fair and Adrian Lawrence and David Vaile},
	Institution = {Cyberspace Law {\&} Policy Centre, University of New South Wales},
	Note = {Accessed on 12 May 2015},
	Title = {Coping when everything is digital? {D}igital documents and issues in document retention},
	Type = {White paper},
	Url = {http://www.cyberlawcentre.org/ddr/},
	Year = {2004}}

@misc{JHOVE-2009a,
	Author = {{JHOVE}},
	Note = {Accessed on 12 May 2015},
	Title = {{JSTOR}/{H}arvard {O}bject {V}alidation {E}nvironment},
	Url = {http://hul.harvard.edu/jhove/},
	Year = {2009}}

@article{Kolowich.S-2009a-Archiving,
	Author = {Steve Kolowich},
	Journal = {The Chronicle of Higher Education},
	Month = apr # {~10},
	Note = {Accessed on 12 May 2015},
	Title = {Archiving writers' work in the age of e-mail},
	Url = {http://chronicle.com/article/Archiving-Writers-Work-in/22770},
	Year = {2009}}

@inproceedings{Li.Q-2010a-SVM,
	Author = {Qiming Li and Alvin Y. Ong and Ponnuthurai N. Suganthan and Vrizlynn L. L. Thing},
	Crossref = {WDFIA-2010a-Proceedings},
	Title = {A novel support vector machine approach to high entropy data fragment classification},
	Url = {http://isis.poly.edu/~qiming/publications/wdfia10.pdf}}

@misc{Microsoft-2014a-Word2013,
	Author = {{Microsoft Corporation}},
	Note = {Accessed on 12 May 2015},
	Title = {File formats that are supported in {W}ord 2013},
	Url = {https://technet.microsoft.com/en-us/library/dd797428(v=office.15).aspx\#section1},
	Year = {2014}}

@inproceedings{Mokhov.S-2008a-File,
	Author = {Serguei A. Mokhov and Mourad Debbabi},
	Bibsource = {DBLP, http://dblp.uni-trier.de},
	Crossref = {Gobel.O-2008a-IMF},
	Pages = {73--86},
	Title = {File type analysis using signal processing techniques and machine learning vs.\ file {U}nix utility for forensic analysis},
	Url = {http://subs.emis.de/LNI/Proceedings/Proceedings140/gi-proc-140-007.pdf}}

@article{Mount.R-2009a-Data,
	Author = {Richard Mount},
	Journal = {Ariadne},
	Month = jan,
	Note = {Accessed on 12 May 2015},
	Number = {58},
	Title = {Data preservation and long-term analysis in high-energy physics},
	Url = {http://www.ariadne.ac.uk/issue58/dplta-hep-rpt/},
	Year = {2009}}
	
@misc{NatLib.NZ-2007a-Metadata,
	Author = {{National Library of New Zealand}},
	Date-Added = {2017-05-08 09:45:26 +0000},
	Date-Modified = {2017-05-08 09:46:27 +0000},
	Note = {Accessed on 12 May 2015},
	Title = {Metadata {E}xtraction {T}ool},
	Url = {http://meta-extractor.sourceforge.net/},
	Year = {2007}}

@article{Pringle.H-2010a-NASA,
	Author = {Heather Pringle},
	Doi = {10.1126/science.327.5971.1322},
	Journal = {Science},
	Month = mar,
	Number = {5971},
	Pages = {1322--1323},
	Title = {{NASA} dives into its past to retrieve vintage satellite data},
	Volume = {327},
	Year = {2010}}

@inproceedings{Roussev.V-2009a-File,
	Author = {Vassil Roussev and Simson L. Garfinkel},
	Bibsource = {DBLP, http://dblp.uni-trier.de},
	Crossref = {SADFE-2009a-Proceedings},
	Doi = {10.1109/SADFE.2009.21},
	Pages = {3--14},
	Title = {File fragment classification---{T}he case for specialized approaches},
	Year = {2009}}

# Crossrefs

@proceedings{Gobel.O-2008a-IMF,
	Address = {Mannheim, Germany},
	Bibsource = {DBLP, http://dblp.uni-trier.de},
	Booktitle = {Proceedings of the 2008 Conference on IT-Incidents Management and IT-Forensics (IMF 2008)},
	Editor = {Oliver G{\"o}bel and Sandra Frings and Detlef G{\"u}nther and Jens Nedon and Dirk Schadt},
	Isbn = {978-3-88579-234-5},
	Month = sep # {~23-25},
	Publisher = {Gesellschaft f{\"u}r Informatik---GI (German Computer Society)},
	Series = {Lecture Notes in Informatics},
	Title = {Proceedings of the 2008 Conference on IT-Incidents Management and IT-Forensics (IMF 2008)},
	Volume = {140},
	Year = {2008}}

@proceedings{SADFE-2009a-Proceedings,
	Address = {Berkeley, California, USA},
	Bibsource = {DBLP, http://dblp.uni-trier.de},
	Booktitle = {Proceedings of the Fourth International IEEE Workshop on Systematic Approaches to Digital Forensic Engineering (SADFE 2009)},
	Isbn = {978-0-7695-3792-4},
	Month = may # {~21},
	Publisher = {IEEE Computer Society},
	Title = {Proceedings of the Fourth International IEEE Workshop on Systematic Approaches to Digital Forensic Engineering (SADFE 2009)},
	Year = {2009}}

@proceedings{WDFIA-2010a-Proceedings,
	Address = {Port Elizabeth, South Africa},
	Booktitle = {Proceedings of the Fifth Annual Workshop on Digital Forensics and Incident Analysis (WDFIA)},
	Date-Added = {2011-05-13 15:54:22 +1200},
	Date-Modified = {2011-05-13 15:55:34 +1200},
	Month = {17--18~} # may,
	Title = {Proceedings of the Fifth Annual Workshop on Digital Forensics and Incident Analysis (WDFIA)},
	Year = {2010}}

Ignore Space Show notes View DP_2017/DP.tex 0 → 100644
\documentclass[12pt]{article} \usepackage[margin=1in]{geometry} \usepackage{fontspec} \usepackage{harvard} \setmainfont{Minion Pro} \setmonofont{Letter Gothic 12 Pitch} \title{Automatic classification of \\ old digital document formats} \author{Nigel Stanger \and Brendon Woodford \and William Sanson} \pagestyle{empty} \begin{document} \maketitle \thispagestyle{empty} \begin{abstract} It’s now common to be unable to open older digital documents because either the creating software has been discontinued, or it no longer supports that format. Worse, modern versions of software may open old documents but lose elements of the original (e.g., formatting). More precise identification of the software version that created a file would enable better recovery or migration of the file. This paper describes preliminary work on attempts to extract such information from old Microsoft Word documents. \end{abstract} \section{Introduction} The world is awash with digital documents going back several decades, with many of significant historical, cultural, scientific, or legal importance. Most people will probably access only very recent documents during normal activities, but there are many reasons to access older digital documents, such as studying the works of important writers \cite{Kolowich.S-2009a-Archiving}, re-analysing old research data using new methods \cite{Mount.R-2009a-Data,Pringle.H-2010a-NASA}, and finding new evidence for a “cold case” through forensic examination. However, digital documents tend to become progressively more difficult to open over time as the software that created them evolves, or is even discontinued. Even documents younger than 20 years are not safe: e.g., Microsoft Word 2013 cannot open documents created by Word 95 or earlier \cite{Microsoft-2014a-Word2013}, and it can be hard to even find a computer that can run such “antique” software. Vinton \citeasnoun{Cerf.V-2015a-Digital}, one of the creators of the Internet, recently warned that we are in danger of a “forgotten generation, or even a forgotten century” due to this “bit rot”. A less obvious, but in many ways more significant issue is that even when modern software \emph{can} open an old document, it may not accurately reproduce the document’s original form due to changes in functionality. This is analogous to human languages, where grammar and meaning change significantly over time. A person who knows only modern English will struggle to accurately comprehend Chaucer. Similarly, Word 2013 may struggle to accurately interpret a document created by Word 98. This issue arises much sooner with digital documents due to the rapid pace of software evolution, and may manifest as anything from subtle layout changes through to entire elements (e.g., graphics) being omitted. Thus, when we open an old document with modern software \emph{we cannot guarantee that it truly represents the original in both appearance and content}. This is a significant issue with historical or cultural material \cite{Kolowich.S-2009a-Archiving}, and is extremely dangerous in a forensic or legal context, where the ability to accurately reproduce a document in its original form may be crucial \cite{Gillespie.J-2004a-Coping}. Imagine, for example, if the Treaty of Waitangi or the Declaration of Independence were originally created in digital form, but when opened 20 years later, important parts were either not displayed at all or were differently formatted. This could completely change the meaning of the document. It is therefore essential from a preservation perspective to open a digital document with the same—or the nearest possible—version of the software that was used to create it. Unfortunately, identifying the correct version is not always a simple task for older digital documents. There are several resources for identifying document formats and extracting useful metadata from them, including the Unix \texttt{file} tool, \citeasnoun{JHOVE-2009a}, DROID \cite{Brown.A-2006a-Automatic}, the UK National Archives’ PRONOM database \cite{Brown.A-2006a-PRONOM}, and the \possessivecite{NatLib.NZ-2007a-Metadata} Metadata Extraction Tool. Most of these use well-known patterns or “signatures” specific to particular document formats. Signature-based methods can typically identify at least the broad class of document format (e.g., Microsoft Word), and can sometimes be more specific (e.g., Word 6/95 vs.\ Word 97–2003). They cannot however identify the specific software sub-version used (e.g., Word 95 version 1.1, or even Word 6 vs.\ Word 95), except in very limited cases. This is because the key differences across software versions are more likely related to the functionality offered (e.g., a new version of a word processor might add “tables”) than to the document format, which may be the same across several different software versions. Using features or characteristics indicating specific functionality to identify the range of possible software versions is a classification problem that may be amneable to automated machine learning. Machine learning techniques has previously been used in digital forensics to identify document formats, but only in the contexts of more reliably identifying the \emph{general type} of a document \cite{Mokhov.S-2008a-File} rather than which specific software version created it, and identifying the format of file \emph{fragments} rather than complete documents \cite{Li.Q-2010a-SVM,Roussev.V-2009a-File}. We therefore set out to determine whether it was possible to apply similar techniques to Microsoft Word documents in “.doc” format. Unfortunately, this proved to be a tougher problem than we initially expected. While we were able to get extract somewhat more version information from these documents than is possible with existing tools, we were unable to identify useful features for machine learning due to the complexity of the file format. \section{Method} \section{Results} \section{Conclusion and further work} \bibliographystyle{dcu} \bibliography{DP} \end{document}

Ignore Space Show notes View

DP_2017/DP.tex 0 → 100644

\documentclass[12pt]{article}

\usepackage[margin=1in]{geometry}
\usepackage{fontspec}
\usepackage{harvard}

\setmainfont{Minion Pro}
\setmonofont{Letter Gothic 12 Pitch}

\title{Automatic classification of \\ old digital document formats}
\author{Nigel Stanger \and Brendon Woodford \and William Sanson}

\pagestyle{empty}

\begin{document}

\maketitle
\thispagestyle{empty}

\begin{abstract}
It’s now common to be unable to open older digital documents because either the creating software has been discontinued, or it no longer supports that format. Worse, modern versions of software may open old documents but lose elements of the original (e.g., formatting). More precise identification of the software version that created a file would enable better recovery or migration of the file. This paper describes preliminary work on attempts to extract such information from old Microsoft Word documents.
\end{abstract}

\section{Introduction}

The world is awash with digital documents going back several decades, with many of significant historical, cultural, scientific, or legal importance. Most people will probably access only very recent documents during normal activities, but there are many reasons to access older digital documents, such as studying the works of important writers \cite{Kolowich.S-2009a-Archiving}, re-analysing old research data using new methods \cite{Mount.R-2009a-Data,Pringle.H-2010a-NASA}, and finding new evidence for a “cold case” through forensic examination. However, digital documents tend to become progressively more difficult to open over time as the software that created them evolves, or is even discontinued. Even documents younger than 20 years are not safe: e.g., Microsoft Word 2013 cannot open documents created by Word 95 or earlier \cite{Microsoft-2014a-Word2013}, and it can be hard to even find a computer that can run such “antique” software. Vinton \citeasnoun{Cerf.V-2015a-Digital}, one of the creators of the Internet, recently warned that we are in danger of a “forgotten generation, or even a forgotten century” due to this “bit rot”.

A less obvious, but in many ways more significant issue is that even when modern software \emph{can} open an old document, it may not accurately reproduce the document’s original form due to changes in functionality. This is analogous to human languages, where grammar and meaning change significantly over time. A person who knows only modern English will struggle to accurately comprehend Chaucer. Similarly, Word 2013 may struggle to accurately interpret a document created by Word 98. This issue arises much sooner with digital documents due to the rapid pace of software evolution, and may manifest as anything from subtle layout changes through to entire elements (e.g., graphics) being omitted. Thus, when we open an old document with modern software \emph{we cannot guarantee that it truly represents the original in both appearance and content}. This is a significant issue with historical or cultural material \cite{Kolowich.S-2009a-Archiving}, and is extremely dangerous in a forensic or legal context, where the ability to accurately reproduce a document in its original form may be crucial \cite{Gillespie.J-2004a-Coping}. Imagine, for example, if the Treaty of Waitangi or the Declaration of Independence were originally created in digital form, but when opened 20 years later, important parts were either not displayed at all or were differently formatted. This could completely change the meaning of the document.

It is therefore essential from a preservation perspective to open a digital document with the same—or the nearest possible—version of the software that was used to create it. Unfortunately, identifying the correct version is not always a simple task for older digital documents. There are several resources for identifying document formats and extracting useful metadata from them, including the Unix \texttt{file} tool, \citeasnoun{JHOVE-2009a}, DROID \cite{Brown.A-2006a-Automatic}, the UK National Archives’ PRONOM database \cite{Brown.A-2006a-PRONOM}, and the \possessivecite{NatLib.NZ-2007a-Metadata} Metadata Extraction Tool. Most of these use well-known patterns or “signatures” specific to particular document formats. Signature-based methods can typically identify at least the broad class of document format (e.g., Microsoft Word), and can sometimes be more specific (e.g., Word 6/95 vs.\ Word 97–2003). They cannot however identify the specific software sub-version used (e.g., Word 95 version 1.1, or even Word 6 vs.\ Word 95), except in very limited cases. This is because the key differences across software versions are more likely related to the functionality offered (e.g., a new version of a word processor might add “tables”) than to the document format, which may be the same across several different software versions.

Using features or characteristics indicating specific functionality to identify the range of possible software versions is a classification problem that may be amneable to automated machine learning. Machine learning techniques has previously been used in digital forensics to identify document formats, but only in the contexts of more reliably identifying the \emph{general type} of a document \cite{Mokhov.S-2008a-File} rather than which specific software version created it, and identifying the format of file \emph{fragments} rather than complete documents \cite{Li.Q-2010a-SVM,Roussev.V-2009a-File}. We therefore set out to determine whether it was possible to apply similar techniques to Microsoft Word documents in “.doc” format. Unfortunately, this proved to be a tougher problem than we initially expected. While we were able to get extract somewhat more version information from these documents than is possible with existing tools, we were unable to identify useful features for machine learning due to the complexity of the file format.

\section{Method}

\section{Results}

\section{Conclusion and further work}

\bibliographystyle{dcu}
\bibliography{DP}

\end{document}

Ignore Space Show notes View DP_2017/Project_Summary.docx 0 → 100644
Not supported

Ignore Space Show notes View DP_2017/Sanson_William_Supervisors_Completion_Report_2016_17.docx 0 → 100644
Not supported

Ignore Space Show notes View Koli_2017/.gitignore 0 → 100644
BDL_ERD.pdf Koli_2017_Stanger.pdf data acmart.pdf acmguide.pdf Koli_2017_submitted_version.pdf

Ignore Space Show notes View Koli_2017/ACM-Reference-Format.bst 0 → 100644
%%% --BibTeX-- %%% ==================================================================== %%% @BibTeX-style-file{ %%% author = "Nelson H. F. Beebe, Boris Veytsman and Gerald Murray", %%% version = "2.1", %%% date = "14 June 2017", %%% filename = "ACM-Reference-Format.bst", %%% email = "borisv@lk.net, boris@varphi.com", %%% codetable = "ISO/ASCII", %%% keywords = "ACM Transactions bibliography style; BibTeX", %%% license = "public domain", %%% supported = "yes", %%% abstract = "", %%% } %%% ==================================================================== %%% Revision history: see source in git ENTRY { address advisor archiveprefix author booktitle chapter city date edition editor eprint howpublished institution journal key month note number organization pages primaryclass publisher school series title type volume year % New keys recognized issue % UTAH: used in, e.g., ACM SIGSAM Bulletin and ACM Communications in Computer Algebra articleno eid day % UTAH: needed for newspapers, weeklies, bi-weeklies doi % UTAH url % UTAH bookpages % UTAH numpages lastaccessed % UTAH: used only for @Misc{...} coden % UTAH isbn % UTAH isbn-13 % UTAH issn % UTAH lccn % UTAH } {} { label.year extra.label sort.year sort.label basic.label.year} INTEGERS { output.state before.all mid.sentence after.sentence after.block } INTEGERS { show-isbn-10-and-13 } % initialized below in begin.bib INTEGERS { nameptr namesleft numnames } INTEGERS { multiresult } INTEGERS { len } INTEGERS { last.extra.num } STRINGS { s t t.org u } STRINGS { last.label next.extra } STRINGS { p1 p2 p3 page.count } FUNCTION { not } { { #0 } { #1 } if$ } FUNCTION { and } { 'skip$ { pop$ #0 } if$ } FUNCTION { or } { { pop$ #1 } 'skip$ if$ } FUNCTION { dump.stack.1 } { duplicate$ "STACK[top] = [" swap$ * "]" * warning$ } FUNCTION { dump.stack.2 } { duplicate$ "STACK[top ] = [" swap$ * "]" * warning$ swap$ duplicate$ "STACK[top-1] = [" swap$ * "]" * warning$ swap$ } FUNCTION { empty.or.unknown } { %% Examine the top stack entry, and push 1 if it is empty, or %% consists only of whitespace, or is a string beginning with two %% queries (??), and otherwise, push 0. %% %% This function provides a replacement for empty$, with the %% convenient feature that unknown values marked by two leading %% queries are treated the same as missing values, and thus, do not %% appear in the output .bbl file, and yet, their presence in .bib %% file(s) serves to mark values which are temporarily missing, but %% are expected to be filled in eventually once more data is %% obtained. The TeX User Group and BibNet bibliography archives %% make extensive use of this practice. %% %% An empty string cannot serve the same purpose, because just as in %% statistics data processing, an unknown value is not the same as an %% empty value. %% %% At entry: stack = ... top:[string] %% At exit: stack = ... top:[0 or 1] duplicate$ empty$ { pop$ #1 } { #1 #2 substring$ "??" = } if$ } FUNCTION { writeln } { %% In BibTeX style files, the sequences %% %% ... "one" "two" output %% ... "one" "two" output.xxx %% %% ship "one" to the output file, possibly following by punctuation, %% leaving the stack with %% %% ... "two" %% %% There is thus a one-string lag in output processing that must be %% carefully handled to avoid duplicating a string in the output %% file. Unless otherwise noted, all output.xxx functions leave %% just one new string on the stack, and that model should be born %% in mind when reading or writing function code. %% %% BibTeX's asynchronous buffering of output from strings from the %% stack is confusing because newline$ bypasses the buffer. It %% would have been so much easier for newline to be a character %% rather than a state of the output-in-progress. %% %% The documentation in btxhak.dvi is WRONG: it says %% %% newline$ Writes onto the bbl file what's accumulated in the %% output buffer. It writes a blank line if and only %% if the output buffer is empty. Since write$ does %% reasonable line breaking, you should use this %% function only when you want a blank line or an %% explicit line break. %% %% write$ Pops the top (string) literal and writes it on the %% output buffer (which will result in stuff being %% written onto the bbl file when the buffer fills %% up). %% %% Examination of the BibTeX source code shows that write$ does %% indeed behave as claimed, but newline$ sends a newline character %% directly to the output file, leaving the stack unchanged. The %% first line "Writes onto ... buffer." is therefore wrong. %% %% The original BibTeX style files almost always use "write$ newline$" %% in that order, so it makes sense to hide that pair in a private %% function like this one, named after a statement in Pascal, %% the programming language embedded in the BibTeX Web program. write$ % output top-of-stack string newline$ % immediate write of newline (not via stack) } FUNCTION { init.state.consts } { #0 'before.all := #1 'mid.sentence := #2 'after.sentence := #3 'after.block := } FUNCTION { output.nonnull } { % Stack in: ... R S T Stack out: ... R T File out: S<comma><space> 's := output.state mid.sentence = { ", " * write$ } { output.state after.block = { add.period$ writeln "\newblock " write$ } { output.state before.all = { write$ } { add.period$ " " * write$ } if$ } if$ mid.sentence 'output.state := } if$ s } FUNCTION { output.nonnull.dot.space } { % Stack in: ... R S T Stack out: ... R T File out: S<dot><space> 's := output.state mid.sentence = % { "<DEBUG output.nonnull.dot.space>. " * write$ } { ". " * write$ } { output.state after.block = { add.period$ writeln "\newblock " write$ } { output.state before.all = { write$ } { add.period$ " " * write$ } if$ } if$ mid.sentence 'output.state := } if$ s } FUNCTION { output.nonnull.remove } { % Stack in: ... R S T Stack out: ... R T File out: S<space> 's := output.state mid.sentence = { " " * write$ } { output.state after.block = { add.period$ writeln "\newblock " write$ } { output.state before.all = { write$ } { add.period$ " " * write$ } if$ } if$ mid.sentence 'output.state := } if$ s } FUNCTION { output.nonnull.removenospace } { % Stack in: ... R S T Stack out: ... R T File out: S 's := output.state mid.sentence = { "" * write$ } { output.state after.block = { add.period$ writeln "\newblock " write$ } { output.state before.all = { write$ } { add.period$ " " * write$ } if$ } if$ mid.sentence 'output.state := } if$ s } FUNCTION { output } { % discard top token if empty, else like output.nonnull duplicate$ empty.or.unknown 'pop$ 'output.nonnull if$ } FUNCTION { output.dot.space } { % discard top token if empty, else like output.nonnull.dot.space duplicate$ empty.or.unknown 'pop$ 'output.nonnull.dot.space if$ } FUNCTION { output.removenospace } { % discard top token if empty, else like output.nonnull.removenospace duplicate$ empty.or.unknown 'pop$ 'output.nonnull.removenospace if$ } FUNCTION { output.check } { % like output, but warn if key name on top-of-stack is not set 't := duplicate$ empty.or.unknown { pop$ "empty " t * " in " * cite$ * warning$ } 'output.nonnull if$ } FUNCTION { bibinfo.output.check } { % like output.check, adding bibinfo field 't := duplicate$ empty.or.unknown { pop$ "empty " t * " in " * cite$ * warning$ } { "\bibinfo{" t "}{" * * swap$ * "}" * output.nonnull } if$ } FUNCTION { output.check.dot.space } { % like output.dot.space, but warn if key name on top-of-stack is not set 't := duplicate$ empty.or.unknown { pop$ "empty " t * " in " * cite$ * warning$ } 'output.nonnull.dot.space if$ } FUNCTION { fin.block } { % functionally, but not logically, identical to fin.entry add.period$ writeln } FUNCTION { fin.entry } { add.period$ writeln } FUNCTION { new.sentence } { % update sentence state, with neither output nor stack change output.state after.block = 'skip$ { output.state before.all = 'skip$ { after.sentence 'output.state := } if$ } if$ } FUNCTION { fin.sentence } { add.period$ write$ new.sentence "" } FUNCTION { new.block } { output.state before.all = 'skip$ { after.block 'output.state := } if$ } FUNCTION { output.coden } % UTAH { % output non-empty CODEN as one-line sentence (stack untouched) coden empty.or.unknown { } { "\showCODEN{" coden * "}" * writeln } if$ } FUNCTION { format.articleno } { articleno empty.or.unknown not eid empty.or.unknown not and { "Both articleno and eid are defined for " cite$ * warning$ } 'skip$ if$ articleno empty.or.unknown eid empty.or.unknown and { "" } { numpages empty.or.unknown { "articleno or eid field, but no numpages field, in " cite$ * warning$ } { } if$ eid empty.or.unknown { "Article \bibinfo{articleno}{" articleno * "}" * } { "Article \bibinfo{articleno}{" eid * "}" * } if$ } if$ } FUNCTION { format.year } { % push year string or "[n. d.]" onto output stack %% Because year is a mandatory field, we always force SOMETHING %% to be output "\bibinfo{year}{" year empty.or.unknown { "[n. d.]" } { year } if$ * "}" * } FUNCTION { format.day.month } { % push "day month " or "month " or "" onto output stack day empty.or.unknown { month empty.or.unknown { "" } { "\bibinfo{date}{" month * "} " } if$ } { month empty.or.unknown { "" } { "\bibinfo{date}{" day " " * month * "} " } if$ } if$ } FUNCTION { format.day.month.year } % UTAH { % if month is empty, push "" else push "(MON.)" or "(DD MON.)" % Needed for frequent periodicals: 2008. ... New York Times C-1, C-2, C-17 (23 Oct.) % acm-.bst addition: prefix parenthesized date string with % ", Article nnn " articleno empty.or.unknown eid empty.or.unknown and { "" } { output.state after.block = {", " format.articleno * } { format.articleno } if$ } if$ " (" * format.day.month * format.year * ")" * } FUNCTION { output.day.month.year } % UTAH { % if month is empty value, do nothing; else output stack top and % leave with new top string "(MON.)" or "(DD MON.)" % Needed for frequent periodicals: 2008. ... New York Times C-1, C-2, C-17 (23 Oct.) format.day.month.year output.nonnull.remove } FUNCTION { strip.doi } % UTAH { % Strip any Web address prefix to recover the bare DOI, leaving the % result on the output stack, as recommended by CrossRef DOI % documentation. % For example, reduce "http://doi.acm.org/10.1145/1534530.1534545" to % "10.1145/1534530.1534545". That is later typeset and displayed as % doi:10.1145/1534530.1534545 as the LAST item in the reference list % entry. Publisher Web sites wrap this with a suitable link to a real % URL to resolve the DOI, and the master https://doi.org/ address is % preferred, since publisher-specific URLs can disappear in response % to economic events. All journals are encouraged by the DOI % authorities to use that typeset format and link procedures for % uniformity across all publications that include DOIs in reference % lists. % The numeric prefix is guaranteed to start with "10.", so we use % that as a test. % 2017-02-04 Added stripping of https:// (Boris) doi #1 #3 substring$ "10." = { doi } { doi 't := % get modifiable copy of DOI % Change https:// to http:// to strip both prefixes (BV) t #1 #8 substring$ "https://" = { "http://" t #9 t text.length$ #8 - substring$ * 't := } { } if$ t #1 #7 substring$ "http://" = { t #8 t text.length$ #7 - substring$ 't := "INTERNAL STYLE-FILE ERROR" 's := % search for next "/" and assign its suffix to s { t text.length$ } { t #1 #1 substring$ "/" = { % save rest of string as true DOI (should be 10.xxxx/yyyy) t #2 t text.length$ #1 - substring$ 's := "" 't := % empty string t terminates the loop } { % discard first character and continue loop: t <= substring(t,2,last) t #2 t text.length$ #1 - substring$ 't := } if$ } while$ % check for valid DOI (should be 10.xxxx/yyyy) s #1 #3 substring$ "10." = { } { "unrecognized DOI substring " s * " in DOI value [" * doi * "]" * warning$ } if$ s % push the stripped DOI on the output stack } { "unrecognized DOI value [" doi * "]" * warning$ doi % push the unrecognized original DOI on the output stack } if$ } if$ } % % Change by BV: added standard prefix to URL % FUNCTION { output.doi } % UTAH { % output non-empty DOI as one-line sentence (stack untouched) doi empty.or.unknown { } { %% NB: We want URLs at beginning of line to reduce likelihood of %% BibTeX's nasty line wrapping after column 79, which then requires %% manual (or automated) editing of the .bbl file to repair. %% The \url{} macro strips percent-newlines, and is thus safe in %% the presence of the line wrapping, but \path\|...\| and %% \verb\|...\| do not. "\showDOI{%" writeln "\url{https://doi.org/" strip.doi * "}}" * writeln } if$ } FUNCTION { output.isbn } % UTAH { % output non-empty ISBN-10 and/or ISBN-13 as one-line sentences (stack untouched) show-isbn-10-and-13 { %% show both 10- and 13-digit ISBNs isbn empty.or.unknown { } { "\showISBNx{" isbn * "}" * writeln } if$ isbn-13 empty.or.unknown { } { "\showISBNxiii{" isbn-13 * "}" * writeln } if$ } { %% show 10-digit ISBNs only if 13-digit ISBNs not available isbn-13 empty.or.unknown { isbn empty.or.unknown { } { "\showISBNx{" isbn * "}" * writeln } if$ } { "\showISBNxiii{" isbn-13 * "}" * writeln } if$ } if$ } FUNCTION { output.issn } % UTAH { % output non-empty ISSN as one-line sentence (stack untouched) issn empty.or.unknown { } { "\showISSN{" issn * "}" * writeln } if$ } FUNCTION { output.issue } { % output non-empty issue number as a one-line sentence (stack untouched) issue empty.or.unknown { } { "Issue " issue * "." * writeln } if$ } FUNCTION { output.lccn } % UTAH { % return with stack untouched lccn empty.or.unknown { } { "\showLCCN{" lccn * "}" * writeln } if$ } FUNCTION { output.note } % UTAH { % return with stack empty note empty.or.unknown { } { "\shownote{" note add.period$ * "}" * writeln } if$ } FUNCTION { output.note.check } % UTAH { % return with stack empty note empty.or.unknown { "empty note in " cite$ * warning$ } { "\shownote{" note add.period$ * "}" * writeln } if$ } FUNCTION { output.eprint } % { % return with stack empty eprint empty.or.unknown { } { "\showeprint" archiveprefix empty.or.unknown { } { "[" archiveprefix "l" change.case$ "]" * * * } if$ "{" * primaryclass empty.or.unknown { } { primaryclass "/" * } if$ eprint "}" * writeln } if$ } % % Changes by BV 2011/04/15. Do not output % url if doi is defined % FUNCTION { output.url } % UTAH { % return with stack untouched % output URL and associated lastaccessed fields doi empty.or.unknown { url empty.or.unknown { } { %% NB: We want URLs at beginning of line to reduce likelihood of %% BibTeX's nasty line wrapping after column 79, which would require %% manual (or automated) editing of the .bbl file to repair. However, %% the \url{} macro handles the unwrapping job automatically. "\showURL{%" writeln lastaccessed empty.or.unknown { "" } { "Retrieved " lastaccessed * " from " * } if$ %% The URL field may contain a semicolon-separated list of Web %% addresses, and we locate and wrap each of them in \url{...}. %% The simplistic approach of putting the entire list into the %% macro argument is that the semicolons are typeset in a %% typewriter font, and no space follows them. %% %% We therefore replace the original code %% "\url{" * url * "}}" * writeln %% with this character-at-a-time loop: "\url{" * url 't := % get modifiable copy of URL list { t text.length$ } { t #1 #1 substring$ ";" = { % then split argument at separator "};" * writeln "\url{" } { % else concatenate nonblank character to argument t #1 #1 substring$ " " = { } { t #1 #1 substring$ * } if$ } if$ t #2 t text.length$ #1 - substring$ 't := } while$ "}}" * writeln } if$ } { } if$ } FUNCTION { output.year.check } { % warn if year empty, output top string and leave " YEAR<label>" on stack in mid-sentence year empty.or.unknown { "empty year in " cite$ * warning$ write$ " \bibinfo{year}{[n. d.]}" "\natexlab{" extra.label * "}" * * mid.sentence 'output.state := } { write$ " \bibinfo{year}{" year * "}" * "\natexlab{" extra.label * "}" * * mid.sentence 'output.state := } if$ } FUNCTION { le } { %% test whether first number is less than or equal to second number %% stack in: n1 n2 %% stack out: if n1 <= n2 then 1 else 0 %% "DEBUG: le " cite$ * warning$ > { #0 } { #1 } if$ } FUNCTION { ge } { %% test whether first number is greater than or equal to second number %% stack in: n1 n2 %% stack out: if n1 >= n2 then 1 else 0 %% "DEBUG: ge " cite$ * warning$ < { #0 } { #1 } if$ } FUNCTION { is.leading.digit } { %% test whether first character of string is a digit %% stack in: string %% stack out: if first-char-is-digit then 1 else 0 #1 #1 substring$ % replace string by string[1:1] duplicate$ % string[1:1] string[1:1] chr.to.int$ "0" chr.to.int$ swap$ le % "0" <= string[1:1] --> 0-or-1 swap$ % 0-or-1 string[1:1] chr.to.int$ "9" chr.to.int$ le % string[1:1} <= "9" --> 0-or-1 and } FUNCTION { skip.digits } { %% skip over leading digits in string %% stack in: string %% stack out: rest-of-string leading-digits %% "DEBUG: enter skip.digits " cite$ * warning$ %% dump.stack.1 duplicate$ 't := 't.org := "" 'u := { t text.length$ } { %% "=================DEBUG: skip.digits t = [" t * "]" * warning$ t is.leading.digit { t #2 t text.length$ #1 - substring$ } { t 'u := "" } if$ 't := } while$ u % rest of string t.org #1 t.org text.length$ u text.length$ - substring$ % leading digits %% "DEBUG: t.org = [" t.org * "]" * warning$ %% "DEBUG: u = [" u * "]" * warning$ %% dump.stack.2 %% "DEBUG: leave skip.digits " cite$ * warning$ } FUNCTION { skip.nondigits } { %% skip over leading nondigits in string %% stack in: string %% stack out: rest-of-string %% "DEBUG: enter skip.nondigits " cite$ * warning$ 't := "" 'u := { t text.length$ } { %% "=================DEBUG: skip.nondigits t = [" t * "]" * warning$ t is.leading.digit { t 'u := "" } { t #2 t text.length$ #1 - substring$ } if$ 't := } while$ u % rest of string %% dump.stack.1 %% "DEBUG: leave skip.nondigits " cite$ * warning$ } FUNCTION { parse.next.number } { %% stack in: string %% stack out: rest-of-string next-numeric-part-of-string %% Example: %% stack in: "123:1--123:59" %% stack out: ":1--123:59" "123" 's := s skip.nondigits 's := s skip.digits } FUNCTION { reduce.pages.to.page.count } { %% Stack in: arbitrary-and-unused %% Stack out: unchanged %% %% For the new-style pagination with article number and numpages or %% pages, we expect to have BibTeX entries containing something like %% articleno = "17", %% pages = "1--23", %% with output "Article 17, 23 pages", %% or %% articleno = "17", %% numpages = "23", %% with output "Article 17, 23 pages", %% or %% articleno = "17", %% pages = "17:1--17:23", %% with output "Article 17, 23 pages", %% %% If articleno is missing or empty, then we should output "1--23", %% "23" (with a warning of a missing articleno), or "17:1--17:23", %% respectively. %% "DEBUG: enter reduce.pages.to.page.count " cite$ * warning$ %% "DEBUG: pages = [" pages * "]" * warning$ pages parse.next.number 'p1 := parse.next.number 'p2 := parse.next.number 'p3 := parse.next.number 'page.count := duplicate$ empty.or.unknown { } { duplicate$ "unexpected trailing garbage [" swap$ * "] after n:p1--n:p2 in pages = [" * pages * "] in " * cite$ * warning$ } if$ pop$ %% "DEBUG: reduce.pages.to.page.count: " %% " p1 = " p1 * * %% " p2 = " p2 * * %% " p3 = " p3 * * %% " p4 = " page.count * * %% " in " cite$ * * warning$ p1 p3 = p2 "1" = and numpages empty.or.unknown and { "INFO: reduced pages = [" pages * "] to numpages = [" * page.count * "]" * warning$ } { numpages empty.or.unknown { pages } { numpages } if$ 'page.count := } if$ p1 "1" = p3 empty.or.unknown and numpages empty.or.unknown and { p2 'page.count := "INFO: reduced pages = [" pages * "] to numpages = [" * page.count * "]" * warning$ } { numpages empty.or.unknown { pages } { numpages } if$ 'page.count := } if$ %% "DEBUG: leave reduce.pages.to.page.count " cite$ * warning$ } FUNCTION { new.block.checkb } { % issue a new.block only if at least one of top two stack strings is not empty empty.or.unknown swap$ empty.or.unknown and 'skip$ 'new.block if$ } FUNCTION { field.or.null } { % convert empty value to null string, else return value duplicate$ empty.or.unknown { pop$ "" } 'skip$ if$ } FUNCTION { emphasize } { % emphasize a non-empty top string on the stack (WITHOUT italic correction) duplicate$ empty.or.unknown { pop$ "" } { "{\em " swap$ * "}" * } if$ } FUNCTION { emphasize.with.italic.correction } { % convert empty string to null string, or emphasize with a trailing italic correction duplicate$ empty.or.unknown { pop$ "" } { "{\em " swap$ * "\/}" * } if$ } FUNCTION { comma } { % convert empty string to null string, or brace string and add trailing comma duplicate$ empty.or.unknown { pop$ "" } { "{" swap$ * "}," * } if$ } FUNCTION { format.names } { % Format bibliographical entries with the first author last name first, % and subsequent authors with initials followed by last name. % All names are formatted in this routine. 's := #1 'nameptr := % nameptr = 1; s num.names$ 'numnames := % numnames = num.name$(s); numnames 'namesleft := { namesleft #0 > } { nameptr #1 = %NO: BAD ORDER: {"{" s nameptr "{ff~}{ll}{, jj}{, vv}" format.name$ * "}" * 't := } %NO: BAD ORDER: {"{" s nameptr "{ff~}{ll}{, jj}{, vv}" format.name$ * "}" * 't := } {"\bibinfo{person}{" s nameptr "{ff }{vv }{ll}{, jj}" format.name$ * "}" * 't := } {"\bibinfo{person}{" s nameptr "{ff }{vv }{ll}{, jj}" format.name$ * "}" * 't := } if$ nameptr #1 > { namesleft #1 > { ", " * t * } { numnames #2 > { "," * } 'skip$ if$ t "\bibinfo{person}{others}" = { " {et~al\mbox{.}}" * } % jrh: avoid spacing problems { " {and} " * t * } % from Chicago Manual of Style if$ } if$ } 't if$ nameptr #1 + 'nameptr := % nameptr += 1; namesleft #1 - 'namesleft := % namesleft =- 1; } while$ } FUNCTION { my.full.label } { 's := #1 'nameptr := % nameptr = 1; s num.names$ 'numnames := % numnames = num.name$(s); numnames 'namesleft := { namesleft #0 > } { s nameptr "{vv~}{ll}" format.name$ 't := % get the next name nameptr #1 > { namesleft #1 > { ", " * t * } { numnames #2 > { "," * } 'skip$ if$ t "others" = { " et~al\mbox{.}" * } % jrh: avoid spacing problems { " and " * t * } % from Chicago Manual of Style if$ } if$ } 't if$ nameptr #1 + 'nameptr := % nameptr += 1; namesleft #1 - 'namesleft := % namesleft =- 1; } while$ } FUNCTION { format.names.fml } { % Format names in "familiar" format, with first initial followed by % last name. Like format.names, ALL names are formatted. % jtb: The names are NOT put in small caps 's := #1 'nameptr := % nameptr = 1; s num.names$ 'numnames := % numnames = num.name$(s); numnames 'namesleft := { namesleft #0 > } { "\bibinfo{person}{" s nameptr "{ff~}{vv~}{ll}{, jj}" format.name$ * "}" * 't := nameptr #1 > { namesleft #1 > { ", " * t * } { numnames #2 > { "," * } 'skip$ if$ t "\bibinfo{person}{others}" = { " {et~al\mbox{.}}" * } { " {and} " * t * } if$ } if$ } 't if$ nameptr #1 + 'nameptr := % nameptr += 1; namesleft #1 - 'namesleft := % namesleft =- 1; } while$ } FUNCTION { format.authors } { author empty.or.unknown { "" } { "\bibfield{author}{" author format.names add.period$ * "}" } % jtb: add period if none before if$ } FUNCTION { format.key } { empty.or.unknown { key field.or.null } { "" } if$ } FUNCTION { format.no.key } { empty.or.unknown { "" } { "" } if$ } FUNCTION { format.editors.fml } { % Format editor names for use in the "in" types: inbook, incollection, % inproceedings: first initial, then last names. When editors are the % LABEL for an entry, then format.editor is used which lists editors % by last name first. editor empty.or.unknown { "" } { "\bibfield{editor}{" editor format.names.fml "}" * editor num.names$ #1 > { " (Eds.)" * } { " (Ed.)" * } if$ } if$ } FUNCTION { format.editors } { % format editor names for use in labels, last names first. editor empty.or.unknown { "" } { "\bibfield{editor}{" editor format.names * "}" * editor num.names$ #1 > { " (Eds.)." * } { " (Ed.)." * } if$ } if$ } FUNCTION { format.articletitle } { title empty.or.unknown { "" } % Use this to preserve lettercase in titles: { "\showarticletitle{" title * "}" * } % Use this for downcase title style: % { \showarticletitle{" title "t" change.case$ * "}" * } if$ } FUNCTION { format.title } { title empty.or.unknown { "" } % Use this to preserve lettercase in titles: { "\bibinfo{title}{" title * "}" * } % Use this for downcase title style: % { title "t" change.case$ } if$ } FUNCTION { n.dashify } { 't := "" { t empty.or.unknown not } { t #1 #1 substring$ "-" = { t #1 #2 substring$ "--" = not { "--" * t #2 global.max$ substring$ 't := } { { t #1 #1 substring$ "-" = } { "-" * t #2 global.max$ substring$ 't := } while$ } if$ } { t #1 #1 substring$ * t #2 global.max$ substring$ 't := } if$ } while$ } FUNCTION { format.btitle } { "\bibinfo{booktitle}{" edition empty.or.unknown { title emphasize } { title empty.or.unknown { title emphasize } % jtb: what is this supposed to do ?!? { "{\em " title * "\/} (\bibinfo{edition}{" * edition "l" change.case$ * "} ed.)" * } % jtb: no parens for ed. if$ } if$ * "}" * } FUNCTION { format.emphasize.booktitle } { % push "" or "{\em booktitle}" or "{\em booktitle}, (second ed.)" on stack "\bibinfo{booktitle}{" edition empty.or.unknown { booktitle emphasize } { booktitle empty.or.unknown { "" } { "{\em " booktitle * "} (\bibinfo{edition}{" * edition "l" change.case$ * "} ed.)" * } if$ } if$ * "}" * } FUNCTION { format.city } { % jtb: if the preceding string (the title of the conference) is non-empty, % jtb: append the location, otherwise leave empty (so as to trigger the % jtb: error message in output.check duplicate$ empty.or.unknown { } { city empty.or.unknown { date empty.or.unknown { } { " (" * date * ")" * } if$ } { date empty.or.unknown { " (" * city * ")" * } { " (" * city * ", " * date * ")" * } if$ } if$ } if$ } FUNCTION { tie.or.space.connect } { duplicate$ text.length$ #3 < { "~" } { " " } if$ swap$ * * } FUNCTION { either.or.check } { empty.or.unknown 'pop$ { "can't use both " swap$ * " fields in " * cite$ * warning$ } if$ } FUNCTION { format.bvolume } { % jtb: If there is a series, this is added and the volume trails after it. % jtb: Otherwise, "Vol" is Capitalized. volume empty.or.unknown { "" } { series empty.or.unknown { "Vol.~\bibinfo{volume}{" volume "}" * } { "\bibinfo{series}{" series "}, " * "Vol.~\bibinfo{volume}{" volume "}" * * } if$ "volume and number" number either.or.check } if$ } FUNCTION { format.bvolume.noseries } { volume empty.or.unknown { "" } { "Vol.~\bibinfo{volume}{" volume "}" * "volume and number" number either.or.check } if$ } FUNCTION { format.series } { series empty.or.unknown {""} {" {\em (\bibinfo{series}{" * series "})}" } if$ } FUNCTION { format.number.series } { volume empty.or.unknown { number empty.or.unknown { volume empty.or.unknown { "" } { series empty.or.unknown { "" } { " (\bibinfo{series}{" series "})" * } if$ } if$ } % { series field.or.null } { output.state mid.sentence = { "Number" } % gnp - changed to mixed case always { "Number" } if$ number tie.or.space.connect series empty.or.unknown { "there's a number but no series in " cite$ * warning$ } { " in \bibinfo{series}{" * series * "}" * } if$ } if$ } { "" } if$ } FUNCTION { multi.page.check } { 't := #0 'multiresult := { multiresult not t empty.or.unknown not and } { t #1 #1 substring$ duplicate$ "-" = swap$ duplicate$ "," = swap$ "+" = or or { #1 'multiresult := } { t #2 global.max$ substring$ 't := } if$ } while$ multiresult } FUNCTION { format.pages } { pages empty.or.unknown { "" } { "\bibinfo{pages}{" pages multi.page.check { pages n.dashify } % gnp - removed () % jtb: removed pp. { pages } if$ * "}" * } if$ } FUNCTION { format.pages.check.without.articleno } { %% format pages field only if articleno is absent %% Stack out: pages-specification numpages missing$ pages missing$ and { "page numbers missing in both pages and numpages fields in " cite$ * warning$ } { } if$ articleno empty.or.unknown eid empty.or.unknown and { pages missing$ { numpages } { format.pages } if$ } { "" } if$ } FUNCTION { format.pages.check } { pages empty.or.unknown { "page numbers missing in " cite$ * warning$ "" } { pages n.dashify } if$ } FUNCTION { format.bookpages } { bookpages empty.or.unknown { "" } { bookpages "book pages" tie.or.space.connect } if$ } FUNCTION { format.named.pages } { pages empty.or.unknown { "" } { format.pages "pages" tie.or.space.connect } if$ } % % Changed by Boris Veytsman, 2011-03-13 % Now the word "pages" is printed even if % there field pages is not empty. % FUNCTION { format.page.count } { page.count empty.or.unknown { "" } { "\bibinfo{numpages}{" page.count * "}~pages" * } if$ } FUNCTION { format.articleno.numpages } { %% There are seven possible outputs, depending on which fields are set. %% %% These four are handled here: %% %% articleno, numpages, pages -> "Article articleno-value, numpages-value pages" %% articleno, numpages -> "Article articleno-value, numpages-value pages" %% articleno, pages -> "Article articleno-value, reduced-pages-value pages" %% articleno -> "Article articleno-value" and warn about missing numpages %% %% The remaining three have already been handled by %% format.pages.check.without.articleno: %% %% numpages, pages -> "pages-value" %% numpages -> "numpages-value" %% pages -> "pages-value" articleno empty.or.unknown eid empty.or.unknown and { numpages empty.or.unknown { } { "numpages field, but no articleno or eid field, in " cite$ * warning$ } if$ "" } { numpages empty.or.unknown { pages empty.or.unknown { "articleno or eid, but no pages or numpages field in " cite$ * warning$ "" 'page.count := } { reduce.pages.to.page.count } if$ } { numpages 'page.count := } if$ %% The Article number is now handled in format.day.month.year because %% ACM prefers the style "Digital Libraries 12, 3, Article 5 (July 2008)" %% over "Digital Libraries 12, 3 (July 2008), Article 5" %% format.articleno output format.page.count } if$ } FUNCTION {calc.format.page.count} { numpages empty.or.unknown { pages empty.or.unknown { "" 'page.count := } { reduce.pages.to.page.count } if$ } { numpages 'page.count := } if$ format.page.count } FUNCTION { format.journal.volume.number.day.month.year } { % By Young (and Spencer) % GNP - fixed bugs with missing volume, number, and/or pages % % Format journal, volume, number, pages for article types. % journal empty.or.unknown { "no journal in " cite$ * warning$ "" } % { journal emphasize.with.italic.correction } { "\bibinfo{journal}{" journal "Journal of the ACM" = { "{\it J. ACM}" } { journal "American Mathematical Society Translations" = { "{\it Amer. Math. Soc. Transl.}" } { journal "Bulletin of the American Mathematical Society" = { "{\it Bull. Amer. Math. Soc.}" } { journal "Proceedings of the American Mathematical Society" = { "{\it Proc. Amer. Math. Soc.}" } { journal "Transactions of the American Mathematical Society" = { "{\it Trans. Amer. Math. Soc.}" } { journal "Communications of the {ACM}" = { "{\it Commun. {ACM}}" } { journal "{ACM} Computing Surveys" = { "{\it Comput. Surveys}" } { journal "{ACM} Transactions on Mathematical Software" = { "{\it {ACM} Trans. Math. Software}" } { journal "{ACM} {SIGNUM} Newsletter" = { "{\it {ACM} {SIGNUM} Newslett.}" } { journal "American Journal of Sociology" = { "{\it Amer. J. Sociology}" } { journal "Journal of the American Statistical Association" = { "{\it J. Amer. Statist. Assoc.}" } { journal "Applied Mathematics and Computation" = { "{\it Appl. Math. Comput.}" } { journal "American Mathematical Monthly" = { "{\it Amer. Math. Monthly}" } { journal "British Journal of Mathematical and Statistical Psychology" = { "{\it Brit. J. Math. Statist. Psych.}" } { journal "Canadian Mathematical Bulletin" = { "{\it Canad. Math. Bull.}" } { journal "Journal of Computational and Applied Mathematics" = { "{\it J. Comput. Appl. Math.}" } { journal "Journal of Computational Physics" = { "{\it J. Comput. Phys.}" } { journal "Computers and Structures" = { "{\it Comput. \& Structures}" } { journal "The Computer Journal" = { "{\it Comput. J.}" } { journal "Journal of Computer and System Sciences" = { "{\it J. Comput. System Sci.}" } { journal "Contemporary Mathematics" = { "{\it Contemp. Math.}" } { journal "Crelle's Journal" = { "{\it Crelle's J.}" } { journal "Giornale di Mathematiche" = { "{\it Giorn. Mat.}" } { journal "{IEEE} Transactions on Computers" = { "{\it {IEEE} Trans. Comput.}" } { journal "{IEEE} Transactions on Automatic Control" = { "{\it {IEEE} Trans. Automat. Control}" } { journal "Proceedings of the {IEEE}" = { "{\it Proc. {IEEE}}" } { journal "{IEEE} Transactions on Aerospace and Electronic Systems" = { "{\it {IEEE} Trans. Aerospace Electron. Systems}" } { journal "{IMA} Journal of Numerical Analysis" = { "{\it {IMA} J. Numer. Anal.}" } { journal "Information Processing Letters" = { "{\it Inform. Process. Lett.}" } { journal "Journal of the Institute of Mathematics and its Applications" = { "{\it J. Inst. Math. Appl.}" } { journal "International Journal of Control" = { "{\it Internat. J. Control}" } { journal "International Journal for Numerical Methods in Engineering" = { "{\it Internat. J. Numer. Methods Engrg.}" } { journal "International Journal of Supercomputing Applications" = { "{\it Internat. J. Supercomputing Applic.}" } { journal "Journal of Research of the National Bureau of Standards" = { "{\it J. Res. Nat. Bur. Standards}" } { journal "Linear Algebra and its Applications" = { "{\it Linear Algebra Appl.}" } { journal "Journal of Mathematical Analysis and Applications" = { "{\it J. Math. Anal. Appl.}" } { journal "Mathematische Annalen" = { "{\it Math. Ann.}" } { journal "Journal of Mathematical Physics" = { "{\it J. Math. Phys.}" } { journal "Mathematics of Computation" = { "{\it Math. Comp.}" } { journal "Mathematica Scandinavica" = { "{\it Math. Scand.}" } { journal "Mathematical Tables and Other Aids to Computation" = { "{\it Math. Tables Aids Comput.}" } { journal "Numerische Mathematik" = { "{\it Numer. Math.}" } { journal "Pacific Journal of Mathematics" = { "{\it Pacific J. Math.}" } { journal "Journal of Parallel and Distributed Computing" = { "{\it J. Parallel and Distrib. Comput.}" } { journal "Parallel Computing" = { "{\it Parallel Comput.}" } { journal "Philosophical Magazine" = { "{\it Philos. Mag.}" } { journal "Proceedings of the National Academy of Sciences of the USA" = { "{\it Proc. Nat. Acad. Sci. U. S. A.}" } { journal "Quarterly Journal of Mathematics, Oxford, Series (2)" = { "{\it Quart. J. Math. Oxford Ser. (2)}" } { journal "Quarterly of Applied Mathematics" = { "{\it Quart. Appl. Math.}" } { journal "Review of the International Statisical Institute" = { "{\it Rev. Inst. Internat. Statist.}" } { journal "Journal of the Society for Industrial and Applied Mathematics" = { "{\it J. Soc. Indust. Appl. Math.}" } { journal "Journal of the Society for Industrial and Applied Mathematics, Series B, Numerical Analysis" = { "{\it J. Soc. Indust. Appl. Math. Ser. B Numer. Anal.}" } { journal "{SIAM} Journal on Algebraic and Discrete Methods" = { "{\it {SIAM} J. Algebraic Discrete Methods}" } { journal "{SIAM} Journal on Applied Mathematics" = { "{\it {SIAM} J. Appl. Math.}" } { journal "{SIAM} Journal on Computing" = { "{\it {SIAM} J. Comput.}" } { journal "{SIAM} Journal on Matrix Analysis and Applications" = { "{\it {SIAM} J. Matrix Anal. Appl.}" } { journal "{SIAM} Journal on Numerical Analysis" = { "{\it {SIAM} J. Numer. Anal.}" } { journal "{SIAM} Review" = { "{\it {SIAM} Rev.}" } { journal "{SIAM} Journal on Scientific and Statistical Computing" = { "{\it {SIAM} J. Sci. Statist. Comput.}" } { journal "Software Practice and Experience" = { "{\it Software Prac. Experience}" } { journal "Statistical Science" = { "{\it Statist. Sci.}" } { journal "{USSR} Computational Mathematics and Mathematical Physics" = { "{\it {U. S. S. R.} Comput. Math. and Math. Phys.}" } { journal "Journal of {VLSI} and Computer Systems" = { "{\it J. {VLSI} Comput. Syst.}" } { journal "Zeitschrift fur Angewandte Mathematik und Mechanik" = { "{\it Z. Angew. Math. Mech.}" } { journal "Zeitschrift fur Angewandte Mathematik und Physik" = { "{\it Z. Angew. Math. Phys.}" } { journal "ACM Computing Surveys" = { "{\it Comput. Surveys}" } { journal "ACM Transactions on Mathematical Software" = { "{\it ACM Trans. Math. Software}" } { journal "ACM {SIGNUM} Newsletter" = { "{\it ACM {SIGNUM} Newslett.}" } { journal "IEEE Transactions on Computers" = { "{\it IEEE Trans. Comput.}" } { journal "IEEE Transactions on Automatic Control" = { "{\it IEEE Trans. Automat. Control}" } { journal "Proceedings of the IEEE" = { "{\it Proc. IEEE}" } { journal "IEEE Transactions on Aerospace and Electronic Systems" = { "{\it IEEE Trans. Aerospace Electron. Systems}" } { journal "IMA Journal of Numerical Analysis" = { "{\it IMA J. Numer. Anal.}" } { journal "SIAM Journal on Algebraic and Discrete Methods" = { "{\it SIAM J. Algebraic Discrete Methods}" } { journal "SIAM Journal on Applied Mathematics" = { "{\it SIAM J. Appl. Math.}" } { journal "SIAM Journal on Computing" = { "{\it SIAM J. Comput.}" } { journal "SIAM Journal on Matrix Analysis and Applications" = { "{\it SIAM J. Matrix Anal. Appl.}" } { journal "SIAM Journal on Numerical Analysis" = { "{\it SIAM J. Numer. Anal.}" } { journal "SIAM Review" = { "{\it SIAM Rev.}" } { journal "SIAM Journal on Scientific and Statistical Computing" = { "{\it SIAM J. Sci. Statist. Comput.}" } { journal "USSR Computational Mathematics and Mathematical Physics" = { "{\it U. S. S. R. Comput. Math. and Math. Phys.}" } { journal "Journal of VLSI and Computer Systems" = { "{\it J. VLSI Comput. Syst.}" } { journal "Communications of the ACM" = { "{\it Commun. ACM}" } %% If no match with cases needing special handling, just output journal name { journal emphasize.with.italic.correction } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ } if$ * "}" * } if$ number empty.or.unknown { volume empty.or.unknown { "no number and no volume in " cite$ * warning$ "" * } { " " * " \bibinfo{volume}{" * volume * "}" * } if$ } { volume empty.or.unknown { "unusual to have number, but no volume, for " cite$ * warning$ " \bibinfo{number}{" * number * "}" * } { " \bibinfo{volume}{" * volume * "}, \bibinfo{number}{" * number * "}" } if$ } if$ after.block 'output.state := % Sometimes proceedings are published in journals % In this case we do not want to put year, day and month here type$ "inproceedings" = { } {format.day.month.year } if$ } FUNCTION { format.chapter.pages } { chapter empty.or.unknown 'format.pages { type empty.or.unknown { "Chapter" } % gnp - changed to mixed case { type "t" change.case$ } if$ chapter tie.or.space.connect pages empty.or.unknown {"page numbers missing in " cite$ * warning$} % gnp - added check { ", " * format.pages * } if$ } if$ } FUNCTION { format.in.emphasize.booktitle } { % jtb: format for collections or proceedings not appearing in a journal booktitle empty.or.unknown { "" } { "In " format.emphasize.booktitle * } if$ } FUNCTION { format.in.booktitle } { % jtb: format for proceedings appearing in a journal booktitle empty.or.unknown { "" } { "In \bibinfo{booktitle}{" booktitle * "}" * } if$ } FUNCTION { format.in.ed.booktitle } { booktitle empty.or.unknown { "" } { editor empty.or.unknown { "In " format.emphasize.booktitle * } % jtb: swapped editor location { "In " format.emphasize.booktitle * ", " * format.editors.fml * } if$ } if$ } FUNCTION { format.thesis.type } { % call with default type on stack top type empty.or.unknown 'skip$ % use default type { pop$ % discard default type % NO: it is silly to have to brace protect every degree type!: type "t" change.case$ type } if$ } FUNCTION { format.tr.number } { "\bibinfo{type}{" type empty.or.unknown { "{T}echnical {R}eport" } 'type if$ "}" * * number empty.or.unknown { "t" change.case$ } %% LOOKS BAD: { "." * number tie.or.space.connect } %% Prefer "Research report RJ687." to "Research report. RJ687." { number tie.or.space.connect } if$ } FUNCTION { format.advisor } { advisor empty.or.unknown { "" } { "Advisor(s) " advisor * } if$ } FUNCTION { format.article.crossref } { "See" "\citeN{" * crossref * "}" * } FUNCTION { format.crossref.editor } { editor #1 "{vv~}{ll}" format.name$ editor num.names$ duplicate$ #2 > { pop$ " et~al\mbox{.}" * } % jrh: avoid spacing problems { #2 < 'skip$ { editor #2 "{ff }{vv }{ll}{ jj}" format.name$ "others" = { " et~al\mbox{.}" * } % jrh: avoid spacing problems { " and " * editor #2 "{vv~}{ll}" format.name$ * } if$ } if$ } if$ } FUNCTION { format.book.crossref } { volume empty.or.unknown { "empty volume in " cite$ * "'s crossref of " * crossref * warning$ "In " } { "Volume" volume tie.or.space.connect % gnp - changed to mixed case " of " * } if$ editor empty.or.unknown editor field.or.null author field.or.null = or { key empty.or.unknown { series empty.or.unknown { "need editor, key, or series for " cite$ * " to crossref " * crossref * warning$ "" * } { "{\em " * series * "\/}" * } if$ } { key * } if$ } { format.crossref.editor * } if$ " \citeN{" * crossref * "}" * } FUNCTION { format.incoll.inproc.crossref } { "See" " \citeN{" * crossref * "}" * } FUNCTION { format.lab.names } { % format.lab.names: % % determines "short" names for the abbreviated author information. % "Long" labels are created in calc.label, using the routine my.full.label % to format author and editor fields. % % There are 4 cases for labels. (n=3 in the example) % a) one author Foo % b) one to n Foo, Bar and Baz % c) use of "and others" Foo, Bar et al. % d) more than n Foo et al. 's := s num.names$ 'numnames := numnames #2 > % change number to number of others allowed before % forcing "et al". { s #1 "{vv~}{ll}" format.name$ " et~al\mbox{.}" * } % jrh: \mbox{} added { numnames #1 - 'namesleft := #2 'nameptr := s #1 "{vv~}{ll}" format.name$ { namesleft #0 > } { nameptr numnames = { s nameptr "{ff }{vv }{ll}{ jj}" format.name$ "others" = { " et~al\mbox{.}" * } % jrh: avoid spacing problems { " and " * s nameptr "{vv~}{ll}" format.name$ * } if$ } { ", " * s nameptr "{vv~}{ll}" format.name$ * } if$ nameptr #1 + 'nameptr := namesleft #1 - 'namesleft := } while$ } if$ } FUNCTION { author.key.label } { author empty.or.unknown { key empty.or.unknown { "no key, author in " cite$ * warning$ cite$ #1 #3 substring$ } 'key if$ } { author format.lab.names } if$ } FUNCTION { editor.key.organization.label } { % added - gnp. Provide label formatting by organization if editor is null. editor empty.or.unknown { organization empty.or.unknown { key empty.or.unknown { "no key, editor or organization in " cite$ * warning$ cite$ #1 #3 substring$ } 'key if$ } { organization } if$ } { editor format.lab.names } if$ } FUNCTION { author.editor.key.label } { author empty.or.unknown { editor empty.or.unknown { key empty.or.unknown { "no key, author, or editor in " cite$ * warning$ cite$ #1 #3 substring$ } 'key if$ } { editor format.lab.names } if$ } { author format.lab.names } if$ } FUNCTION { author.editor.key.organization.label } { % added - gnp. Provide label formatting by organization if author is null. author empty.or.unknown { editor empty.or.unknown { organization empty.or.unknown { key empty.or.unknown { "no key, author, editor or organization in " cite$ * warning$ cite$ #1 #3 substring$ } 'key if$ } { organization } if$ } { editor format.lab.names } if$ } { author format.lab.names } if$ } % Calculate label and leave it on stack FUNCTION { calc.basic.label } { type$ "book" = type$ "inbook" = or type$ "article" = or 'author.editor.key.label { type$ "proceedings" = type$ "periodical" = or 'editor.key.organization.label { type$ "manual" = 'author.editor.key.organization.label 'author.key.label if$ } if$ } if$ duplicate$ year empty.or.unknown { "[n. d.]" } { year field.or.null purify$ #-1 #4 substring$} if$ * 'basic.label.year := } FUNCTION { calc.label } { % Changed - GNP. See also author.editor.organization.sort, editor.organization.sort % Form label for BibTeX entry. The classification of which fields are used % for which type of entry (book, inbook, etc.) are taken from alpha.bst. % The change here from newapa is to also include organization as a % citation label if author or editor is missing. calc.basic.label author empty.or.unknown % generate the full label citation information. { editor empty.or.unknown { organization empty.or.unknown { key empty.or.unknown { "no author, editor, organization, or key in " cite$ * warning$ "??" } { key } if$ } { organization } if$ } { editor my.full.label } if$ } { author my.full.label } if$ % leave label on the stack, to be popped when required. "}{" * swap$ * "}{" * % year field.or.null purify$ #-1 #4 substring$ * % % save the year for sort processing afterwards (adding a, b, c, etc.) % year empty.or.unknown { "[n. d.]" } { year field.or.null purify$ #-1 #4 substring$} if$ 'label.year := } FUNCTION { output.bibitem } { newline$ "\bibitem[\protect\citeauthoryear{" write$ calc.label write$ sort.year write$ "}]%" writeln " {" write$ cite$ write$ "}" writeln "" before.all 'output.state := } FUNCTION { output.issue.doi.coden.isxn.lccn.url.eprint } { % enter and return with stack empty %% We switch now from buffered output to output of complete lines, so %% that the Issue .. URL data have their own lines, and are less likely %% to be line-wrapped by BibTeX's short-sighted algorithm, which wraps %% lines longer than 79 characters, backtracking to what it thinks is %% a break point in the string. Any such wrapping MUST be undone to %% prevent percent-newline from appearing in DOIs and URLs. The %% output data are intentionally wrapped in \showxxx{} macros at %% beginning of line, and that supply their own punctuation (if they %% are not defined to suppress output entirely), to make it easier for %% other software to recover them from .bbl files. %% %% It also makes it possible to later change the macro definitions %% to suppress particular output values, or alter their appearance. %% %% Note that it is possible for theses, technical reports, and %% manuals to have ISBNs, and anything that has an ISBN may also %% have an ISSN. When there are no values for these keys, there %% is no output generated for them here. "\newblock" writeln after.block 'output.state := output.issue output.isbn output.coden % CODEN is functionally like ISSN, so output them sequentially output.issn output.lccn output.doi % DOI is ALWAYS last according to CrossRef DOI documentation output.eprint output.url % but ACM wants URL last } FUNCTION { output.issue.doi.coden.isxn.lccn.url.eprint.note } { % enter with stack empty, return with empty string on stack output.issue.doi.coden.isxn.lccn.url.eprint note empty.or.unknown { } { "\newblock" writeln output.note } if$ "" } FUNCTION { output.issue.doi.coden.isxn.lccn.url.eprint.note.check } { % enter with stack empty, return with empty string on stack output.issue.doi.coden.isxn.lccn.url.eprint note empty.or.unknown { } { "\newblock" writeln output.note.check } if$ "" } FUNCTION { article } { output.bibitem author empty.or.unknown { editor empty.or.unknown { "neither author and editor supplied for " cite$ * warning$ } { format.editors "editor" output.check } if$ } { format.authors "author" output.check } if$ author format.no.key output % added output.year.check % added new.block format.articletitle "title" output.check new.block howpublished empty.or.unknown { } { "\bibinfo{howpublished}{" howpublished "}" * * output } if$ crossref missing$ { format.journal.volume.number.day.month.year output} { "cross reference in @Article{...} is unusual" warning$ format.article.crossref output.nonnull } if$ format.pages.check.without.articleno output format.articleno.numpages output fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { book } { output.bibitem author empty.or.unknown { format.editors "author and editor" output.check } { format.authors output.nonnull crossref missing$ { "author and editor" editor either.or.check } 'skip$ if$ } if$ output.year.check % added new.block format.btitle "title" output.check crossref missing$ { new.sentence % jtb: start a new sentence for series/volume format.bvolume output new.block format.number.series output new.sentence publisher "publisher" bibinfo.output.check address "address" bibinfo.output.check % jtb: require address fin.sentence pages empty.or.unknown { format.bookpages } % use bookpages when pages empty { format.pages.check "pages" tie.or.space.connect } if$ output } { new.block format.book.crossref output.nonnull } if$ fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { booklet } { output.bibitem format.authors output author format.key output % added output.year.check % added new.block format.title "title" output.check new.block howpublished empty.or.unknown { } { "\bibinfo{howpublished}{" howpublished "}" * * output } if$ address output fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { inbook } { output.bibitem author empty.or.unknown { format.editors "author and editor" output.check } { format.authors output.nonnull crossref missing$ { "author and editor" editor either.or.check } 'skip$ if$ } if$ output.year.check % added new.block format.btitle "title" output.check crossref missing$ { new.sentence % jtb: start a new sentence for series/volume format.bvolume output new.block format.number.series output new.sentence publisher "publisher" bibinfo.output.check address "address" bibinfo.output.check % jtb: require address format.bookpages output format.chapter.pages "chapter and pages" output.check % jtb: moved from before publisher } { format.bookpages output format.chapter.pages "chapter and pages" output.check new.block format.book.crossref output.nonnull } if$ fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { incollection } { output.bibitem format.authors "author" output.check author format.key output % added output.year.check % added new.block format.articletitle "title" output.check new.block crossref missing$ { format.in.ed.booktitle "booktitle" output.check new.sentence % jtb: start a new sentence for series/volume format.bvolume output format.number.series output new.sentence publisher "publisher" bibinfo.output.check address "address" bibinfo.output.check % jtb: require address format.bookpages output format.chapter.pages output % gnp - was special.output.nonnull % left out comma before page numbers % jtb: moved from before publisher } { format.incoll.inproc.crossref output.nonnull format.chapter.pages output } if$ fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { inproceedings } { output.bibitem format.authors "author" output.check author format.key output % added output.year.check % added new.block format.articletitle "title" output.check howpublished empty.or.unknown { } { "\bibinfo{howpublished}{" howpublished "}" * * output.dot.space } if$ crossref missing$ { journal missing$ % jtb: proceedings appearing in journals { format.in.emphasize.booktitle format.city "booktitle" output.check.dot.space format.series output.removenospace format.editors.fml output % BV 2011/09/27 Moved dot to comma format.bvolume.noseries output new.sentence organization output publisher "publisher" bibinfo.output.check % jtb: require publisher (?) address "address" bibinfo.output.check % jtb: require address format.bookpages output } { format.in.booktitle format.city "booktitle" output.check format.editors.fml output new.sentence format.journal.volume.number.day.month.year output } if$ format.articleno output format.pages.check.without.articleno output } { format.incoll.inproc.crossref output.nonnull format.articleno output format.pages.check.without.articleno output } if$ format.articleno.numpages output fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { conference } { inproceedings } FUNCTION { manual } { output.bibitem author empty.or.unknown { editor empty.or.unknown { organization "organization" output.check organization format.key output } % if all else fails, use key { format.editors "author and editor" output.check } if$ } { format.authors output.nonnull } if$ output.year.check % added new.block format.btitle "title" output.check organization address new.block.checkb % jtb: back to normal style: organization, address organization "organization" output.check address output fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { mastersthesis } { output.bibitem format.authors "author" output.check author format.key output % added output.year.check % added new.block format.title emphasize "title" output.check % NB: ACM style requires emphasized thesis title new.block "\bibinfo{thesistype}{Master's\ thesis}" format.thesis.type output new.sentence school "school" bibinfo.output.check address empty.or.unknown { } { "\bibinfo{address}{" address * "}" * output } if$ new.block format.advisor output fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { misc } { output.bibitem format.authors "author" output.check author format.key output % added output.year.check % added title howpublished new.block.checkb format.title output new.block howpublished empty.or.unknown { } { "\bibinfo{howpublished}{" howpublished "}" * * output } if$ "" output.nonnull.dot.space output.day.month.year calc.format.page.count output fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { online } { misc } FUNCTION { game } { misc } FUNCTION { phdthesis } { output.bibitem format.authors "author" output.check author format.key output % added output.year.check % added new.block format.title emphasize "title" output.check % NB: ACM style requires emphasized thesis title new.block "\bibinfo{thesistype}{Ph.D. Dissertation}" format.thesis.type output new.sentence school "school" bibinfo.output.check address empty.or.unknown { } { "\bibinfo{address}{" address * "}" * output } if$ new.block format.advisor output fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION {format.date} { year empty.or.unknown { month empty.or.unknown { "" % output empty date if year/month both empty day empty.or.unknown { } { "there's a day but no month or year in " cite$ * warning$ } if$ } { "there's a month but no year in " cite$ * warning$ month day empty.or.unknown { } { " " * day * } if$ } if$ } { month empty.or.unknown { year % output only year if month empty day empty.or.unknown { } { "there's a day and year but no month in " cite$ * warning$ } if$ } { month " " * day empty.or.unknown { } { day * ", " * } if$ year * } if$ } if$ } FUNCTION {new.block.checka} { empty.or.unknown 'skip$ 'new.block if$ } FUNCTION { periodical } { output.bibitem editor empty.or.unknown { organization output } { format.editors output.nonnull } if$ new.block output.year.check new.sentence format.articletitle "title" output.check format.journal.volume.number.day.month.year output calc.format.page.count output fin.entry } FUNCTION { proceedings } { output.bibitem editor empty.or.unknown { organization output organization format.key output } % gnp - changed from author format.key { format.editors output.nonnull } if$ % author format.key output % gnp - removed (should be either % editor or organization output.year.check % added (newapa) new.block format.btitle format.city "title" output.check % jtb: added city new.sentence format.bvolume output format.number.series output new.sentence organization output % jtb: normal order: publisher, address publisher empty.or.unknown { } { "\bibinfo{publisher}{" publisher * "}" * output } if$ address empty.or.unknown { } { "\bibinfo{address}{" address * "}" * output } if$ fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { techreport } { output.bibitem format.authors "author" output.check author format.key output % added output.year.check % added new.block format.btitle "title" output.check new.block % format.tr.number output % jtb: moved month ... format.tr.number output new.sentence % Gerry - need dot 2011/09/28 institution "institution" bibinfo.output.check address empty.or.unknown { } { "\bibinfo{address}{" address "}" * * output } if$ new.sentence format.named.pages output % ACM omits year at end in transactions style % format.day.month.year output.nonnull.dot.space % jtb: ... to here (no parens) fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note fin.entry } FUNCTION { unpublished } { output.bibitem format.authors "author" output.check author format.key output % added output.year.check % added new.block format.title "title" output.check fin.sentence output.day.month.year % UTAH calc.format.page.count output fin.block output.issue.doi.coden.isxn.lccn.url.eprint.note.check fin.entry } FUNCTION { default.type } { misc } %%% ACM journal-style month definitions: full name if 1--5 letters, else %%% abbreviation of 3 or 4 characters and a dot MACRO {jan} {"Jan."} MACRO {feb} {"Feb."} MACRO {mar} {"March"} MACRO {apr} {"April"} MACRO {may} {"May"} MACRO {jun} {"June"} MACRO {jul} {"July"} MACRO {aug} {"Aug."} MACRO {sep} {"Sept."} MACRO {oct} {"Oct."} MACRO {nov} {"Nov."} MACRO {dec} {"Dec."} READ FUNCTION { sortify } { purify$ "l" change.case$ } FUNCTION { chop.word } { 's := 'len := s #1 len substring$ = { s len #1 + global.max$ substring$ } 's if$ } FUNCTION { sort.format.names } { 's := #1 'nameptr := "" s num.names$ 'numnames := numnames 'namesleft := { namesleft #0 > } { nameptr #1 > { " " * } 'skip$ if$ s nameptr "{vv{ } }{ll{ }}{ f{ }}{ jj{ }}" format.name$ 't := nameptr numnames = t "others" = and { " et~al" * } { t sortify * } if$ nameptr #1 + 'nameptr := namesleft #1 - 'namesleft := } while$ } FUNCTION { sort.format.title } { 't := "A " #2 "An " #3 "The " #4 t chop.word chop.word chop.word sortify #1 global.max$ substring$ } FUNCTION { author.sort } { author empty.or.unknown { key empty.or.unknown { "to sort, need author or key in " cite$ * warning$ "" } { key sortify } if$ } { author sort.format.names } if$ } FUNCTION { author.editor.sort } { author empty.or.unknown { editor empty.or.unknown { key empty.or.unknown { "to sort, need author, editor, or key in " cite$ * warning$ "" } { key sortify } if$ } { editor sort.format.names } if$ } { author sort.format.names } if$ } FUNCTION { editor.organization.sort } { % added - GNP. Stack editor or organization for sorting (from alpha.bst). % Unlike alpha.bst, we need entire names, not abbreviations editor empty.or.unknown { organization empty.or.unknown { key empty.or.unknown { "to sort, need editor, organization, or key in " cite$ * warning$ "" } { key sortify } if$ } { organization sortify } if$ } { editor sort.format.names } if$ } FUNCTION { author.editor.organization.sort } { % added - GNP. Stack author or organization for sorting (from alpha.bst). % Unlike alpha.bst, we need entire names, not abbreviations author empty.or.unknown { editor empty.or.unknown { organization empty.or.unknown { key empty.or.unknown { "to sort, need author, editor, or key in " cite$ * warning$ "" } { key sortify } if$ } { organization sortify } if$ } { editor sort.format.names } if$ } { author sort.format.names } if$ } FUNCTION { presort } { % Presort creates the bibentry's label via a call to calc.label, and then % sorts the entries based on entry type. Chicago.bst adds support for % including organizations as the sort key; the following is stolen from % alpha.bst. calc.label basic.label.year swap$ " " swap$ * * " " * sortify year field.or.null purify$ #-1 #4 substring$ * % add year " " * type$ "book" = type$ "inbook" = or type$ "article" = or 'author.editor.sort { type$ "proceedings" = type$ "periodical" = or 'editor.organization.sort { type$ "manual" = 'author.editor.organization.sort 'author.sort if$ } if$ } if$ #1 entry.max$ substring$ % added for newapa 'sort.label := % added for newapa sort.label % added for newapa * " " * title field.or.null sort.format.title * #1 entry.max$ substring$ 'sort.key$ := } ITERATE { presort } SORT % by label, year, author/editor, title FUNCTION { initialize.extra.label.stuff } { #0 int.to.chr$ 'last.label := "" 'next.extra := #0 'last.extra.num := } FUNCTION { forward.pass } { % Pass through all entries, comparing current entry to last one. % Need to concatenate year to the stack (done by calc.label) to determine % if two entries are the same (see presort) last.label % OLD:calc.label year field.or.null purify$ #-1 #4 substring$ * % add year % NEW: calc.basic.label year field.or.null purify$ #-1 #4 substring$ * % add year #1 entry.max$ substring$ = % are they equal? { last.extra.num #1 + 'last.extra.num := last.extra.num int.to.chr$ 'extra.label := } { "a" chr.to.int$ 'last.extra.num := "" 'extra.label := % OLD: calc.label year field.or.null purify$ #-1 #4 substring$ * % add year % NEW: calc.basic.label year field.or.null purify$ #-1 #4 substring$ * % add year #1 entry.max$ substring$ 'last.label := % assign to last.label } if$ } FUNCTION { reverse.pass } { next.extra "b" = { "a" 'extra.label := } 'skip$ if$ label.year extra.label * 'sort.year := extra.label 'next.extra := } EXECUTE {initialize.extra.label.stuff} ITERATE {forward.pass} REVERSE {reverse.pass} FUNCTION { bib.sort.order } { sort.label " " * year field.or.null sortify * " " * title field.or.null sort.format.title * #1 entry.max$ substring$ 'sort.key$ := } ITERATE { bib.sort.order } SORT % by sort.label, year, title --- giving final bib. order. FUNCTION { begin.bib } { %% Set to #0 show 13-digit ISBN in preference to 10-digit ISBN. %% Set to #1 to show both 10-digit and 13-digit ISBNs. #1 'show-isbn-10-and-13 := "%%% --BibTeX--" writeln "%%% Do NOT edit. File created by BibTeX with style" writeln "%%% ACM-Reference-Format-Journals [18-Jan-2012]." writeln "" writeln preamble$ empty.or.unknown 'skip$ { preamble$ writeln } if$ "\begin{thebibliography}{00}" writeln "" writeln "%%% ====================================================================" writeln "%%% NOTE TO THE USER: you can override these defaults by providing" writeln "%%% customized versions of any of these macros before the \bibliography" writeln "%%% command. Each of them MUST provide its own final punctuation," writeln "%%% except for \shownote{}, \showDOI{}, and \showURL{}. The latter two" writeln "%%% do not use final punctuation, in order to avoid confusing it with" writeln "%%% the Web address." writeln "%%%" writeln "%%% To suppress output of a particular field, define its macro to expand" writeln "%%% to an empty string, or better, \unskip, like this:" writeln "%%%" writeln "%%% \newcommand{\showDOI}[1]{\unskip} % LaTeX syntax" writeln "%%%" writeln "%%% \def \showDOI #1{\unskip} % plain TeX syntax" writeln "%%%" writeln "%%% ====================================================================" writeln "" writeln %% ACM publications do not use CODEN, ISSN, and LCCN data, so their default %% macro wrappers expand to \unskip, discarding their values and unwanted %% space. %% %% For other publications, prior definitions like these may be useful: %% %% Plain TeX: %% \def \showCODEN #1{CODEN #1.} %% \def \showISSN #1{ISSN #1.} %% \def \showLCCN #1{LCCN #1.} %% %% LaTeX: %% \newcommand{\showCODEN}[1]{CODEN #1.} %% \newcommand{\showISSN}[1]#1{ISSN #1.} %% \newcommand{\showLCCN}[1]{LCCN #1.} "\ifx \showCODEN \undefined \def \showCODEN #1{\unskip} \fi" writeln "\ifx \showDOI \undefined \def \showDOI #1{#1}\fi" writeln % ACM styles omit ISBNs, but they can be included by suitable definitions of % \showISBNx and \showISBNxiii before the .bbl file is read "\ifx \showISBNx \undefined \def \showISBNx #1{\unskip} \fi" writeln "\ifx \showISBNxiii \undefined \def \showISBNxiii #1{\unskip} \fi" writeln "\ifx \showISSN \undefined \def \showISSN #1{\unskip} \fi" writeln "\ifx \showLCCN \undefined \def \showLCCN #1{\unskip} \fi" writeln "\ifx \shownote \undefined \def \shownote #1{#1} \fi" writeln % NB: final period supplied by add.period$ above "\ifx \showarticletitle \undefined \def \showarticletitle #1{#1} \fi" writeln "\ifx \showURL \undefined \def \showURL {\relax} \fi" writeln "% The following commands are used for tagged output and should be " writeln "% invisible to TeX" writeln "\providecommand\bibfield[2]{#2}" writeln "\providecommand\bibinfo[2]{#2}" writeln "\providecommand\natexlab[1]{#1}" writeln "\providecommand\showeprint[2][]{arXiv:#2}" writeln } EXECUTE {begin.bib} EXECUTE {init.state.consts} ITERATE {call.type$} FUNCTION { end.bib } { newline$ "\end{thebibliography}" writeln } EXECUTE {end.bib}

Ignore Space Show notes View

Koli_2017/ACM-Reference-Format.bst 0 → 100644

%%% -*-BibTeX-*-
%%% ====================================================================
%%%  @BibTeX-style-file{
%%%     author          = "Nelson H. F. Beebe, Boris Veytsman and Gerald Murray",
%%%     version         = "2.1",
%%%     date            = "14 June 2017",
%%%     filename        = "ACM-Reference-Format.bst",
%%%     email           = "borisv@lk.net, boris@varphi.com",
%%%     codetable       = "ISO/ASCII",
%%%     keywords        = "ACM Transactions bibliography style; BibTeX",
%%%     license         = "public domain",
%%%     supported       = "yes",
%%%     abstract        = "",
%%%  }
%%% ====================================================================

%%% Revision history:  see source in git

ENTRY
  { address
    advisor
    archiveprefix
    author
    booktitle
    chapter
    city     
    date     
    edition
    editor
    eprint
    howpublished
    institution
    journal
    key
    month
    note
    number
    organization
    pages
    primaryclass
    publisher
    school
    series
    title
    type
    volume
    year
        % New keys recognized
        issue         % UTAH: used in, e.g., ACM SIGSAM Bulletin and ACM Communications in Computer Algebra
        articleno
	eid           
        day           % UTAH: needed for newspapers, weeklies, bi-weeklies
        doi           % UTAH
        url           % UTAH
        bookpages     % UTAH
        numpages
        lastaccessed  % UTAH: used only for @Misc{...}
        coden         % UTAH
        isbn          % UTAH
        isbn-13       % UTAH
        issn          % UTAH
        lccn          % UTAH
  }
  {}
  { label.year extra.label sort.year sort.label basic.label.year}

INTEGERS { output.state before.all mid.sentence after.sentence after.block }

INTEGERS { show-isbn-10-and-13 }  % initialized below in begin.bib

INTEGERS { nameptr namesleft numnames }

INTEGERS { multiresult }

INTEGERS { len }

INTEGERS { last.extra.num }

STRINGS { s t t.org u }

STRINGS { last.label next.extra }

STRINGS { p1 p2 p3 page.count }

FUNCTION { not }
{
    { #0 }
    { #1 }
  if$
}

FUNCTION { and }
{
    'skip$
    { pop$ #0 }
  if$
}

FUNCTION { or }
{
   { pop$ #1 }
    'skip$
  if$
}

FUNCTION { dump.stack.1 }
{
    duplicate$ "STACK[top] = [" swap$ * "]" * warning$
}

FUNCTION { dump.stack.2 }
{
    duplicate$ "STACK[top  ] = [" swap$ * "]" * warning$
    swap$
    duplicate$ "STACK[top-1] = [" swap$ * "]" * warning$
    swap$
}

FUNCTION { empty.or.unknown }
{
  %% Examine the top stack entry, and push 1 if it is empty, or
  %% consists only of whitespace, or is a string beginning with two
  %% queries (??), and otherwise, push 0.
  %%
  %% This function provides a replacement for empty$, with the
  %% convenient feature that unknown values marked by two leading
  %% queries are treated the same as missing values, and thus, do not
  %% appear in the output .bbl file, and yet, their presence in .bib
  %% file(s) serves to mark values which are temporarily missing, but
  %% are expected to be filled in eventually once more data is
  %% obtained.  The TeX User Group and BibNet bibliography archives
  %% make extensive use of this practice.
  %%
  %% An empty string cannot serve the same purpose, because just as in
  %% statistics data processing, an unknown value is not the same as an
  %% empty value.
  %%
  %% At entry: stack = ... top:[string]
  %% At exit:  stack = ... top:[0 or 1]

duplicate$ empty$
    { pop$ #1 }
    { #1 #2 substring$ "??" = }
  if$
}

FUNCTION { writeln }
{
  %% In BibTeX style files, the sequences
  %%
  %%     ... "one" "two" output
  %%     ... "one" "two" output.xxx
  %%
  %% ship "one" to the output file, possibly following by punctuation,
  %% leaving the stack with
  %%
  %%     ... "two"
  %%
  %% There is thus a one-string lag in output processing that must be
  %% carefully handled to avoid duplicating a string in the output
  %% file.  Unless otherwise noted, all output.xxx functions leave
  %% just one new string on the stack, and that model should be born
  %% in mind when reading or writing function code.
  %%
  %% BibTeX's asynchronous buffering of output from strings from the
  %% stack is confusing because newline$ bypasses the buffer.  It
  %% would have been so much easier for newline to be a character
  %% rather than a state of the output-in-progress.
  %%
  %% The documentation in btxhak.dvi is WRONG:  it says
  %%
  %%    newline$ Writes onto the bbl file what's accumulated in the
  %%             output buffer. It writes a blank line if and only
  %%             if the output buffer is empty. Since write$ does
  %%             reasonable line breaking, you should use this
  %%             function only when you want a blank line or an
  %%             explicit line break.
  %%
  %%    write$   Pops the top (string) literal and writes it on the
  %%             output buffer (which will result in stuff being
  %%             written onto the bbl file when the buffer fills
  %%             up).
  %%
  %% Examination of the BibTeX source code shows that write$ does
  %% indeed behave as claimed, but newline$ sends a newline character
  %% directly to the output file, leaving the stack unchanged.  The
  %% first line "Writes onto ... buffer." is therefore wrong.
  %%
  %% The original BibTeX style files almost always use "write$ newline$"
  %% in that order, so it makes sense to hide that pair in a private
  %% function like this one, named after a statement in Pascal,
  %% the programming language embedded in the BibTeX Web program.

write$                % output top-of-stack string
  newline$              % immediate write of newline (not via stack)
}

FUNCTION { init.state.consts }
{
  #0 'before.all :=
  #1 'mid.sentence :=
  #2 'after.sentence :=
  #3 'after.block :=
}

FUNCTION { output.nonnull }
{ % Stack in: ... R S T  Stack out: ... R T   File out: S<comma><space>
  's :=
  output.state mid.sentence =
    {
      ", " * write$
    }
    {
      output.state after.block =
        {
          add.period$ writeln
          "\newblock " write$
        }
        {
          output.state before.all =
            {
              write$
            }
            {
              add.period$ " " * write$
            }
          if$
        }
      if$
      mid.sentence 'output.state :=
    }
  if$
  s
}

FUNCTION { output.nonnull.dot.space }
{ % Stack in: ... R S T  Stack out: ... R T   File out: S<dot><space>
  's :=
  output.state mid.sentence =           % { "<DEBUG output.nonnull.dot.space>. " * write$ }
    {
      ". " * write$
    }
    {
      output.state after.block =
        {
          add.period$ writeln "\newblock " write$
        }
        {
          output.state before.all =
            {
              write$
            }
            {
              add.period$ " " * write$
            }
          if$
        }
      if$
      mid.sentence 'output.state :=
    }
  if$
  s
}

FUNCTION { output.nonnull.remove }
{ % Stack in: ... R S T  Stack out: ... R T   File out: S<space>
  's :=
  output.state mid.sentence =
    {
      " " * write$
    }
    {
      output.state after.block =
        {
          add.period$ writeln "\newblock " write$
        }
        {
          output.state before.all =
            {
              write$
            }
            {
              add.period$ " " * write$
            }
          if$
        }
      if$
      mid.sentence 'output.state :=
    }
  if$
  s
}

FUNCTION { output.nonnull.removenospace }
{ % Stack in: ... R S T  Stack out: ... R T   File out: S
  's :=
  output.state mid.sentence =
    {
      "" * write$
    }
    {
      output.state after.block =
        {
          add.period$ writeln "\newblock " write$
        }
        {
          output.state before.all =
            {
              write$
            }
            {
              add.period$ " " * write$
            }
          if$
        }
      if$
      mid.sentence 'output.state :=
    }
  if$
  s
}

FUNCTION { output }
{ % discard top token if empty, else like output.nonnull
  duplicate$ empty.or.unknown
    'pop$
    'output.nonnull
  if$
}

FUNCTION { output.dot.space }
{ % discard top token if empty, else like output.nonnull.dot.space
  duplicate$ empty.or.unknown
    'pop$
    'output.nonnull.dot.space
  if$
}

FUNCTION { output.removenospace }
{ % discard top token if empty, else like output.nonnull.removenospace
  duplicate$ empty.or.unknown
    'pop$
    'output.nonnull.removenospace
  if$
}

FUNCTION { output.check }
{ % like output, but warn if key name on top-of-stack is not set
  't :=
  duplicate$ empty.or.unknown
    { pop$ "empty " t * " in " * cite$ * warning$ }
    'output.nonnull
  if$
}

FUNCTION { bibinfo.output.check }
{ % like output.check, adding bibinfo field
  't :=
  duplicate$ empty.or.unknown
    { pop$ "empty " t * " in " * cite$ * warning$ }
    { "\bibinfo{" t "}{" * * swap$ * "}" *
      output.nonnull }
  if$
}

FUNCTION { output.check.dot.space }
{ % like output.dot.space, but warn if key name on top-of-stack is not set
  't :=
  duplicate$ empty.or.unknown
    { pop$ "empty " t * " in " * cite$ * warning$ }
    'output.nonnull.dot.space
  if$
}

FUNCTION { fin.block }
{ % functionally, but not logically, identical to fin.entry
   add.period$
   writeln
}

FUNCTION { fin.entry }
{
   add.period$
   writeln
}

FUNCTION { new.sentence }
{ % update sentence state, with neither output nor stack change
  output.state after.block =
    'skip$
    {
      output.state before.all =
        'skip$
        { after.sentence 'output.state := }
      if$
    }
  if$
}

FUNCTION { fin.sentence }
{
   add.period$
   write$
   new.sentence
   ""
}

FUNCTION { new.block }
{
  output.state before.all =
    'skip$
    { after.block 'output.state := }
  if$
}

FUNCTION { output.coden }       % UTAH
{ % output non-empty CODEN as one-line sentence (stack untouched)
  coden empty.or.unknown
    { }
    { "\showCODEN{" coden * "}" * writeln }
  if$
}

FUNCTION { format.articleno }
{
  articleno empty.or.unknown not eid empty.or.unknown not and
     { "Both articleno and eid are defined for " cite$ * warning$ }
     'skip$
  if$
  articleno empty.or.unknown eid empty.or.unknown and
     { "" }
     {
        numpages empty.or.unknown
          { "articleno or eid field, but no numpages field, in " 
            cite$ * warning$ }
          { }
        if$
        eid empty.or.unknown 
          { "Article \bibinfo{articleno}{" articleno * "}" * }
          { "Article \bibinfo{articleno}{" eid * "}" * }
        if$
     }
  if$
}

FUNCTION { format.year }
{ % push year string or "[n. d.]" onto output stack
  %% Because year is a mandatory field, we always force SOMETHING
  %% to be output
  "\bibinfo{year}{"
  year empty.or.unknown
    { "[n. d.]" }
    { year }
  if$
  *  "}" *
}

FUNCTION { format.day.month }
{ % push "day month " or "month " or "" onto output stack
  day empty.or.unknown
    {
      month empty.or.unknown
        { "" }
        { "\bibinfo{date}{" month * "} " *}
      if$
    }
    {
      month empty.or.unknown
        { "" }
        { "\bibinfo{date}{" day * " " * month * "} " *}
      if$
    }
  if$
}

FUNCTION { format.day.month.year }     % UTAH
{ % if month is empty, push "" else push "(MON.)" or "(DD MON.)"
  % Needed for frequent periodicals: 2008. ... New York Times C-1, C-2, C-17 (23 Oct.)
  % acm-*.bst addition: prefix parenthesized date string with
  % ", Article nnn "
  articleno empty.or.unknown eid empty.or.unknown and
    { "" }
    { output.state after.block =
       {", " format.articleno * }
       { format.articleno  }
      if$
    }
  if$
  " (" * format.day.month * format.year * ")" *
}

FUNCTION { output.day.month.year }     % UTAH
{ % if month is empty value, do nothing; else output stack top and
  % leave with new top string "(MON.)" or "(DD MON.)"
  % Needed for frequent periodicals: 2008. ... New York Times C-1, C-2, C-17 (23 Oct.)
  format.day.month.year
  output.nonnull.remove
}

FUNCTION { strip.doi } % UTAH
{ % Strip any Web address prefix to recover the bare DOI, leaving the
  % result on the output stack, as recommended by CrossRef DOI
  % documentation.
  % For example, reduce "http://doi.acm.org/10.1145/1534530.1534545" to
  % "10.1145/1534530.1534545".  That is later typeset and displayed as
  % doi:10.1145/1534530.1534545 as the LAST item in the reference list
  % entry.  Publisher Web sites wrap this with a suitable link to a real
  % URL to resolve the DOI, and the master https://doi.org/ address is
  % preferred, since publisher-specific URLs can disappear in response
  % to economic events.  All journals are encouraged by the DOI
  % authorities to use that typeset format and link procedures for
  % uniformity across all publications that include DOIs in reference
  % lists.
  % The numeric prefix is guaranteed to start with "10.", so we use
  % that as a test.
  % 2017-02-04 Added stripping of https:// (Boris)
  doi #1 #3 substring$ "10." =
    { doi }
    {
      doi 't :=  % get modifiable copy of DOI

% Change https:// to http:// to strip both prefixes (BV)

t #1 #8 substring$ "https://" =
        { "http://"  t #9 t text.length$ #8 - substring$ * 't := }
        { }
      if$

t #1 #7 substring$ "http://" =
        {
            t #8 t text.length$ #7 - substring$ 't :=

"INTERNAL STYLE-FILE ERROR" 's :=

% search for next "/" and assign its suffix to s

{ t text.length$ }
            {
              t #1 #1 substring$ "/" =
                {
                  % save rest of string as true DOI (should be 10.xxxx/yyyy)
                  t #2 t text.length$ #1 - substring$ 's :=
                  "" 't :=    % empty string t terminates the loop
                }
                {
                  % discard first character and continue loop: t <= substring(t,2,last)
                  t #2 t text.length$ #1 - substring$ 't :=
                }
              if$
            }
            while$

% check for valid DOI (should be 10.xxxx/yyyy)
            s #1 #3 substring$ "10." =
              { }
              { "unrecognized DOI substring " s * " in DOI value [" * doi * "]" * warning$ }
            if$

s   % push the stripped DOI on the output stack

}
        {
          "unrecognized DOI value [" doi * "]" * warning$
          doi   % push the unrecognized original DOI on the output stack
        }
      if$
    }
  if$
}

%
% Change by BV: added standard prefix to URL
%
FUNCTION { output.doi } % UTAH
{ % output non-empty DOI as one-line sentence (stack untouched)
  doi empty.or.unknown
    { }
    {
      %% NB: We want URLs at beginning of line to reduce likelihood of
      %% BibTeX's nasty line wrapping after column 79, which then requires
      %% manual (or automated) editing of the .bbl file to repair.
      %% The \url{} macro strips percent-newlines, and is thus safe in
      %% the presence of the line wrapping, but \path|...| and
      %% \verb|...| do not.
      "\showDOI{%" writeln
      "\url{https://doi.org/" strip.doi * "}}" * writeln
    }
  if$
}

FUNCTION { output.isbn }                % UTAH
{ % output non-empty ISBN-10 and/or ISBN-13 as one-line sentences (stack untouched)
  show-isbn-10-and-13
    {
      %% show both 10- and 13-digit ISBNs
      isbn empty.or.unknown
        { }
        {
          "\showISBNx{" isbn * "}" * writeln
        }
      if$
      isbn-13 empty.or.unknown
        { }
        {
          "\showISBNxiii{" isbn-13 * "}" * writeln
        }
      if$
    }
    {
      %% show 10-digit ISBNs only if 13-digit ISBNs not available
      isbn-13 empty.or.unknown
        {
          isbn empty.or.unknown
            { }
            {
              "\showISBNx{" isbn * "}" * writeln
            }
          if$
        }
        {
          "\showISBNxiii{" isbn-13 * "}" * writeln
        }
      if$
    }
  if$
}

FUNCTION { output.issn } % UTAH
{ % output non-empty ISSN as one-line sentence (stack untouched)
  issn empty.or.unknown
    { }
    { "\showISSN{" issn * "}" * writeln }
  if$
}

FUNCTION { output.issue }
{ % output non-empty issue number as a one-line sentence (stack untouched)
  issue empty.or.unknown
    { }
    { "Issue " issue * "." * writeln }
  if$
}

FUNCTION { output.lccn } % UTAH
{ % return with stack untouched
  lccn empty.or.unknown
    { }
    { "\showLCCN{" lccn * "}" * writeln }
  if$
}

FUNCTION { output.note } % UTAH
{ % return with stack empty
  note empty.or.unknown
    { }
    { "\shownote{" note add.period$ * "}" * writeln }
  if$
}

FUNCTION { output.note.check } % UTAH
{ % return with stack empty
  note empty.or.unknown
    { "empty note in " cite$ * warning$ }
    { "\shownote{" note add.period$ * "}" * writeln }
  if$
}

FUNCTION { output.eprint } %
{ % return with stack empty
  eprint empty.or.unknown
    { }
    { "\showeprint"
         archiveprefix empty.or.unknown
           { }
           { "["  archiveprefix "l" change.case$ "]" * * * }
         if$
         "{" *
         primaryclass empty.or.unknown
           { }
           { primaryclass "/" * *}
         if$
         eprint "}" * *
         writeln
    }
  if$
}

%
% Changes by BV 2011/04/15.  Do not output
% url if doi is defined
%
FUNCTION { output.url } % UTAH
{ % return with stack untouched
  % output URL and associated lastaccessed fields
  doi empty.or.unknown
  {
    url empty.or.unknown
      { }
      {
          %% NB: We want URLs at beginning of line to reduce likelihood of
          %% BibTeX's nasty line wrapping after column 79, which would require
          %% manual (or automated) editing of the .bbl file to repair.  However,
          %% the \url{} macro handles the unwrapping job automatically.
          "\showURL{%" writeln
          lastaccessed empty.or.unknown
            { "" }
            { "Retrieved " lastaccessed * " from " * }
          if$

%% The URL field may contain a semicolon-separated list of Web
          %% addresses, and we locate and wrap each of them in \url{...}.
          %% The simplistic approach of putting the entire list into the
          %% macro argument is that the semicolons are typeset in a
          %% typewriter font, and no space follows them.
          %%
          %% We therefore replace the original code
          %%    "\url{" * url * "}}" * writeln
          %% with this character-at-a-time loop:

"\url{" *

url 't :=                       % get modifiable copy of URL list

{ t text.length$ }
          {
            t #1 #1 substring$ ";" =
              {                         % then split argument at separator
                "};" * writeln
                "\url{"
              }
              {                         % else concatenate nonblank character to argument
                t #1 #1 substring$ " " =
                  { }
                  { t #1 #1 substring$ * }
                if$
              }
            if$

t #2 t text.length$ #1 - substring$ 't :=
          }
          while$

"}}" * writeln
      }
    if$
  }
  { }
  if$
}

FUNCTION { output.year.check }
{ % warn if year empty, output top string and leave " YEAR<label>" on stack in mid-sentence
  year empty.or.unknown
     { "empty year in " cite$ * warning$ 
       write$
       " \bibinfo{year}{[n. d.]}"  
       "\natexlab{" extra.label * "}" * *
       mid.sentence 'output.state :=
     }
     { write$
       " \bibinfo{year}{" year * "}"  *
       "\natexlab{" extra.label * "}" * *
       mid.sentence 'output.state :=
     }
  if$
}

FUNCTION { le }
{
  %% test whether first number is less than or equal to second number
  %% stack in:  n1 n2
  %% stack out: if n1 <= n2 then 1 else 0

%% "DEBUG: le " cite$ * warning$
  > { #0 } { #1 } if$
}

FUNCTION { ge }
{
  %% test whether first number is greater than or equal to second number
  %% stack in:  n1 n2
  %% stack out: if n1 >= n2 then 1 else 0

%% "DEBUG: ge " cite$ * warning$
  < { #0 } { #1 } if$
}

FUNCTION { is.leading.digit }
{
  %% test whether first character of string is a digit
  %% stack in:  string
  %% stack out: if first-char-is-digit then 1 else 0

#1 #1 substring$                      % replace string by string[1:1]
  duplicate$                            % string[1:1] string[1:1]
  chr.to.int$
  "0" chr.to.int$ swap$ le              % "0" <= string[1:1] --> 0-or-1
  swap$                                 % 0-or-1 string[1:1]
  chr.to.int$
  "9" chr.to.int$ le                    % string[1:1} <= "9" --> 0-or-1
  and
}

FUNCTION { skip.digits }
{
  %% skip over leading digits in string
  %% stack in:  string
  %% stack out: rest-of-string leading-digits

%% "DEBUG: enter skip.digits " cite$ * warning$

%% dump.stack.1

duplicate$
  't :=
  't.org :=
  "" 'u :=

{ t text.length$ }
  {
    %% "=================DEBUG: skip.digits   t = [" t * "]" * warning$
    t is.leading.digit
      { t #2 t text.length$ #1 - substring$ }
      {
        t 'u :=
        ""
      }
    if$
    't :=
  }
  while$

u                                                             % rest of string
  t.org #1 t.org text.length$ u text.length$ - substring$       % leading digits

%% "DEBUG: t.org = [" t.org * "]" * warning$
  %% "DEBUG: u     = [" u * "]" * warning$

%% dump.stack.2

%% "DEBUG: leave skip.digits " cite$ * warning$
}

FUNCTION { skip.nondigits }
{
  %% skip over leading nondigits in string
  %% stack in:  string
  %% stack out: rest-of-string

%% "DEBUG: enter skip.nondigits " cite$ * warning$

't :=
  "" 'u :=

{ t text.length$ }
  {
    %% "=================DEBUG: skip.nondigits   t = [" t * "]" * warning$
    t is.leading.digit
      {
        t 'u :=
        ""
      }
      { t #2 t text.length$ #1 - substring$ }
    if$
    't :=
  }
  while$

u                     % rest of string

%% dump.stack.1
  %% "DEBUG: leave skip.nondigits " cite$ * warning$
}

FUNCTION { parse.next.number }
{
  %% stack in:  string
  %% stack out: rest-of-string next-numeric-part-of-string
  %% Example:
  %% stack in:  "123:1--123:59"
  %% stack out: ":1--123:59" "123"

's :=
  s skip.nondigits 's :=
  s skip.digits
}

FUNCTION { reduce.pages.to.page.count }
{
  %% Stack in:  arbitrary-and-unused
  %% Stack out: unchanged
  %%
  %% For the new-style pagination with article number and numpages or
  %% pages, we expect to have BibTeX entries containing something like
  %%     articleno = "17",
  %%     pages     = "1--23",
  %% with output "Article 17, 23 pages",
  %% or
  %%     articleno = "17",
  %%     numpages  = "23",
  %% with output "Article 17, 23 pages",
  %% or
  %%     articleno = "17",
  %%     pages     = "17:1--17:23",
  %% with output "Article 17, 23 pages",
  %%
  %% If articleno is missing or empty, then we should output "1--23",
  %% "23" (with a warning of a missing articleno), or "17:1--17:23",
  %% respectively.

%% "DEBUG: enter reduce.pages.to.page.count " cite$ * warning$

%% "DEBUG: pages = [" pages * "]" * warning$

pages
  parse.next.number 'p1 :=
  parse.next.number 'p2 :=
  parse.next.number 'p3 :=
  parse.next.number 'page.count :=

duplicate$
  empty.or.unknown
    {  }
    {
      duplicate$ "unexpected trailing garbage [" swap$ *
      "] after n:p1--n:p2 in pages = [" *
      pages *
      "] in " *
      cite$ *
      warning$
    }
  if$

pop$

%% "DEBUG: reduce.pages.to.page.count: "
  %% " p1 = " p1 * *
  %% " p2 = " p2 * *
  %% " p3 = " p3 * *
  %% " p4 = " page.count * *
  %% " in " cite$ * * warning$

p1 p3 =   p2 "1" =   and   numpages empty.or.unknown   and
    { "INFO: reduced pages = [" pages * "] to numpages = [" * page.count * "]" * warning$ }
    {
      numpages empty.or.unknown
        { pages }
        { numpages }
      if$
      'page.count :=
    }
  if$

p1 "1" =   p3 empty.or.unknown   and   numpages empty.or.unknown   and
    {
      p2 'page.count :=
      "INFO: reduced pages = [" pages * "] to numpages = [" * page.count * "]" * warning$
    }
    {
      numpages empty.or.unknown
        { pages }
        { numpages }
      if$
      'page.count :=
    }
  if$

%% "DEBUG: leave reduce.pages.to.page.count " cite$ * warning$
}

FUNCTION { new.block.checkb }
{ % issue a new.block only if at least one of top two stack strings is not empty
  empty.or.unknown
  swap$ empty.or.unknown
  and
    'skip$
    'new.block
  if$
}

FUNCTION { field.or.null }
{ % convert empty value to null string, else return value
  duplicate$ empty.or.unknown
    { pop$ "" }
    'skip$
  if$
}

FUNCTION { emphasize }
{ % emphasize a non-empty top string on the stack (WITHOUT italic correction)
  duplicate$ empty.or.unknown
    { pop$ "" }
    { "{\em " swap$ * "}" * }
  if$
}

FUNCTION { emphasize.with.italic.correction }
{ % convert empty string to null string, or emphasize with a trailing italic correction
  duplicate$ empty.or.unknown
    { pop$ "" }
    { "{\em " swap$ * "\/}" * }
  if$
}

FUNCTION { comma }
{ % convert empty string to null string, or brace string and add trailing comma
  duplicate$ empty.or.unknown
    { pop$ "" }
    { "{" swap$ * "}," * }
  if$
}

FUNCTION { format.names }
{
  % Format bibliographical entries with the first author last name first,
  % and subsequent authors with initials followed by last name.
  % All names are formatted in this routine.

's :=
  #1 'nameptr :=               % nameptr = 1;
  s num.names$ 'numnames :=    % numnames = num.name$(s);
  numnames 'namesleft :=
    { namesleft #0 > }
    { nameptr #1 =
        %NO: BAD ORDER: {"{" s nameptr "{ff~}{ll}{, jj}{, vv}" format.name$ * "}" * 't := }
        %NO: BAD ORDER: {"{" s nameptr "{ff~}{ll}{, jj}{, vv}" format.name$ * "}" * 't := }
        {"\bibinfo{person}{" s nameptr "{ff }{vv }{ll}{, jj}" format.name$ * "}" * 't := }
        {"\bibinfo{person}{" s nameptr "{ff }{vv }{ll}{, jj}" format.name$ * "}" * 't := }
      if$
      nameptr #1 >
        {
          namesleft #1 >
            { ", " * t * }
            {
              numnames #2 >
                { "," * }
                'skip$
              if$
              t "\bibinfo{person}{others}" =
                { " {et~al\mbox{.}}" * } % jrh: avoid spacing problems
                { " {and} " * t * } % from Chicago Manual of Style
              if$
            }
          if$
        }
        't
      if$
      nameptr #1 + 'nameptr :=          % nameptr += 1;
      namesleft #1 - 'namesleft :=      % namesleft =- 1;
    }
  while$
}

FUNCTION { my.full.label }
{
  's :=
  #1 'nameptr :=               % nameptr = 1;
  s num.names$ 'numnames :=    % numnames = num.name$(s);
  numnames 'namesleft :=
    { namesleft #0 > }

{ s nameptr "{vv~}{ll}" format.name$ 't :=  % get the next name
      nameptr #1 >
        {
          namesleft #1 >
            { ", " * t * }
            {
              numnames #2 >
                { "," * }
                'skip$
              if$
              t "others" =
                { " et~al\mbox{.}" * } % jrh: avoid spacing problems
                { " and " * t * } % from Chicago Manual of Style
              if$
            }
          if$
        }
        't
      if$
      nameptr #1 + 'nameptr :=          % nameptr += 1;
      namesleft #1 - 'namesleft :=      % namesleft =- 1;
    }
  while$

}

FUNCTION { format.names.fml }
{
  % Format names in "familiar" format, with first initial followed by
  % last name. Like format.names, ALL names are formatted.
  % jtb: The names are NOT put in small caps

's :=
  #1 'nameptr :=               % nameptr = 1;
  s num.names$ 'numnames :=    % numnames = num.name$(s);
  numnames 'namesleft :=
    { namesleft #0 > }

{
      "\bibinfo{person}{" s nameptr "{ff~}{vv~}{ll}{, jj}" format.name$ * "}" * 't :=

nameptr #1 >
        {
          namesleft #1 >
            { ", " * t * }
            {
              numnames #2 >
                { "," * }
                'skip$
              if$
              t "\bibinfo{person}{others}" =
                { " {et~al\mbox{.}}" * }
                { " {and} " * t * }
              if$
            }
          if$
        }
        't
      if$
      nameptr #1 + 'nameptr :=          % nameptr += 1;
      namesleft #1 - 'namesleft :=      % namesleft =- 1;
    }
  while$
}

FUNCTION { format.authors }
{
  author empty.or.unknown
    { "" }
    {
      "\bibfield{author}{"
      author format.names add.period$ * "}" *} % jtb: add period if none before
  if$
}

FUNCTION { format.key }
{
  empty.or.unknown
    { key field.or.null }
    { "" }
  if$
}

FUNCTION { format.no.key }
{
  empty.or.unknown
    { "" }
    { "" }
  if$
}

FUNCTION { format.editors.fml }
{
  % Format editor names for use in the "in" types: inbook, incollection,
  % inproceedings: first initial, then last names. When editors are the
  % LABEL for an entry, then format.editor is used which lists editors
  % by last name first.

editor empty.or.unknown
    { "" }
    {
      "\bibfield{editor}{"
      editor format.names.fml
      *  "}" *
      editor num.names$ #1 >
        { " (Eds.)" * }
        { " (Ed.)" * }
      if$
    }
  if$
}

FUNCTION { format.editors }
{ % format editor names for use in labels, last names first.
  editor empty.or.unknown
    { "" }
    {
      "\bibfield{editor}{"
      editor format.names
      *  "}" *
      editor num.names$ #1 >
        { " (Eds.)." * }
        { " (Ed.)." * }
      if$
    }
  if$
}

FUNCTION { format.articletitle }
{
  title empty.or.unknown
    { "" }
    % Use this to preserve lettercase in titles:
    { "\showarticletitle{" title * "}" * }
    % Use this for downcase title style:
    % { \showarticletitle{" title "t" change.case$ * "}" * }
  if$
}

FUNCTION { format.title }
{
  title empty.or.unknown
    { "" }
    % Use this to preserve lettercase in titles:
    { "\bibinfo{title}{" title * "}" * }
    % Use this for downcase title style:
    % { title "t" change.case$ }
  if$
}

FUNCTION { n.dashify }
{
  't :=
  ""
    { t empty.or.unknown not }
    {
      t #1 #1 substring$ "-" =
        {
          t #1 #2 substring$ "--" = not
            { "--" *
              t #2 global.max$ substring$ 't :=
            }
            {
              { t #1 #1 substring$ "-" = }
              {
                "-" *
                t #2 global.max$ substring$ 't :=
              }
              while$
            }
          if$
        }
        {
          t #1 #1 substring$ *
          t #2 global.max$ substring$ 't :=
        }
      if$
    }
  while$
}

FUNCTION { format.btitle }
{
  "\bibinfo{booktitle}{"
  edition empty.or.unknown
  { title emphasize }
  { title empty.or.unknown
    { title emphasize } % jtb: what is this supposed to do ?!?
    { "{\em " title * "\/} (\bibinfo{edition}{" * edition "l" change.case$ *
    "} ed.)" * } % jtb: no parens for ed.
    if$
  }
  if$
  * "}" *
}

FUNCTION { format.emphasize.booktitle }
{ % push "" or "{\em booktitle}" or "{\em booktitle}, (second ed.)" on stack
  "\bibinfo{booktitle}{"
  edition empty.or.unknown
    { booktitle emphasize }
    { booktitle empty.or.unknown
      { "" }
      { "{\em " booktitle * "} (\bibinfo{edition}{" *
      edition "l" change.case$ * "} ed.)" * }
      if$
    }
  if$
  * "}" *
}

FUNCTION { format.city }
{
  % jtb: if the preceding string (the title of the conference) is non-empty,
  % jtb: append the location, otherwise leave empty (so as to trigger the
  % jtb: error message in output.check

duplicate$ empty.or.unknown
    { }
    {
      city empty.or.unknown
        {
          date empty.or.unknown
            { }
            { " (" * date * ")" * }
          if$
        }
        {
          date empty.or.unknown
            { " (" * city * ")" * }
            { " (" * city * ", " * date * ")" * }
          if$
        }
      if$
    }
  if$
}

FUNCTION { tie.or.space.connect }
{
  duplicate$ text.length$ #3 <
    { "~" }
    { " " }
  if$
  swap$ * *
}

FUNCTION { either.or.check }
{
  empty.or.unknown
    'pop$
    { "can't use both " swap$ * " fields in " * cite$ * warning$ }
  if$
}

FUNCTION { format.bvolume }
{
  % jtb: If there is a series, this is added and the volume trails after it.
  % jtb: Otherwise, "Vol" is Capitalized.

volume empty.or.unknown
    { "" }
    {
      series empty.or.unknown
        { "Vol.~\bibinfo{volume}{" volume "}" * *}
        { "\bibinfo{series}{" series "}, " * *
          "Vol.~\bibinfo{volume}{" volume "}" * * *}
      if$
      "volume and number" number either.or.check
    }
  if$
}

FUNCTION { format.bvolume.noseries }
{
  volume empty.or.unknown
    { "" }
    { "Vol.~\bibinfo{volume}{" volume "}" * *
      "volume and number" number either.or.check
    }
  if$
}

FUNCTION { format.series }
{
  series empty.or.unknown
    {""}
    {" {\em (\bibinfo{series}{" * series "})}" *}
  if$
}

FUNCTION { format.number.series }
{
  volume empty.or.unknown
    {
      number empty.or.unknown
        {
          volume empty.or.unknown
          { "" }
          {
            series empty.or.unknown
              { "" }
              { " (\bibinfo{series}{" series * "})" * }
            if$
          }
          if$
        }                                       %    { series field.or.null }
        {
          output.state mid.sentence =
            { "Number" }                        % gnp - changed to mixed case always
            { "Number" }
          if$
          number tie.or.space.connect series empty.or.unknown
            { "there's a number but no series in " cite$ * warning$ }
            { " in \bibinfo{series}{" * series * "}" * }
          if$
        }
      if$
    }
    {
      ""
    }
  if$
}

FUNCTION { multi.page.check }
{
  't :=
  #0 'multiresult :=
    { multiresult not
      t empty.or.unknown not
      and
    }
    { t #1 #1 substring$
      duplicate$ "-" =
      swap$ duplicate$ "," =
      swap$ "+" =
      or or
    { #1 'multiresult := }
    { t #2 global.max$ substring$ 't := }
      if$
    }
  while$
  multiresult
}

FUNCTION { format.pages }
{
  pages empty.or.unknown
    { "" }
    { "\bibinfo{pages}{"
      pages multi.page.check
        { pages n.dashify } % gnp - removed () % jtb: removed pp.
        { pages }
      if$
      * "}" *
    }
  if$
}

FUNCTION { format.pages.check.without.articleno }
{ %% format pages field only if articleno is absent
  %% Stack out: pages-specification
  numpages missing$ pages missing$ and
    { "page numbers missing in both pages and numpages fields in " cite$ * warning$ }
    { }
  if$

articleno empty.or.unknown eid empty.or.unknown and
    {
      pages missing$
        { numpages }
        { format.pages }
      if$
    }
    { "" }
  if$
}

FUNCTION { format.pages.check }
{
  pages empty.or.unknown
    { "page numbers missing in " cite$ * warning$ "" }
    { pages n.dashify }
  if$
}

FUNCTION { format.bookpages }
{
  bookpages empty.or.unknown
    { "" }
    { bookpages "book pages" tie.or.space.connect }
  if$
}

FUNCTION { format.named.pages }
{
  pages empty.or.unknown
    { "" }
    { format.pages "pages" tie.or.space.connect }
  if$
}

%
% Changed by Boris Veytsman, 2011-03-13
% Now the word "pages" is printed even if
% there field pages is not empty.
%

FUNCTION { format.page.count }
{
  page.count empty.or.unknown
    { "" }
    { "\bibinfo{numpages}{" page.count * "}~pages" * }
  if$
}

FUNCTION { format.articleno.numpages }
{
  %% There are seven possible outputs, depending on which fields are set.
  %%
  %% These four are handled here:
  %%
  %%     articleno, numpages, pages     -> "Article articleno-value, numpages-value pages"
  %%     articleno, numpages            -> "Article articleno-value, numpages-value pages"
  %%     articleno, pages               -> "Article articleno-value, reduced-pages-value pages"
  %%     articleno                      -> "Article articleno-value" and warn about missing numpages
  %%
  %% The remaining three have already been handled by
  %% format.pages.check.without.articleno:
  %%
  %%     numpages, pages                -> "pages-value"
  %%     numpages                       -> "numpages-value"
  %%     pages                          -> "pages-value"

articleno empty.or.unknown eid empty.or.unknown and
    {
      numpages empty.or.unknown
        { }
        { "numpages field, but no articleno or eid field, in "
	  cite$ * warning$ }
      if$
      ""
    }
    {
      numpages empty.or.unknown
        {
          pages empty.or.unknown
            {
              "articleno or eid, but no pages or numpages field in "
	         cite$ * warning$
              "" 'page.count :=
            }
            { reduce.pages.to.page.count }
          if$
        }
        { numpages 'page.count := }
      if$

%% The Article number is now handled in format.day.month.year because
      %% ACM prefers the style "Digital Libraries 12, 3, Article 5 (July 2008)"
      %% over "Digital Libraries 12, 3 (July 2008), Article 5"
      %% format.articleno output
      format.page.count
    }
  if$
}

FUNCTION {calc.format.page.count}
{
  numpages empty.or.unknown
   {
     pages empty.or.unknown
        {
        "" 'page.count :=
        }
        { reduce.pages.to.page.count }
     if$
   }
   { numpages 'page.count := }
  if$
  format.page.count
}

FUNCTION { format.journal.volume.number.day.month.year }
{
  % By Young (and Spencer)
  % GNP - fixed bugs with missing volume, number, and/or pages
  %
  % Format journal, volume, number, pages for article types.
  %
  journal empty.or.unknown
    { "no journal in " cite$ * warning$
      "" }
%    { journal emphasize.with.italic.correction }
     {
       "\bibinfo{journal}{"
           journal "Journal of the ACM" =
       { "{\it J. ACM}" }
       {
           journal "American Mathematical Society Translations" =
       { "{\it Amer. Math. Soc. Transl.}" }
       {
           journal "Bulletin of the American Mathematical Society" =
       { "{\it Bull. Amer. Math. Soc.}" }
       {
           journal "Proceedings of the American Mathematical Society" =
       { "{\it Proc. Amer. Math. Soc.}" }
       {
           journal "Transactions of the American Mathematical Society" =
       { "{\it Trans. Amer. Math. Soc.}" }
       {
           journal "Communications of the {ACM}" =
       { "{\it Commun. {ACM}}" }
       {
           journal "{ACM} Computing Surveys" =
       { "{\it Comput. Surveys}" }
       {
           journal "{ACM} Transactions on Mathematical Software" =
       { "{\it {ACM} Trans. Math. Software}" }
       {
           journal "{ACM} {SIGNUM} Newsletter" =
       { "{\it {ACM} {SIGNUM} Newslett.}" }
       {
           journal "American Journal of Sociology" =
       { "{\it Amer. J. Sociology}" }
       {
           journal "Journal of the American Statistical Association" =
       { "{\it J. Amer. Statist. Assoc.}" }
       {
           journal "Applied Mathematics and Computation" =
       { "{\it Appl. Math. Comput.}" }
       {
           journal "American Mathematical Monthly" =
       { "{\it Amer. Math. Monthly}" }
       {
           journal "British Journal of Mathematical and Statistical Psychology" =
       { "{\it Brit. J. Math. Statist. Psych.}" }
       {
           journal "Canadian Mathematical Bulletin" =
       { "{\it Canad. Math. Bull.}" }
       {
           journal "Journal of Computational and Applied Mathematics" =
       { "{\it J. Comput. Appl. Math.}" }
       {
           journal "Journal of Computational Physics" =
       { "{\it J. Comput. Phys.}" }
       {
           journal "Computers and Structures" =
       { "{\it Comput. \& Structures}" }
       {
           journal "The Computer Journal" =
       { "{\it Comput. J.}" }
       {
           journal "Journal of Computer and System Sciences" =
       { "{\it J. Comput. System Sci.}" }
       {
           journal "Contemporary Mathematics" =
       { "{\it Contemp. Math.}" }
       {
           journal "Crelle's Journal" =
       { "{\it Crelle's J.}" }
       {
           journal "Giornale di Mathematiche" =
       { "{\it Giorn. Mat.}" }
       {
           journal "{IEEE} Transactions on Computers" =
       { "{\it {IEEE} Trans. Comput.}" }
       {
           journal "{IEEE} Transactions on Automatic Control" =
       { "{\it {IEEE} Trans. Automat. Control}" }
       {
           journal "Proceedings of the {IEEE}" =
       { "{\it Proc. {IEEE}}" }
       {
           journal "{IEEE} Transactions on Aerospace and Electronic Systems" =
       { "{\it {IEEE} Trans. Aerospace Electron. Systems}" }
       {
           journal "{IMA} Journal of Numerical Analysis" =
       { "{\it {IMA} J. Numer. Anal.}" }
       {
           journal "Information Processing Letters" =
       { "{\it Inform. Process. Lett.}" }
       {
           journal "Journal of the Institute of Mathematics and its Applications" =
       { "{\it J. Inst. Math. Appl.}" }
       {
           journal "International Journal of Control" =
       { "{\it Internat. J. Control}" }
       {
           journal "International Journal for Numerical Methods in Engineering" =
       { "{\it Internat. J. Numer. Methods Engrg.}" }
       {
           journal "International Journal of Supercomputing Applications" =
       { "{\it Internat. J. Supercomputing Applic.}" }
       {
           journal "Journal of Research of the National Bureau of Standards" =
       { "{\it J. Res. Nat. Bur. Standards}" }
       {
           journal "Linear Algebra and its Applications" =
       { "{\it Linear Algebra Appl.}" }
       {
           journal "Journal of Mathematical Analysis and Applications" =
       { "{\it J. Math. Anal. Appl.}" }
       {
           journal "Mathematische Annalen" =
       { "{\it Math. Ann.}" }
       {
           journal "Journal of Mathematical Physics" =
       { "{\it J. Math. Phys.}" }
       {
           journal "Mathematics of Computation" =
       { "{\it Math. Comp.}" }
       {
           journal "Mathematica Scandinavica" =
       { "{\it Math. Scand.}" }
       {
           journal "Mathematical Tables and Other Aids to Computation" =
       { "{\it Math. Tables Aids Comput.}" }
       {
           journal "Numerische Mathematik" =
       { "{\it Numer. Math.}" }
       {
           journal "Pacific Journal of Mathematics" =
       { "{\it Pacific J. Math.}" }
       {
           journal "Journal of Parallel and Distributed Computing" =
       { "{\it J. Parallel and Distrib. Comput.}" }
       {
           journal "Parallel Computing" =
       { "{\it Parallel Comput.}" }
       {
           journal "Philosophical Magazine" =
       { "{\it Philos. Mag.}" }
       {
           journal "Proceedings of the National Academy of Sciences of the USA" =
       { "{\it Proc. Nat. Acad. Sci. U. S. A.}" }
       {
           journal "Quarterly Journal of Mathematics, Oxford, Series (2)" =
       { "{\it Quart. J. Math. Oxford Ser. (2)}" }
       {
           journal "Quarterly of Applied Mathematics" =
       { "{\it Quart. Appl. Math.}" }
       {
           journal "Review of the International Statisical Institute" =
       { "{\it Rev. Inst. Internat. Statist.}" }
       {
           journal "Journal of the Society for Industrial and Applied Mathematics" =
       { "{\it J. Soc. Indust. Appl. Math.}" }
       {
           journal "Journal of the Society for Industrial and Applied Mathematics, Series B, Numerical Analysis" =
       { "{\it J. Soc. Indust. Appl. Math. Ser. B Numer. Anal.}" }
       {
           journal "{SIAM} Journal on Algebraic and Discrete Methods" =
       { "{\it {SIAM} J. Algebraic Discrete Methods}" }
       {
           journal "{SIAM} Journal on Applied Mathematics" =
       { "{\it {SIAM} J. Appl. Math.}" }
       {
           journal "{SIAM} Journal on Computing" =
       { "{\it {SIAM} J. Comput.}" }
       {
           journal "{SIAM} Journal on Matrix Analysis and Applications" =
       { "{\it {SIAM} J. Matrix Anal. Appl.}" }
       {
           journal "{SIAM} Journal on Numerical Analysis" =
       { "{\it {SIAM} J. Numer. Anal.}" }
       {
           journal "{SIAM} Review" =
       { "{\it {SIAM} Rev.}" }
       {
           journal "{SIAM} Journal on Scientific and Statistical Computing" =
       { "{\it {SIAM} J. Sci. Statist. Comput.}" }
       {
           journal "Software Practice and Experience" =
       { "{\it Software Prac. Experience}" }
       {
           journal "Statistical Science" =
       { "{\it Statist. Sci.}" }
       {
           journal "{USSR} Computational Mathematics and Mathematical Physics" =
       { "{\it {U. S. S. R.} Comput. Math. and Math. Phys.}" }
       {
           journal "Journal of {VLSI} and Computer Systems" =
       { "{\it J. {VLSI} Comput. Syst.}" }
       {
           journal "Zeitschrift fur Angewandte Mathematik und Mechanik" =
       { "{\it Z. Angew. Math. Mech.}" }
       {
           journal "Zeitschrift fur Angewandte Mathematik und Physik" =
       { "{\it Z. Angew. Math. Phys.}" }
       {
           journal "ACM Computing Surveys" =
       { "{\it Comput. Surveys}" }
       {
           journal "ACM Transactions on Mathematical Software" =
       { "{\it ACM Trans. Math. Software}" }
       {
           journal "ACM {SIGNUM} Newsletter" =
       { "{\it ACM {SIGNUM} Newslett.}" }
       {
           journal "IEEE Transactions on Computers" =
       { "{\it IEEE Trans. Comput.}" }
       {
           journal "IEEE Transactions on Automatic Control" =
       { "{\it IEEE Trans. Automat. Control}" }
       {
           journal "Proceedings of the IEEE" =
       { "{\it Proc. IEEE}" }
       {
           journal "IEEE Transactions on Aerospace and Electronic Systems" =
       { "{\it IEEE Trans. Aerospace Electron. Systems}" }
       {
           journal "IMA Journal of Numerical Analysis" =
       { "{\it IMA J. Numer. Anal.}" }
       {
           journal "SIAM Journal on Algebraic and Discrete Methods" =
       { "{\it SIAM J. Algebraic Discrete Methods}" }
       {
           journal "SIAM Journal on Applied Mathematics" =
       { "{\it SIAM J. Appl. Math.}" }
       {
           journal "SIAM Journal on Computing" =
       { "{\it SIAM J. Comput.}" }
       {
           journal "SIAM Journal on Matrix Analysis and Applications" =
       { "{\it SIAM J. Matrix Anal. Appl.}" }
       {
           journal "SIAM Journal on Numerical Analysis" =
       { "{\it SIAM J. Numer. Anal.}" }
       {
           journal "SIAM Review" =
       { "{\it SIAM Rev.}" }
       {
           journal "SIAM Journal on Scientific and Statistical Computing" =
       { "{\it SIAM J. Sci. Statist. Comput.}" }
       {
           journal "USSR Computational Mathematics and Mathematical Physics" =
       { "{\it U. S. S. R. Comput. Math. and Math. Phys.}" }
       {
           journal "Journal of VLSI and Computer Systems" =
       { "{\it J. VLSI Comput. Syst.}" }
       {
           journal "Communications of the ACM" =
       { "{\it Commun. ACM}" }
       %% If no match with cases needing special handling, just output journal name
       {  journal emphasize.with.italic.correction  }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
           }
     if$
     * "}" *
         }
    if$

number empty.or.unknown
    {
      volume empty.or.unknown
        { "no number and no volume in " cite$ * warning$ "" * }
        { " " * " \bibinfo{volume}{" * volume * "}" * }
      if$
    }
    {
      volume empty.or.unknown
        {
          "unusual to have number, but no volume, for " cite$ * warning$
          " \bibinfo{number}{" * number * "}" *
        }
        { " \bibinfo{volume}{" * volume  * "}, \bibinfo{number}{" *
          number * "}" *}
      if$
    }
  if$
  after.block 'output.state :=

% Sometimes proceedings are published in journals
  % In this case we do not want to put year, day and month here

type$ "inproceedings" =
    { }
    {format.day.month.year * }
  if$
}

FUNCTION { format.chapter.pages }
{
  chapter empty.or.unknown
    'format.pages
    { type empty.or.unknown
        { "Chapter" } % gnp - changed to mixed case
        { type "t" change.case$ }
      if$
      chapter tie.or.space.connect
      pages empty.or.unknown
        {"page numbers missing in " cite$ * warning$} % gnp - added check
        { ", " * format.pages * }
      if$
    }
  if$
}

FUNCTION { format.in.emphasize.booktitle }
{ % jtb: format for collections or proceedings not appearing in a journal
  booktitle empty.or.unknown
  { "" }
  { "In " format.emphasize.booktitle * }
  if$
}

FUNCTION { format.in.booktitle }
{ % jtb: format for proceedings appearing in a journal
  booktitle empty.or.unknown
  { "" }
  { "In \bibinfo{booktitle}{" booktitle * "}" * }
  if$
}

FUNCTION { format.in.ed.booktitle }
{
  booktitle empty.or.unknown
  { "" }
  { editor empty.or.unknown
    { "In " format.emphasize.booktitle * }
                % jtb: swapped editor location
    { "In " format.emphasize.booktitle * ", " * format.editors.fml * }
    if$
  }
  if$
}

FUNCTION { format.thesis.type }
{ % call with default type on stack top
  type empty.or.unknown
    'skip$    % use default type
    {
      pop$    % discard default type
      % NO: it is silly to have to brace protect every degree type!:  type "t" change.case$
      type
    }
  if$
}

FUNCTION { format.tr.number }
{
  "\bibinfo{type}{"
  type empty.or.unknown
    { "{T}echnical {R}eport" }
    'type
  if$
  "}" * *
  number empty.or.unknown
    { "t" change.case$ }
    %% LOOKS BAD: { "." * number tie.or.space.connect }
    %% Prefer "Research report RJ687." to "Research report. RJ687."
    { number tie.or.space.connect }
  if$
}

FUNCTION { format.advisor }
{
  advisor empty.or.unknown
    { "" }
    { "Advisor(s) " advisor * }
  if$
}

FUNCTION { format.article.crossref }
{ "See"
  "\citeN{" * crossref * "}" *
}

FUNCTION { format.crossref.editor }
{
  editor #1 "{vv~}{ll}" format.name$
  editor num.names$ duplicate$
  #2 >
    { pop$ " et~al\mbox{.}" * }         % jrh: avoid spacing problems
    { #2 <
    'skip$
    { editor #2 "{ff }{vv }{ll}{ jj}" format.name$ "others" =
        { " et~al\mbox{.}" * }          % jrh: avoid spacing problems
        { " and " * editor #2 "{vv~}{ll}" format.name$ * }
      if$
    }
      if$
    }
  if$
}

FUNCTION { format.book.crossref }
{
  volume empty.or.unknown
    { "empty volume in " cite$ * "'s crossref of " * crossref * warning$
      "In "
    }
    { "Volume" volume tie.or.space.connect % gnp - changed to mixed case
      " of " *
    }
  if$
  editor empty.or.unknown
  editor field.or.null author field.or.null =
  or
    { key empty.or.unknown
    { series empty.or.unknown
        { "need editor, key, or series for " cite$ * " to crossref " *
          crossref * warning$
          "" *
        }
        { "{\em " * series * "\/}" * }
      if$
    }
    { key * }
      if$
    }
    { format.crossref.editor * }
  if$
  " \citeN{" * crossref * "}" *
}

FUNCTION { format.incoll.inproc.crossref }
{ "See"
  " \citeN{" * crossref * "}" *
}

FUNCTION { format.lab.names }
{
  % format.lab.names:
  %
  % determines "short" names for the abbreviated author information.
  % "Long" labels are created in calc.label, using the routine my.full.label
  % to format author and editor fields.
  %
  % There are 4 cases for labels.   (n=3 in the example)
  % a) one author             Foo
  % b) one to n               Foo, Bar and Baz
  % c) use of "and others"    Foo, Bar et al.
  % d) more than n            Foo et al.

's :=
  s num.names$ 'numnames :=
  numnames #2 >    % change number to number of others allowed before
                   % forcing "et al".
    { s #1 "{vv~}{ll}" format.name$ " et~al\mbox{.}" * } % jrh: \mbox{} added
    {
      numnames #1 - 'namesleft :=
      #2 'nameptr :=
      s #1 "{vv~}{ll}" format.name$
        { namesleft #0 > }
        { nameptr numnames =
            { s nameptr "{ff }{vv }{ll}{ jj}" format.name$ "others" =
                { " et~al\mbox{.}" * }          % jrh: avoid spacing problems
                { " and " * s nameptr "{vv~}{ll}" format.name$ * }
              if$
            }
            { ", " * s nameptr "{vv~}{ll}" format.name$ * }
          if$
          nameptr #1 + 'nameptr :=
          namesleft #1 - 'namesleft :=
        }
      while$
    }
  if$
}

FUNCTION { author.key.label }
{
  author empty.or.unknown
    { key empty.or.unknown
          { "no key, author in " cite$ * warning$
            cite$ #1 #3 substring$ }
         'key
      if$
    }
    { author format.lab.names }
  if$
}

FUNCTION { editor.key.organization.label }
{ % added - gnp. Provide label formatting by organization if editor is null.
  editor empty.or.unknown
    { organization empty.or.unknown
        { key empty.or.unknown
            { "no key, editor or organization in " cite$ * warning$
              cite$ #1 #3 substring$ }
            'key
          if$
        }
        { organization }
      if$
    }
    { editor format.lab.names }
  if$
}

FUNCTION { author.editor.key.label }
{
  author empty.or.unknown
    { editor empty.or.unknown
          { key empty.or.unknown
               { "no key, author, or editor in " cite$ * warning$
                 cite$ #1 #3 substring$ }
             'key
           if$
         }
          { editor format.lab.names }
      if$
    }
    { author format.lab.names }
  if$
}

FUNCTION { author.editor.key.organization.label }
{ % added - gnp. Provide label formatting by organization if author is null.
  author empty.or.unknown
    { editor empty.or.unknown
        { organization empty.or.unknown
            { key empty.or.unknown
               { "no key, author, editor or organization in " cite$ * warning$
                 cite$ #1 #3 substring$ }
               'key
              if$
            }
            { organization }
          if$
        }
        { editor format.lab.names }
      if$
    }
    { author format.lab.names }
  if$
}

% Calculate label and leave it on stack
FUNCTION { calc.basic.label }
{
  type$ "book" =
  type$ "inbook" =
  or
  type$ "article" =
  or
    'author.editor.key.label
    { type$ "proceedings" =
      type$ "periodical" =
      or
        'editor.key.organization.label
        { type$ "manual" =
            'author.editor.key.organization.label
            'author.key.label
          if$
        }
      if$
    }
  if$
  duplicate$
  year empty.or.unknown 
    { "[n. d.]" }
    { year field.or.null purify$ #-1 #4 substring$}
  if$
  *
  'basic.label.year :=
}

FUNCTION { calc.label }
{
  % Changed - GNP. See also author.editor.organization.sort, editor.organization.sort
  % Form label for BibTeX entry. The classification of which fields are used
  % for which type of entry (book, inbook, etc.) are taken from alpha.bst.
  % The change here from newapa is to also include organization as a
  % citation label if author or editor is missing.

calc.basic.label

author empty.or.unknown  % generate the full label citation information.
    {
      editor empty.or.unknown
        {
          organization empty.or.unknown
            {
              key empty.or.unknown
                {
                  "no author, editor, organization, or key in " cite$ * warning$
                  "??"
                }
                { key }
              if$
            }
            { organization }
          if$
        }
        { editor my.full.label }
      if$
    }
    { author my.full.label }
  if$

% leave label on the stack, to be popped when required.

"}{" * swap$ * "}{" *
  %  year field.or.null purify$ #-1 #4 substring$ *
  %
  % save the year for sort processing afterwards (adding a, b, c, etc.)
  %
  year empty.or.unknown 
    { "[n. d.]" }
    { year field.or.null purify$ #-1 #4 substring$}
  if$
  'label.year :=
}

FUNCTION { output.bibitem }
{
  newline$
  "\bibitem[\protect\citeauthoryear{" write$
  calc.label write$
  sort.year write$
  "}]%" writeln
  "        {" write$
  cite$ write$
  "}" writeln
  ""
  before.all 'output.state :=
}

FUNCTION { output.issue.doi.coden.isxn.lccn.url.eprint }
{ % enter and return with stack empty
  %% We switch now from buffered output to output of complete lines, so
  %% that the Issue .. URL data have their own lines, and are less likely
  %% to be line-wrapped by BibTeX's short-sighted algorithm, which wraps
  %% lines longer than 79 characters, backtracking to what it thinks is
  %% a break point in the string.  Any such wrapping MUST be undone to
  %% prevent percent-newline from appearing in DOIs and URLs.  The
  %% output data are intentionally wrapped in \showxxx{} macros at
  %% beginning of line, and that supply their own punctuation (if they
  %% are not defined to suppress output entirely), to make it easier for
  %% other software to recover them from .bbl files.
  %%
  %% It also makes it possible to later change the macro definitions
  %% to suppress particular output values, or alter their appearance.
  %%
  %% Note that it is possible for theses, technical reports, and
  %% manuals to have ISBNs, and anything that has an ISBN may also
  %% have an ISSN.  When there are no values for these keys, there
  %% is no output generated for them here.

"\newblock" writeln
  after.block 'output.state :=

output.issue
  output.isbn
  output.coden  % CODEN is functionally like ISSN, so output them sequentially
  output.issn
  output.lccn
  output.doi    % DOI is ALWAYS last according to CrossRef DOI documentation
  output.eprint
  output.url    % but ACM wants URL last
}

FUNCTION { output.issue.doi.coden.isxn.lccn.url.eprint.note }
{ % enter with stack empty, return with empty string on stack
  output.issue.doi.coden.isxn.lccn.url.eprint
  note empty.or.unknown
    { }
    {
      "\newblock" writeln
      output.note
    }
  if$
  ""
}

FUNCTION { output.issue.doi.coden.isxn.lccn.url.eprint.note.check }
{ % enter with stack empty, return with empty string on stack
  output.issue.doi.coden.isxn.lccn.url.eprint
  note empty.or.unknown
    { }
    {
      "\newblock" writeln
      output.note.check
    }
  if$
  ""
}

FUNCTION { article }
{
  output.bibitem

author empty.or.unknown
    {
      editor empty.or.unknown
        { "neither author and editor supplied for " cite$ * warning$ }
        { format.editors "editor" output.check }
      if$
    }
    { format.authors "author" output.check }
  if$

author format.no.key output       % added
  output.year.check                 % added
  new.block
  format.articletitle "title" output.check
  new.block
  howpublished empty.or.unknown
    { }
    { "\bibinfo{howpublished}{" howpublished "}" * * output }
  if$

crossref missing$
    { format.journal.volume.number.day.month.year output}
    {
      "cross reference in @Article{...} is unusual" warning$
      format.article.crossref output.nonnull
    }
  if$

format.pages.check.without.articleno output
  format.articleno.numpages output
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { book }
{
  output.bibitem
  author empty.or.unknown
    { format.editors "author and editor" output.check }
    { format.authors output.nonnull
      crossref missing$
        { "author and editor" editor either.or.check }
        'skip$
      if$
    }
  if$
  output.year.check       % added
  new.block
  format.btitle "title" output.check
  crossref missing$
    { new.sentence              % jtb: start a new sentence for series/volume
      format.bvolume output
      new.block
      format.number.series output
      new.sentence
      publisher "publisher" bibinfo.output.check
      address "address" bibinfo.output.check    % jtb: require address
      fin.sentence
      pages empty.or.unknown
        { format.bookpages }    % use bookpages when pages empty
        { format.pages.check "pages" tie.or.space.connect }
      if$
      output
    }
    { new.block
      format.book.crossref output.nonnull
    }
  if$
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { booklet }
{
  output.bibitem
  format.authors output
  author format.key output          % added
  output.year.check                 % added
  new.block
  format.title "title" output.check
  new.block
    howpublished empty.or.unknown
    { }
    { "\bibinfo{howpublished}{" howpublished "}" * * output }
  if$
  address output
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { inbook }
{
  output.bibitem
  author empty.or.unknown
    { format.editors
      "author and editor" output.check
    }
    { format.authors output.nonnull
      crossref missing$
    { "author and editor" editor either.or.check }
    'skip$
      if$
    }
  if$
  output.year.check                 % added
  new.block
  format.btitle "title" output.check
  crossref missing$
    { new.sentence              % jtb: start a new sentence for series/volume
      format.bvolume output
      new.block
      format.number.series output
      new.sentence
      publisher "publisher" bibinfo.output.check
      address "address" bibinfo.output.check    % jtb: require address
      format.bookpages output
      format.chapter.pages
      "chapter and pages" output.check  % jtb: moved from before publisher
    }
    {
      format.bookpages output
      format.chapter.pages "chapter and pages" output.check
      new.block
      format.book.crossref output.nonnull
    }
  if$
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { incollection }
{
  output.bibitem
  format.authors "author" output.check
  author format.key output       % added
  output.year.check              % added
  new.block
  format.articletitle "title" output.check
  new.block
  crossref missing$
    { format.in.ed.booktitle "booktitle" output.check
      new.sentence                % jtb: start a new sentence for series/volume
      format.bvolume output
      format.number.series output
      new.sentence
      publisher "publisher" bibinfo.output.check
      address "address" bibinfo.output.check      % jtb: require address
      format.bookpages output
      format.chapter.pages output % gnp - was special.output.nonnull
                                  % left out comma before page numbers
                                  % jtb: moved from before publisher
    }
    {
      format.incoll.inproc.crossref output.nonnull
      format.chapter.pages output
    }
  if$
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { inproceedings }
{
  output.bibitem
  format.authors "author" output.check
  author format.key output            % added
  output.year.check                   % added
  new.block
  format.articletitle "title" output.check
  howpublished empty.or.unknown
    { }
    { "\bibinfo{howpublished}{" howpublished "}" * * output.dot.space }
  if$
  crossref missing$
    {
      journal missing$          % jtb: proceedings appearing in journals
        { format.in.emphasize.booktitle format.city "booktitle"  output.check.dot.space
          format.series output.removenospace
          format.editors.fml output % BV 2011/09/27 Moved dot to comma
          format.bvolume.noseries output
          new.sentence
          organization output
          publisher "publisher" bibinfo.output.check % jtb: require publisher (?)
          address "address" bibinfo.output.check  % jtb: require address
          format.bookpages output
        }
        {
           format.in.booktitle format.city "booktitle" output.check
           format.editors.fml output
           new.sentence
           format.journal.volume.number.day.month.year output
        }
      if$
      format.articleno output
      format.pages.check.without.articleno output
    }
    {
      format.incoll.inproc.crossref output.nonnull
      format.articleno output
      format.pages.check.without.articleno output
    }
  if$
  format.articleno.numpages output
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { conference } { inproceedings }

FUNCTION { manual }
{
  output.bibitem
  author empty.or.unknown
    { editor empty.or.unknown
      { organization "organization" output.check
        organization format.key output }  % if all else fails, use key
      { format.editors "author and editor" output.check }
      if$
    }
    { format.authors output.nonnull }
    if$
  output.year.check                 % added
  new.block
  format.btitle "title" output.check
  organization address new.block.checkb
  % jtb: back to normal style: organization, address
  organization "organization" output.check
  address output
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { mastersthesis }
{
  output.bibitem
  format.authors "author" output.check
  author format.key output          % added
  output.year.check                 % added
  new.block
  format.title emphasize "title" output.check  % NB: ACM style requires emphasized thesis title
  new.block
  "\bibinfo{thesistype}{Master's\ thesis}" format.thesis.type output
  new.sentence
  school "school" bibinfo.output.check
  address empty.or.unknown
     { }
     { "\bibinfo{address}{" address * "}" * output }
  if$
  new.block
  format.advisor output
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { misc }
{
  output.bibitem
  format.authors "author" output.check
  author format.key output            % added
  output.year.check                   % added
  title howpublished new.block.checkb
  format.title output
  new.block
  howpublished empty.or.unknown
    { }
    { "\bibinfo{howpublished}{" howpublished "}" * * output }
  if$
  "" output.nonnull.dot.space
  output.day.month.year
  calc.format.page.count output
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { online } { misc }

FUNCTION { game } { misc }

FUNCTION { phdthesis }
{
  output.bibitem
  format.authors "author" output.check
  author format.key output          % added
  output.year.check                 % added
  new.block
  format.title emphasize "title" output.check  % NB: ACM style requires emphasized thesis title
  new.block
  "\bibinfo{thesistype}{Ph.D. Dissertation}" format.thesis.type output
  new.sentence
  school "school" bibinfo.output.check
  address empty.or.unknown
     { }
     { "\bibinfo{address}{" address * "}" * output }
  if$
  new.block
  format.advisor output
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION {format.date}
{ year empty.or.unknown
    { month empty.or.unknown
        {
          ""                    % output empty date if year/month both empty
          day empty.or.unknown
            {  }
            { "there's a day but no month or year in " cite$ * warning$ }
          if$
        }
        { "there's a month but no year in " cite$ * warning$
          month
          day empty.or.unknown
            { }
            { " " * day * }
          if$
        }
      if$
    }
    { month empty.or.unknown
        {
          year                  % output only year if month empty
          day empty.or.unknown
            {  }
            { "there's a day and year but no month in " cite$ * warning$ }
          if$
        }
        {
          month " " *
          day empty.or.unknown
            { }
            { day * ", " * }
          if$
          year *
        }
      if$
    }
  if$
}

FUNCTION {new.block.checka}
{
  empty.or.unknown
    'skip$
    'new.block
  if$
}

FUNCTION { periodical }
{
  output.bibitem
  editor empty.or.unknown
    { organization output }
    { format.editors output.nonnull }
  if$
  new.block
  output.year.check
  new.sentence
  format.articletitle "title" output.check
  format.journal.volume.number.day.month.year output
  calc.format.page.count output
  fin.entry
}

FUNCTION { proceedings }
{
  output.bibitem
  editor empty.or.unknown
    { organization output
      organization format.key output }  % gnp - changed from author format.key
    { format.editors output.nonnull }
  if$
  % author format.key output             % gnp - removed (should be either
  %                                        editor or organization
  output.year.check                    % added (newapa)
  new.block
  format.btitle format.city "title" output.check        % jtb: added city
  new.sentence
  format.bvolume output
  format.number.series output
  new.sentence
  organization output
  % jtb: normal order: publisher, address
  publisher empty.or.unknown
     { }
     { "\bibinfo{publisher}{" publisher * "}" * output }
  if$
  address empty.or.unknown
     { }
     { "\bibinfo{address}{" address * "}" * output }
  if$
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { techreport }
{
  output.bibitem
  format.authors "author" output.check
  author format.key output             % added
  output.year.check                    % added
  new.block
  format.btitle "title" output.check
  new.block
%   format.tr.number output               % jtb: moved month ...
  format.tr.number output new.sentence    % Gerry  - need dot 2011/09/28
  institution "institution" bibinfo.output.check
  address empty.or.unknown
    { }
    { "\bibinfo{address}{" address "}" * * output }
  if$
  new.sentence
  format.named.pages output
  % ACM omits year at end in transactions style
  % format.day.month.year output.nonnull.dot.space  % jtb: ... to here (no parens)
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note
  fin.entry
}

FUNCTION { unpublished }
{
  output.bibitem
  format.authors
  "author" output.check
  author format.key output              % added
  output.year.check                     % added
  new.block
  format.title "title" output.check
  fin.sentence
  output.day.month.year                 % UTAH
  calc.format.page.count output
  fin.block
  output.issue.doi.coden.isxn.lccn.url.eprint.note.check
  fin.entry
}

FUNCTION { default.type } { misc }

%%% ACM journal-style month definitions: full name if 1--5 letters, else
%%% abbreviation of 3 or 4 characters and a dot

MACRO {jan}             {"Jan."}

MACRO {feb}             {"Feb."}

MACRO {mar}             {"March"}

MACRO {apr}             {"April"}

MACRO {may}             {"May"}

MACRO {jun}             {"June"}

MACRO {jul}             {"July"}

MACRO {aug}             {"Aug."}

MACRO {sep}             {"Sept."}

MACRO {oct}             {"Oct."}

MACRO {nov}             {"Nov."}

MACRO {dec}             {"Dec."}

READ

FUNCTION { sortify }
{
  purify$
  "l" change.case$
}

FUNCTION { chop.word }
{
  's :=
  'len :=
  s #1 len substring$ =
    { s len #1 + global.max$ substring$ }
    's
  if$
}

FUNCTION { sort.format.names }
{
  's :=
  #1 'nameptr :=
  ""
  s num.names$ 'numnames :=
  numnames 'namesleft :=
    { namesleft #0 > }
    { nameptr #1 >
          { "   " * }
         'skip$
      if$
      s nameptr "{vv{ } }{ll{ }}{  f{ }}{  jj{ }}" format.name$ 't :=
      nameptr numnames = t "others" = and
          { " et~al" * }
          { t sortify * }
      if$
      nameptr #1 + 'nameptr :=
      namesleft #1 - 'namesleft :=
    }
  while$
}

FUNCTION { sort.format.title }
{
  't :=
  "A " #2
    "An " #3
      "The " #4 t chop.word
    chop.word
  chop.word
  sortify
  #1 global.max$ substring$
}

FUNCTION { author.sort }
{
  author empty.or.unknown
    { key empty.or.unknown
         { "to sort, need author or key in " cite$ * warning$
           "" }
         { key sortify }
      if$
    }
    { author sort.format.names }
  if$
}

FUNCTION { author.editor.sort }
{
  author empty.or.unknown
    {
      editor empty.or.unknown
         {
           key empty.or.unknown
             { "to sort, need author, editor, or key in " cite$ * warning$
               ""
             }
             { key sortify }
           if$
         }
         { editor sort.format.names }
      if$
    }
    { author sort.format.names }
  if$
}

FUNCTION { editor.organization.sort }
{
  % added - GNP. Stack editor or organization for sorting (from alpha.bst).
  % Unlike alpha.bst, we need entire names, not abbreviations

editor empty.or.unknown
    { organization empty.or.unknown
        { key empty.or.unknown
            { "to sort, need editor, organization, or key in " cite$ * warning$
              ""
            }
            { key sortify }
          if$
        }
        { organization sortify }
      if$
    }
    { editor sort.format.names }
  if$
}

FUNCTION { author.editor.organization.sort }
{
  % added - GNP. Stack author or organization for sorting (from alpha.bst).
  % Unlike alpha.bst, we need entire names, not abbreviations

author empty.or.unknown
    {
      editor empty.or.unknown
        { organization empty.or.unknown
            { key empty.or.unknown
                { "to sort, need author, editor, or key in " cite$ * warning$
                ""
                }
                { key sortify }
              if$
            }
            { organization sortify }
          if$
        }
        { editor sort.format.names }
      if$
    }
    { author sort.format.names }
  if$
}

FUNCTION { presort }
{
  % Presort creates the bibentry's label via a call to calc.label, and then
  % sorts the entries based on entry type. Chicago.bst adds support for
  % including organizations as the sort key; the following is stolen from
  % alpha.bst.

calc.label 
  basic.label.year 
  swap$
  "    " 
  swap$
  * *
  "    " 
  *
  sortify
  year field.or.null purify$ #-1 #4 substring$ * % add year
  "    "
  *
  type$ "book" =
  type$ "inbook" =
  or
  type$ "article" =
  or
    'author.editor.sort
    { type$ "proceedings" =
      type$ "periodical" =
      or
        'editor.organization.sort
        { type$ "manual" =
            'author.editor.organization.sort
            'author.sort
          if$
        }
      if$
    }
  if$
  #1 entry.max$ substring$        % added for newapa
  'sort.label :=                  % added for newapa
  sort.label                      % added for newapa
  *
  "    "
  *
  title field.or.null
  sort.format.title
  *
  #1 entry.max$ substring$
  'sort.key$ :=
}

ITERATE { presort }

SORT             % by label, year, author/editor, title

FUNCTION { initialize.extra.label.stuff }
{ #0 int.to.chr$ 'last.label :=
  "" 'next.extra :=
  #0 'last.extra.num :=
}

FUNCTION { forward.pass }
{
  % Pass through all entries, comparing current entry to last one.
  % Need to concatenate year to the stack (done by calc.label) to determine
  % if two entries are the same (see presort)

last.label
  % OLD:calc.label year field.or.null purify$ #-1 #4 substring$ * % add year
  % NEW:
  calc.basic.label year field.or.null purify$ #-1 #4 substring$ * % add year
  #1 entry.max$ substring$ =     % are they equal?
     { last.extra.num #1 + 'last.extra.num :=
       last.extra.num int.to.chr$ 'extra.label :=
     }
     { "a" chr.to.int$ 'last.extra.num :=
       "" 'extra.label :=
       % OLD: calc.label year field.or.null purify$ #-1 #4 substring$ * % add year
       % NEW:
       calc.basic.label year field.or.null purify$ #-1 #4 substring$ * % add year
       #1 entry.max$ substring$ 'last.label := % assign to last.label
     }
  if$
}

FUNCTION { reverse.pass }
{
  next.extra "b" =
    { "a" 'extra.label := }
     'skip$
  if$
  label.year extra.label * 'sort.year :=
  extra.label 'next.extra :=
}

EXECUTE {initialize.extra.label.stuff}

ITERATE {forward.pass}

REVERSE {reverse.pass}

FUNCTION { bib.sort.order }
{
  sort.label
  "    "
  *
  year field.or.null sortify
  *
  "    "
  *
  title field.or.null
  sort.format.title
  *
  #1 entry.max$ substring$
  'sort.key$ :=
}

ITERATE { bib.sort.order }

SORT             % by sort.label, year, title --- giving final bib. order.

FUNCTION { begin.bib }
{
  %% Set to #0 show 13-digit ISBN in preference to 10-digit ISBN.
  %% Set to #1 to show both 10-digit and 13-digit ISBNs.
  #1 'show-isbn-10-and-13 :=

"%%% -*-BibTeX-*-" writeln
  "%%% Do NOT edit. File created by BibTeX with style" writeln
  "%%% ACM-Reference-Format-Journals [18-Jan-2012]." writeln
  "" writeln

preamble$ empty.or.unknown
    'skip$
    { preamble$ writeln }
  if$
  "\begin{thebibliography}{00}" writeln
  ""                                                                         writeln
  "%%% ====================================================================" writeln
  "%%% NOTE TO THE USER: you can override these defaults by providing"       writeln
  "%%% customized versions of any of these macros before the \bibliography"  writeln
  "%%% command.  Each of them MUST provide its own final punctuation,"       writeln
  "%%% except for \shownote{}, \showDOI{}, and \showURL{}.  The latter two"  writeln
  "%%% do not use final punctuation, in order to avoid confusing it with"    writeln
  "%%% the Web address."                                                     writeln
  "%%%"                                                                      writeln
  "%%% To suppress output of a particular field, define its macro to expand" writeln
  "%%% to an empty string, or better, \unskip, like this:"                   writeln
  "%%%"                                                                      writeln
  "%%% \newcommand{\showDOI}[1]{\unskip}   % LaTeX syntax"                   writeln
  "%%%"                                                                      writeln
  "%%% \def \showDOI #1{\unskip}           % plain TeX syntax"               writeln
  "%%%"                                                                      writeln
  "%%% ====================================================================" writeln
  ""                                                                         writeln

%% ACM publications do not use CODEN, ISSN, and LCCN data, so their default
  %% macro wrappers expand to \unskip, discarding their values and unwanted
  %% space.
  %%
  %% For other publications, prior definitions like these may be useful:
  %%
  %%     Plain TeX:
  %%         \def \showCODEN     #1{CODEN #1.}
  %%         \def \showISSN      #1{ISSN #1.}
  %%         \def \showLCCN      #1{LCCN #1.}
  %%
  %%     LaTeX:
  %%         \newcommand{\showCODEN}[1]{CODEN #1.}
  %%         \newcommand{\showISSN}[1]#1{ISSN #1.}
  %%         \newcommand{\showLCCN}[1]{LCCN #1.}

"\ifx \showCODEN    \undefined \def \showCODEN     #1{\unskip}     \fi" writeln
  "\ifx \showDOI      \undefined \def \showDOI       #1{#1}\fi" writeln
  % ACM styles omit ISBNs, but they can be included by suitable definitions of
  % \showISBNx and \showISBNxiii before the .bbl file is read
  "\ifx \showISBNx    \undefined \def \showISBNx     #1{\unskip}     \fi" writeln
  "\ifx \showISBNxiii \undefined \def \showISBNxiii  #1{\unskip}     \fi" writeln
  "\ifx \showISSN     \undefined \def \showISSN      #1{\unskip}     \fi" writeln
  "\ifx \showLCCN     \undefined \def \showLCCN      #1{\unskip}     \fi" writeln
  "\ifx \shownote     \undefined \def \shownote      #1{#1}          \fi" writeln % NB: final period supplied by add.period$ above
  "\ifx \showarticletitle \undefined \def \showarticletitle #1{#1}   \fi" writeln
  "\ifx \showURL      \undefined \def \showURL       {\relax}        \fi" writeln
  "% The following commands are used for tagged output and should be " writeln
  "% invisible to TeX" writeln
  "\providecommand\bibfield[2]{#2}" writeln
  "\providecommand\bibinfo[2]{#2}" writeln
  "\providecommand\natexlab[1]{#1}" writeln
  "\providecommand\showeprint[2][]{arXiv:#2}" writeln
}

EXECUTE {begin.bib}

EXECUTE {init.state.consts}

ITERATE {call.type$}

FUNCTION { end.bib }
{
  newline$
  "\end{thebibliography}"
  writeln
}

EXECUTE {end.bib}

Ignore Space Show notes View Koli_2017/Database_assignment_marks.xlsx 0 → 100644
Not supported

Ignore Space Show notes View Koli_2017/Koli_2017_Stanger.bib 0 → 100644
@string{jeric = {Journal on Educational Resources in Computing}} @string{sej = {Software Engineering Journal}} @string{pvldb = {Proceedings of the VLDB Endowment}} @string{vldb = {The VLDB Journal}} @online{Ambler.S-2006a-Database, Author = {Scott W. Ambler}, Howpublished = {Essay, \emph{Agile Data} web site}, Lastaccessed = {2017-07-20}, Title = {Database testing: {H}ow to regression test a relational database}, Url = {http://www.agiledata.org/essays/databaseTesting.html}, Year = {2006}} @manual{Apache-2017a-CQL, Address = {Forest Hill, Maryland, USA}, Author = {{Apache Software Foundation}}, Booktitle = {The Cassandra Query Language (CQL) v3.4.0}, Lastaccessed = {2017-07-20}, Month = jun, Title = {The Cassandra Query Language (CQL) v3.4.0}, Url = {http://cassandra.apache.org/doc/old/CQL-3.0.html}, Year = {2017}} @manual{Apache-2017a-Hive, Address = {Forest Hill, Maryland, USA}, Author = {{Apache Software Foundation}}, Booktitle = {Hive Language Manual}, Month = jun, Title = {Hive Language Manual}, Url = {https://cwiki.apache.org/confluence/display/Hive/LanguageManual}, Lastaccessed = {2017-07-20}, Year = {2017}} @article{Bench-Capon.T-1998a-Report, Author = {T. Bench-Capon and D. Castelli and F. Coenen and L. Devendeville-Brisoux and B. Eaglestone and N. Fiddian and A. Gray and A. Ligeza and A. Vermesan}, Journal = {Information Research}, Month = oct, Number = {2}, Articleno = {7}, Title = {Report on the 1st {I}nternational {W}orkshop on {V}alidation, {V}erification and {I}ntegrity {I}ssues of {E}xpert and {D}atabase {S}ystems}, Url = {http://www.informationr.net/ir/4-3/paper55.html}, Volume = {4}, Year = {1998}} @online{Bergmann.S-2017a-PHPUnit, Author = {Sebastian Bergmann}, Title = {{PHPUnit} Documentation, Chapter 8: Database Testing}, Url = {https://phpunit.de/manual/current/en/database.html}, Lastaccessed = {2017-07-31}, Year = {2017}} @inproceedings{Bhangdiya.A-2015a-XDa-TA, Author = {Amol Bhangdiya and Bikash Chandra and Biplab Kar and Bharath Radhakrishnan and K. V. Maheshwara Reddy and Shetal Shah and S. Sudarshan}, Crossref = {Gehrke.J-2015a-ICDE}, Pages = {1468--1471}, Doi = {10.1109/ICDE.2015.7113403}, Title = {The {XDa-TA} system for automated grading of {SQL} query assignments}} @inproceedings{Binnig.C-2008a-Multi-RQP, Articleno = {5}, Numpages = {6}, Author = {Carsten Binnig and Donald Kossmann and Eric Lo}, Crossref = {Giakoumakis.L-2008a-DBTest}, Doi = {10.1145/1385269.1385276}, Isbn = {978-1-60558-233-7}, Title = {Multi-{RQP}: {G}enerating test databases for the functional testing of {OLTP} applications}} @article{Brusilovsky.P-2010a-Learning, Articleno = {19}, Author = {Peter Brusilovsky and Sergey Sosnovsky and Michael V. Yudelson and Danielle H. Lee and Vladimir Zadorozhny and Xin Zhou}, Doi = {10.1145.1656255.1656257}, Journal = {ACM Transactions on Computing Education}, Month = jan, Number = {4}, Title = {Learning {SQL} programming with interactive tools: {F}rom integration to personalization}, Volume = {9}, Numpages = {15}, Year = {2010}} @inproceedings{Calero.C-2001a-Database, Author = {Coral Calero and Mario Piattini and Marcela Genero}, Crossref = {Oliveira.J-2001a-QuaTIC}, Pages = {79--85}, Title = {Database complexity metrics}, Url = {http://ceur-ws.org/Vol-1284/paper9.pdf}} @book{Cattell.R-2000a-ODMG3, Address = {San Francisco, California, USA}, Author = {R. G. G. Cattell and Douglas K. Barry and Mark Berler and Jeff Eastman and David Jordan and Craig Russell and Olaf Schadow and Torsten Stanienda and Fernando Velez}, Booktitle = {The Object Database Standard: ODMG 3.0}, Isbn = {978-1-55860-647-5}, Publisher = {Morgan Kaufmann}, Title = {The Object Database Standard: ODMG 3.0}, Year = {2000}} @article{Chandra.B-2015a-Data, Author = {Bikash Chandra and Bhupesh Chawda and Biplab Kar and K. V. Maheshwara Reddy and Shetal Shah and S. Sudarshan}, Journal = vldb, Month = dec, Number = {6}, Pages = {731--755}, Title = {Data generation for testing and grading {SQL} queries}, Volume = {24}, Doi = {10.1007/s00778-015-0395-0}, Year = {2015}} @article{Chandra.B-2016a-Partial, Author = {Bikash Chandra and Mathew Joseph and Bharath Radhakrishnan and Shreevidhya Acharya and S. Sudarshan}, Journal = pvldb, Month = sep, Number = {13}, Pages = {1541--1544}, Title = {Partial marking for automated grading of {SQL} queries}, Volume = {9}, Doi = {10.14778/3007263.3007304}, Year = {2016}} @inproceedings{Chays.D-2008a-Query-based, Articleno = {6}, Numpages = {6}, Author = {David Chays and John Shahid and Phyllis G. Frankl}, Crossref = {Giakoumakis.L-2008a-DBTest}, Doi = {10.1145/1385269.1385277}, Isbn = {978-1-60558-233-7}, Keywords = {database, software testing}, Title = {Query-based test generation for database applications}} @techreport{Choppella.V-2006a-Constructing, Address = {Bloomington, Indiana, USA}, Author = {Venkatesh Choppella and Arijit Sengupta and Edward L. Robertson and Steven D, Johnson}, Institution = {Indiana University, School of Informatics and Computing}, Month = apr, Number = {TR632}, Title = {Constructing and Validating entity-relationship data models in the {PVS} specification language: {A} case study using a text-book example}, Type = {Technical report}, Url = {https://www.cs.indiana.edu/cgi-bin/techreports/TRNNN.cgi?trnum=TR632}, Year = {2006}} @book{Date.C-2009a-SQL-and-Relational, Author = {Chris J. Date}, Address = {Sebastopol, California, USA}, Booktitle = {SQL and Relational Theory: How to Write Accurate SQL Code}, Edition = {first}, Isbn = {9780596523060}, Publisher = {O'Reilly}, Series = {Theory in Practice}, Title = {SQL and Relational Theory: How to Write Accurate SQL Code}, Year = {2009}} @inproceedings{Dekeyser.S-2007a-Computer, Author = {Stijn Dekeyser and Michael de Raadt and Tien Yu Lee}, Crossref = {Bailey.J-2007a-ADC}, Pages = {53--62}, Url = {http://dl.acm.org/citation.cfm?id=1273730.1273737}, Title = {Computer assisted assessment of {SQL} query skills}} @article{Dietrich.S-1993a-An-educational, Author = {Suzanne W. Dietrich}, Doi = {10.1080/0899340930040201}, Journal = {Computer Science Education}, Number = {2}, Pages = {157--184}, Title = {An educational tool for formal relational database query languages}, Volume = {4}, Year = {1993}} @inproceedings{Dietrich.S-1997a-WinRDBI, Author = {Suzanne W. Dietrich and Eric Eckert and Kevin Piscator}, Crossref = {Lewis.J-1998a-SIGCSE}, Doi = {10.1145/268085.268131}, Pages = {126--130}, Title = {{WinRDBI}: {A} {W}indows-based relational database educational tool}} @article{Douce.C-2005a-Automatic, Articleno = {4}, Author = {Christopher Douce and David Livingstone and James Orwell}, Doi = {10.1145/1163405.1163409}, Journal = jeric, Keywords = {education, computer-based training, learning, programming assessment}, Month = sep, Number = {3}, Title = {Automatic test-based assessment of programming: {A} review}, Volume = {5}, Numpages = {13}, Year = {2005}} @inproceedings{Farre.C-2008a-SVTe, Articleno = {9}, Numpages = {6}, Author = {Carles Farr{\'e} and Guillem Rull and Ernest Teniente and Toni Urp{\'\i}}, Crossref = {Giakoumakis.L-2008a-DBTest}, Doi = {10.1145/1385269.1385281}, Isbn = {978-1-60558-233-7}, Title = {{SVTe}: {A} tool to validate database schemas giving explanations}} @online{Gong.A-2015a-CS-121-Automation, Author = {Angela Gong}, Lastaccessed = {2017-07-20}, Month = oct, Title = {CS 121 Automation Tool}, Url = {https://github.com/anjoola/cs12x-automate}, Year = {2015}} @inproceedings{Haller.K-2010a-Test, Articleno = {6}, Numpages = {6}, Author = {Klaus Haller}, Crossref = {Babu.S-2010a-DBTest}, Doi = {10.1145/1838126.1838132}, Keywords = {databases, information systems, test coverage, test data, testing}, Title = {The test data challenge for database-driven applications}} @inproceedings{Jamil.B-2010a-SMARtS, Articleno = {27}, Author = {Bushra Jamil and Asma Batool}, Crossref = {ICIET-2010a-Proceedings}, Doi = {10.1109/ICIET.2010.5625716}, Numpages = {6}, Title = {{SMARtS}: {S}oftware metric analyzer for relational database systems}} @inproceedings{Kearns.R-1997a-A-teaching, Author = {R. Kearns and Stephen Shead and Alan Fekete}, Crossref = {Sondergaard.H-1997a-ACSE}, Doi = {10.1145/299359.299391}, Pages = {224--231}, Title = {A teaching system for {SQL}}} @inproceedings{Kenny.C-2005a-Automated, Author = {Claire Kenny and Claus Pahl}, Crossref = {Dann.W-2005a-SIGCSE}, Pages = {58--62}, Doi = {10.1145/1047124.1047377}, Title = {Automated tutoring for a database skills training environment}} @inproceedings{Kleiner.C-2013a-Automated, Author = {Carsten Kleiner and Christopher Tebbe and Felix Heine}, Crossref = {Laakso.M-2013a-KoliCalling}, Pages = {161--168}, Doi = {10.1145/2526968.2526986}, Title = {Automated grading and tutoring of {SQL} statements to improve student learning}} @inproceedings{Lukovic.I-2003a-Proceedings, Author = {Ivan Lukovi{\'c} and Sonja Risti{\'c} and Pavle Mogin}, Crossref = {SISY-2003a-Proceedings}, Pages = {125--136}, Title = {On the formal specification of database schema constraints}, Url = {http://www.mcs.vuw.ac.nz/~pmogin/SISY_2003.pdf}} @inproceedings{Marcozzi.M-2012a-Test, Articleno = {6}, Numpages = {6}, Author = {Micha{\"e}l Marcozzi and Wim Vanhoof and Jean-Luc Hainaut}, Crossref = {Lo.E-2012a-DBTest}, Doi = {10.1145/2304510.2304518}, Isbn = {978-1-4503-1429-9}, Title = {Test input generation for database programs using relational constraints}} @inproceedings{Mitrovic.A-1998a-Learning, Author = {Antonija Mitrovic}, Crossref = {Lewis.J-1998a-SIGCSE}, Doi = {10.1145/274790.274318}, Pages = {307--311}, Title = {Learning {SQL} with a computerized tutor}} @article{Ohlsson.S-1992a-Constraint-based, Author = {Stellan Ohlsson}, Journal = {Journal of Artificial Intelligence in Education}, Number = {4}, Pages = {429--447}, Title = {Constraint-based student modelling}, Volume = {3}, Year = {1992}} @article{Ohlsson.S-2016a-Constraint-based, Author = {Stellan Ohlsson}, Doi = {10.1007/s40593-015-0075-7}, Journal = {International Journal of Artificial Intelligence in Education}, Number = {1}, Pages = {457--473}, Title = {Constraint-based modeling: {F}rom cognitive theory to computer tutoring -- and back again}, Volume = {26}, Year = {2016}} @inproceedings{Pavlic.M-2008a-Database, Author = {Mile Pavli{\'c} and Marin Kalu{\v z}a and Neven Vr{\v c}ek}, Crossref = {Aurer.B-2008a-CECIIS}, Pages = {577--583}, Title = {Database complexity measuring method}, Url = {http://archive.ceciis.foi.hr/app/index.php/ceciis/2008/paper/view/84}} @article{Piattini.M-2001a-Table, Author = {Mario Piattini and Coral Calero and Marcela Genero}, Doi = {10.1023/A:1016670717863}, Journal = {Software Quality Journal}, Month = jun, Number = {2}, Pages = {79--97}, Title = {Table oriented metrics for relational databases}, Volume = {9}, Year = {2001}} @inproceedings{Prior.J-2004a-Backwash, Author = {Julia Coleman Prior and Raymond Lister}, Crossref = {Boyle.R-2004a-ITiCSE}, Pages = {32--36}, Doi = {10.1145/1026487.1008008}, Title = {The backwash effect on {SQL} skills grading}} @inproceedings{Russell.G-2004a-Improving, Author = {Gordon Russell and Andrew Cumming}, Crossref = {Kinshuk-2004a-CELDA}, Pages = {281--288}, Url = {http://www.napier.ac.uk/research-and-innovation/research-search/outputs/improving-the-student-learning-experience-for-sql-using-automatic-marking}, Title = {Improving the student learning experience for {SQL} using automatic marking}} @inproceedings{Russell.G-2005a-Online, Author = {Gordon Russell and Andrew Cumming}, Crossref = {OReilly.U-2005a-TLAD}, Pages = {46--50}, Url = {http://www.napier.ac.uk/research-and-innovation/research-search/outputs/online-assessment-and-checking-of-sql-detecting-and-preventing-plagiarism}, Title = {Online assessment and checking of {SQL}: {D}etecting and preventing plagiarism}} @inproceedings{Sadiq.S-2004a-SQLator, Author = {Shazia Wasim Sadiq and Maria E. Orlowska and Wasim Sadiq and Joe Y.-C. Lin}, Crossref = {Boyle.R-2004a-ITiCSE}, Pages = {223--227}, Doi = {10.1145/1026487.1008055}, Title = {{SQLator}: {A}n online {SQL} learning workbench}} @article{Sinha.B-2014a-Estimation, Author = {Bhaskar R. Sinha and Gordon W. Romney and Pradip P. Dey and Mohammad N. Amin}, Issn = {1937-4771}, Journal = {Journal of Computing Sciences in Colleges}, Month = dec, Number = {2}, Pages = {95--104}, Title = {Estimation of database complexity from modeling schemas}, Url = {http://dl.acm.org/citation.cfm?id=2667432.2667445}, Volume = {30}, Year = {2014}} @article{Spivey.J-1989a-An-introduction, Author = {J.M. Spivey}, Doi = {10.1049/sej.1989.0006}, Journal = sej, Month = jan, Number = {1}, Pages = {40--50}, Title = {An introduction to {Z} and formal specifications}, Volume = {4}, Year = {1989}} @article{Vatanawood.W-2004a-Formal, Author = {Wiwat Vatanawood and Wanchai Rivepiboon}, Doi = {10.1002/int.10159}, Journal = {International Journal of Intelligent Systems}, Month = jan # {--} # feb, Number = {1--2}, Pages = {159--175}, Title = {Formal specification synthesis for relational database model}, Volume = {19}, Year = {2004}} # Crossrefs @proceedings{Aurer.B-2008a-CECIIS, Address = {Vara{\v z}din, Croatia}, Booktitle = {Proceedings of the 19th Central European Conference on Information and Intelligent Systems (CECIIS 2008)}, Editor = {Boris Aurer and Miroslav Ba{\v c}a and Kornelije Rabuzin}, Isbn = {978-953-6071-04-3}, Month = {24--26~} # sep, Publisher = {University of Zagreb}, Title = {Proceedings of the 19th Central European Conference on Information and Intelligent Systems (CECIIS 2008)}, Year = {2008}} @proceedings{Babu.S-2010a-DBTest, Address = {Indianapolis, Indiana, USA}, Booktitle = {Proceedings of the 3rd International Workshop on Testing Database Systems (DBTest 2010)}, Editor = {Shivnath Babu and G. N. Paulley}, Isbn = {978-1-4503-0190-9}, Month = jun # {~7}, Publisher = {ACM}, Title = {Proceedings of the 3rd International Workshop on Testing Database Systems (DBTest 2010)}, Year = {2010}} @proceedings{Bailey.J-2007a-ADC, Address = {Ballarat, Victoria, Australia}, Editor = {James Bailey and Alan Fekete}, Month = jan # {~29--} # feb # {~2}, Publisher = {Australian Computer Society}, Booktitle = {Database Technologies 2007, Proceedings of the 18th Australasian Database Conference (ADC 2007)}, Title = {Database Technologies 2007, Proceedings of the 18th Australasian Database Conference (ADC 2007)}, Volume = {63}, Series = {Conferences in Research and Practice in Information Technology}, Year = {2007}} @proceedings{Boyle.R-2004a-ITiCSE, Address = {Leeds, UK}, Editor = {Roger D. Boyle and Martyn Clark and Amruth N. Kumar}, Month = jun # {~28--30}, Publisher = {ACM}, Booktitle = {Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE 2004)}, Title = {Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE 2004)}, Year = {2004}} @proceedings{Dann.W-2005a-SIGCSE, Address = {St. Louis, Missouri, USA}, Editor = {Wanda Dann and Thomas L. Naps and Paul T. Tymann and Doug Baldwin}, Month = feb # {~23--27}, Publisher = {ACM}, Booktitle = {Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education (SIGCSE 2005)}, Title = {Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education (SIGCSE 2005)}, Year = {2005}} @proceedings{Gehrke.J-2015a-ICDE, Address = {Seoul, South Korea}, Editor = {Johannes Gehrke and Wolfgang Lehner and Kyuseok Shim and Sang Kyun Cha and Guy M. Lohman}, Month = apr # {~13--17}, Publisher = {IEEE Computer Society}, Booktitle = {Proceedings of the 31st IEEE International Conference on Data Engineering (ICDE 2015)}, Title = {Proceedings of the 31st IEEE International Conference on Data Engineering (ICDE 2015)}, Year = {2015}} @proceedings{Giakoumakis.L-2008a-DBTest, Address = {Vancouver, British Columbia, Canada}, Booktitle = {Proceedings of the 1st International Workshop on Testing Database Systems (DBTest 2008)}, Editor = {Leo Giakoumakis and Donald Kossmann}, Isbn = {978-1-60558-233-7}, Month = jun # {~13}, Publisher = {ACM}, Title = {Proceedings of the 1st International Workshop on Testing Database Systems (DBTest 2008)}, Year = {2008}} @proceedings{Kinshuk-2004a-CELDA, Address = {Lisbon, Portugal}, Editor = {{Kinshuk} and Demetrios G. Sampson and Pedro T. Isa{\'\i}as}, Month = {15--17~} # dec, Publisher = {IADIS}, Booktitle = {Proceedings of the IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA'04)}, Title = {Proceedings of the IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA'04)}, Year = {2004}} @proceedings{OReilly.U-2005a-TLAD, Address = {Sunderland, UK}, Editor = {Una O'Reilly and Richard Cooper}, Month = jul, Publisher = {LTSN-ICS}, Booktitle = {Proceedings of the 3rd HEA-ICS Workshop on Teaching Learning and Assessment in Databases (TLAD 2005)}, Title = {Proceedings of the 3rd HEA-ICS Workshop on Teaching Learning and Assessment in Databases (TLAD 2005)}, Year = {2005}} @proceedings{ICIET-2010a-Proceedings, Address = {Karachi, Pakistan}, Booktitle = {Proceedings of the 2010 International Conference on Information and Emerging Technologies}, Month = {14--16~} # jun, Publisher = {IEEE Computer Society}, Title = {Proceedings of the 2010 International Conference on Information and Emerging Technologies}, Year = {2010}} @proceedings{Laakso.M-2013a-KoliCalling, Address = {Koli, Finland}, Editor = {Mikko-Jussi Laakso and {Simon}}, Month = nov # {~14--17}, Publisher = {ACM}, Booktitle = {Proceedings of the 13th Koli Calling International Conference on Computing Education Research (Koli Calling '13)}, Title = {Proceedings of the 13th Koli Calling International Conference on Computing Education Research (Koli Calling '13)}, Year = {2013}} @proceedings{Lewis.J-1998a-SIGCSE, Address = {Atlanta, Georgia, USA}, Booktitle = {Proceedings of the 29th SIGCSE Technical Symposium on Computer Science Education (SIGCSE'98)}, Editor = {John Lewis and Jane Prey and Daniel Joyce and John Impagliazzo}, Isbn = {0-89791-994-7}, Month = feb # {~26--} # mar # {~1}, Publisher = {ACM}, Title = {Proceedings of the 29th SIGCSE Technical Symposium on Computer Science Education (SIGCSE'98)}, Year = {1998}} @proceedings{Lo.E-2012a-DBTest, Address = {Scottsdale, Arizona, USA}, Booktitle = {Proceedings of the 5th International Workshop on Testing Database Systems (DBTest 2012)}, Editor = {Eric Lo and Florian Waas}, Isbn = {978-1-4503-1429-9}, Month = may # {~21}, Publisher = {ACM}, Title = {Proceedings of the 5th International Workshop on Testing Database Systems (DBTest 2012)}, Year = {2012}} @proceedings{Oliveira.J-2001a-QuaTIC, Address = {Lisbon, Portugal}, Booktitle = {Proceedings of the 4th International Conference on the Quality of Information and Communications Technology (QuaTIC 2001)}, Editor = {Jo{\~a}o Nuno Oliveira and Fernando {Brito e Abreu} and Mario Piattini and Carlos Barreiras}, Month = mar # {~12--14}, Series = {CEUR Workshop Proceedings}, Title = {Proceedings of the 4th International Conference on the Quality of Information and Communications Technology (QuaTIC 2001)}, Volume = {1284}, Year = {2001}} @proceedings{SISY-2003a-Proceedings, Address = {Subotica, Serbia}, Booktitle = {Proceedings of the 1st Serbian-Hungarian Joint Symposium on Intelligent Systems (SISY 2003)}, Month = {19--20~} # sep, Publisher = {IEEE Computer Society}, Title = {Proceedings of the 1st Serbian-Hungarian Joint Symposium on Intelligent Systems (SISY 2003)}, Year = {2003}} @proceedings{Sondergaard.H-1997a-ACSE, Address = {Melbourne, Victoria, Australia}, Booktitle = {Proceedings of the ACM SIGCSE 2nd Australasian Conference on Computer Science Education (ACSE 1997)}, Editor = {Harald S{\o}ndergaard and A. John Hurst}, Isbn = {0-89791-958-0}, Publisher = {ACM}, Title = {Proceedings of the ACM SIGCSE 2nd Australasian Conference on Computer Science Education (ACSE 1997)}, Year = {1997}}

Ignore Space Show notes View

Koli_2017/Koli_2017_Stanger.bib 0 → 100644

@string{jeric = {Journal on Educational Resources in Computing}}
@string{sej = {Software Engineering Journal}}
@string{pvldb = {Proceedings of the VLDB Endowment}}
@string{vldb = {The VLDB Journal}}

@online{Ambler.S-2006a-Database,
	Author = {Scott W. Ambler},
	Howpublished = {Essay, \emph{Agile Data} web site},
	Lastaccessed = {2017-07-20},
	Title = {Database testing: {H}ow to regression test a relational database},
	Url = {http://www.agiledata.org/essays/databaseTesting.html},
	Year = {2006}}

@manual{Apache-2017a-CQL,
	Address = {Forest Hill, Maryland, USA},
	Author = {{Apache Software Foundation}},
	Booktitle = {The Cassandra Query Language (CQL) v3.4.0},
	Lastaccessed = {2017-07-20},
	Month = jun,
	Title = {The Cassandra Query Language (CQL) v3.4.0},
	Url = {http://cassandra.apache.org/doc/old/CQL-3.0.html},
	Year = {2017}}

@manual{Apache-2017a-Hive,
	Address = {Forest Hill, Maryland, USA},
	Author = {{Apache Software Foundation}},
	Booktitle = {Hive Language Manual},
	Month = jun,
	Title = {Hive Language Manual},
	Url = {https://cwiki.apache.org/confluence/display/Hive/LanguageManual},
	Lastaccessed = {2017-07-20},
	Year = {2017}}

@article{Bench-Capon.T-1998a-Report,
	Author = {T. Bench-Capon and D. Castelli and F. Coenen and L. Devendeville-Brisoux and B. Eaglestone and N. Fiddian and A. Gray and A. Ligeza and A. Vermesan},
	Journal = {Information Research},
	Month = oct,
	Number = {2},
	Articleno = {7},
	Title = {Report on the 1st {I}nternational {W}orkshop on {V}alidation, {V}erification and {I}ntegrity {I}ssues of {E}xpert and {D}atabase {S}ystems},
	Url = {http://www.informationr.net/ir/4-3/paper55.html},
	Volume = {4},
	Year = {1998}}

@online{Bergmann.S-2017a-PHPUnit,
	Author = {Sebastian Bergmann},
	Title = {{PHPUnit} Documentation, Chapter 8: Database Testing},
	Url = {https://phpunit.de/manual/current/en/database.html},
	Lastaccessed = {2017-07-31},
	Year = {2017}}

@inproceedings{Bhangdiya.A-2015a-XDa-TA,
	Author = {Amol Bhangdiya and Bikash Chandra and Biplab Kar and Bharath Radhakrishnan and K. V. Maheshwara Reddy and Shetal Shah and S. Sudarshan},
	Crossref = {Gehrke.J-2015a-ICDE},
	Pages = {1468--1471},
	Doi = {10.1109/ICDE.2015.7113403},
	Title = {The {XDa-TA} system for automated grading of {SQL} query assignments}}

@inproceedings{Binnig.C-2008a-Multi-RQP,
	Articleno = {5},
	Numpages = {6},
	Author = {Carsten Binnig and Donald Kossmann and Eric Lo},
	Crossref = {Giakoumakis.L-2008a-DBTest},
	Doi = {10.1145/1385269.1385276},
	Isbn = {978-1-60558-233-7},
	Title = {Multi-{RQP}: {G}enerating test databases for the functional testing of {OLTP} applications}}

@article{Brusilovsky.P-2010a-Learning,
	Articleno = {19},
	Author = {Peter  Brusilovsky and Sergey Sosnovsky and Michael V. Yudelson and Danielle H. Lee and Vladimir Zadorozhny and Xin Zhou},
	Doi = {10.1145.1656255.1656257},
	Journal = {ACM Transactions on Computing Education},
	Month = jan,
	Number = {4},
	Title = {Learning {SQL} programming with interactive tools: {F}rom integration to personalization},
	Volume = {9},
	Numpages = {15},
	Year = {2010}}

@inproceedings{Calero.C-2001a-Database,
	Author = {Coral Calero and Mario Piattini and Marcela Genero},
	Crossref = {Oliveira.J-2001a-QuaTIC},
	Pages = {79--85},
	Title = {Database complexity metrics},
	Url = {http://ceur-ws.org/Vol-1284/paper9.pdf}}

@book{Cattell.R-2000a-ODMG3,
	Address = {San Francisco, California, USA},
	Author = {R. G. G. Cattell and Douglas K. Barry and Mark Berler and Jeff Eastman and David Jordan and Craig Russell and Olaf Schadow and Torsten Stanienda and Fernando Velez},
	Booktitle = {The Object Database Standard: ODMG 3.0},
	Isbn = {978-1-55860-647-5},
	Publisher = {Morgan Kaufmann},
	Title = {The Object Database Standard: ODMG 3.0},
	Year = {2000}}

@article{Chandra.B-2015a-Data,
	Author = {Bikash Chandra and Bhupesh Chawda and Biplab Kar and K. V. Maheshwara Reddy and Shetal Shah and S. Sudarshan},
	Journal = vldb,
	Month = dec,
	Number = {6},
	Pages = {731--755},
	Title = {Data generation for testing and grading {SQL} queries},
	Volume = {24},
	Doi = {10.1007/s00778-015-0395-0},
	Year = {2015}}

@article{Chandra.B-2016a-Partial,
	Author = {Bikash Chandra and Mathew Joseph and Bharath Radhakrishnan and Shreevidhya Acharya and S. Sudarshan},
	Journal = pvldb,
	Month = sep,
	Number = {13},
	Pages = {1541--1544},
	Title = {Partial marking for automated grading of {SQL} queries},
	Volume = {9},
	Doi = {10.14778/3007263.3007304},
	Year = {2016}}

@inproceedings{Chays.D-2008a-Query-based,
	Articleno = {6},
	Numpages = {6},
	Author = {David Chays and John Shahid and Phyllis G. Frankl},
	Crossref = {Giakoumakis.L-2008a-DBTest},
	Doi = {10.1145/1385269.1385277},
	Isbn = {978-1-60558-233-7},
	Keywords = {database, software testing},
	Title = {Query-based test generation for database applications}}

@techreport{Choppella.V-2006a-Constructing,
	Address = {Bloomington, Indiana, USA},
	Author = {Venkatesh Choppella and Arijit Sengupta and Edward L. Robertson and Steven D, Johnson},
	Institution = {Indiana University, School of Informatics and Computing},
	Month = apr,
	Number = {TR632},
	Title = {Constructing and Validating entity-relationship data models in the {PVS} specification language: {A} case study using a text-book example},
	Type = {Technical report},
	Url = {https://www.cs.indiana.edu/cgi-bin/techreports/TRNNN.cgi?trnum=TR632},
	Year = {2006}}

@book{Date.C-2009a-SQL-and-Relational,
	Author = {Chris J. Date},
	Address = {Sebastopol, California, USA},
	Booktitle = {SQL and Relational Theory: How to Write Accurate SQL Code},
	Edition = {first},
	Isbn = {9780596523060},
	Publisher = {O'Reilly},
	Series = {Theory in Practice},
	Title = {SQL and Relational Theory: How to Write Accurate SQL Code},
	Year = {2009}}

@inproceedings{Dekeyser.S-2007a-Computer,
	Author = {Stijn Dekeyser and Michael de Raadt and Tien Yu Lee},
	Crossref = {Bailey.J-2007a-ADC},
	Pages = {53--62},
	Url = {http://dl.acm.org/citation.cfm?id=1273730.1273737},
	Title = {Computer assisted assessment of {SQL} query skills}}

@article{Dietrich.S-1993a-An-educational,
	Author = {Suzanne W. Dietrich},
	Doi = {10.1080/0899340930040201},
	Journal = {Computer Science Education},
	Number = {2},
	Pages = {157--184},
	Title = {An educational tool for formal relational database query languages},
	Volume = {4},
	Year = {1993}}

@inproceedings{Dietrich.S-1997a-WinRDBI,
	Author = {Suzanne W. Dietrich and Eric Eckert and Kevin Piscator},
	Crossref = {Lewis.J-1998a-SIGCSE},
	Doi = {10.1145/268085.268131},
	Pages = {126--130},
	Title = {{WinRDBI}: {A} {W}indows-based relational database educational tool}}

@article{Douce.C-2005a-Automatic,
	Articleno = {4},
	Author = {Christopher Douce and David Livingstone and James Orwell},
	Doi = {10.1145/1163405.1163409},
	Journal = jeric,
	Keywords = {education, computer-based training, learning, programming assessment},
	Month = sep,
	Number = {3},
	Title = {Automatic test-based assessment of programming: {A} review},
	Volume = {5},
	Numpages = {13},
	Year = {2005}}

@inproceedings{Farre.C-2008a-SVTe,
	Articleno = {9},
	Numpages = {6},
	Author = {Carles Farr{\'e} and Guillem Rull and Ernest Teniente and Toni Urp{\'\i}},
	Crossref = {Giakoumakis.L-2008a-DBTest},
	Doi = {10.1145/1385269.1385281},
	Isbn = {978-1-60558-233-7},
	Title = {{SVTe}: {A} tool to validate database schemas giving explanations}}

@online{Gong.A-2015a-CS-121-Automation,
	Author = {Angela Gong},
	Lastaccessed = {2017-07-20},
	Month = oct,
	Title = {CS 121 Automation Tool},
	Url = {https://github.com/anjoola/cs12x-automate},
	Year = {2015}}

@inproceedings{Haller.K-2010a-Test,
	Articleno = {6},
	Numpages = {6},
	Author = {Klaus Haller},
	Crossref = {Babu.S-2010a-DBTest},
	Doi = {10.1145/1838126.1838132},
	Keywords = {databases, information systems, test coverage, test data, testing},
	Title = {The test data challenge for database-driven applications}}

@inproceedings{Jamil.B-2010a-SMARtS,
	Articleno = {27},
	Author = {Bushra Jamil and Asma Batool},
	Crossref = {ICIET-2010a-Proceedings},
	Doi = {10.1109/ICIET.2010.5625716},
	Numpages = {6},
	Title = {{SMARtS}: {S}oftware metric analyzer for relational database systems}}

@inproceedings{Kearns.R-1997a-A-teaching,
	Author = {R. Kearns and Stephen Shead and Alan Fekete},
	Crossref = {Sondergaard.H-1997a-ACSE},
	Doi = {10.1145/299359.299391},
	Pages = {224--231},
	Title = {A teaching system for {SQL}}}

@inproceedings{Kenny.C-2005a-Automated,
	Author = {Claire Kenny and Claus Pahl},
	Crossref = {Dann.W-2005a-SIGCSE},
	Pages = {58--62},
	Doi = {10.1145/1047124.1047377},
	Title = {Automated tutoring for a database skills training environment}}

@inproceedings{Kleiner.C-2013a-Automated,
	Author = {Carsten Kleiner and Christopher Tebbe and Felix Heine},
	Crossref = {Laakso.M-2013a-KoliCalling},
	Pages = {161--168},
	Doi = {10.1145/2526968.2526986},
	Title = {Automated grading and tutoring of {SQL} statements to improve student learning}}

@inproceedings{Lukovic.I-2003a-Proceedings,
	Author = {Ivan Lukovi{\'c} and Sonja Risti{\'c} and Pavle Mogin},
	Crossref = {SISY-2003a-Proceedings},
	Pages = {125--136},
	Title = {On the formal specification of database schema constraints},
	Url = {http://www.mcs.vuw.ac.nz/~pmogin/SISY_2003.pdf}}

@inproceedings{Marcozzi.M-2012a-Test,
	Articleno = {6},
	Numpages = {6},
	Author = {Micha{\"e}l Marcozzi and Wim Vanhoof and Jean-Luc Hainaut},
	Crossref = {Lo.E-2012a-DBTest},
	Doi = {10.1145/2304510.2304518},
	Isbn = {978-1-4503-1429-9},
	Title = {Test input generation for database programs using relational constraints}}

@inproceedings{Mitrovic.A-1998a-Learning,
	Author = {Antonija Mitrovic},
	Crossref = {Lewis.J-1998a-SIGCSE},
	Doi = {10.1145/274790.274318},
	Pages = {307--311},
	Title = {Learning {SQL} with a computerized tutor}}

@article{Ohlsson.S-1992a-Constraint-based,
	Author = {Stellan Ohlsson},
	Journal = {Journal of Artificial Intelligence in Education},
	Number = {4},
	Pages = {429--447},
	Title = {Constraint-based student modelling},
	Volume = {3},
	Year = {1992}}

@article{Ohlsson.S-2016a-Constraint-based,
	Author = {Stellan Ohlsson},
	Doi = {10.1007/s40593-015-0075-7},
	Journal = {International Journal of Artificial Intelligence in Education},
	Number = {1},
	Pages = {457--473},
	Title = {Constraint-based modeling: {F}rom cognitive theory to computer tutoring -- and back again},
	Volume = {26},
	Year = {2016}}

@inproceedings{Pavlic.M-2008a-Database,
	Author = {Mile Pavli{\'c} and Marin Kalu{\v z}a and Neven Vr{\v c}ek},
	Crossref = {Aurer.B-2008a-CECIIS},
	Pages = {577--583},
	Title = {Database complexity measuring method},
	Url = {http://archive.ceciis.foi.hr/app/index.php/ceciis/2008/paper/view/84}}

@article{Piattini.M-2001a-Table,
	Author = {Mario Piattini and Coral Calero and Marcela Genero},
	Doi = {10.1023/A:1016670717863},
	Journal = {Software Quality Journal},
	Month = jun,
	Number = {2},
	Pages = {79--97},
	Title = {Table oriented metrics for relational databases},
	Volume = {9},
	Year = {2001}}

@inproceedings{Prior.J-2004a-Backwash,
	Author = {Julia Coleman Prior and Raymond Lister},
	Crossref = {Boyle.R-2004a-ITiCSE},
	Pages = {32--36},
	Doi = {10.1145/1026487.1008008},
	Title = {The backwash effect on {SQL} skills grading}}

@inproceedings{Russell.G-2004a-Improving,
	Author = {Gordon Russell and Andrew Cumming},
	Crossref = {Kinshuk-2004a-CELDA},
	Pages = {281--288},
	Url = {http://www.napier.ac.uk/research-and-innovation/research-search/outputs/improving-the-student-learning-experience-for-sql-using-automatic-marking},
	Title = {Improving the student learning experience for {SQL} using automatic marking}}

@inproceedings{Russell.G-2005a-Online,
	Author = {Gordon Russell and Andrew Cumming},
	Crossref = {OReilly.U-2005a-TLAD},
	Pages = {46--50},
	Url = {http://www.napier.ac.uk/research-and-innovation/research-search/outputs/online-assessment-and-checking-of-sql-detecting-and-preventing-plagiarism},
	Title = {Online assessment and checking of {SQL}: {D}etecting and preventing plagiarism}}

@inproceedings{Sadiq.S-2004a-SQLator,
	Author = {Shazia Wasim Sadiq and Maria E. Orlowska and Wasim Sadiq and Joe Y.-C. Lin},
	Crossref = {Boyle.R-2004a-ITiCSE},
	Pages = {223--227},
	Doi = {10.1145/1026487.1008055},
	Title = {{SQLator}: {A}n online {SQL} learning workbench}}

@article{Sinha.B-2014a-Estimation,
	Author = {Bhaskar R. Sinha and Gordon W. Romney and Pradip P. Dey and Mohammad N. Amin},
	Issn = {1937-4771},
	Journal = {Journal of Computing Sciences in Colleges},
	Month = dec,
	Number = {2},
	Pages = {95--104},
	Title = {Estimation of database complexity from modeling schemas},
	Url = {http://dl.acm.org/citation.cfm?id=2667432.2667445},
	Volume = {30},
	Year = {2014}}

@article{Spivey.J-1989a-An-introduction,
	Author = {J.M. Spivey},
	Doi = {10.1049/sej.1989.0006},
	Journal = sej,
	Month = jan,
	Number = {1},
	Pages = {40--50},
	Title = {An introduction to {Z} and formal specifications},
	Volume = {4},
	Year = {1989}}

@article{Vatanawood.W-2004a-Formal,
	Author = {Wiwat Vatanawood and Wanchai Rivepiboon},
	Doi = {10.1002/int.10159},
	Journal = {International Journal of Intelligent Systems},
	Month = jan # {--} # feb,
	Number = {1--2},
	Pages = {159--175},
	Title = {Formal specification synthesis for relational database model},
	Volume = {19},
	Year = {2004}}

# Crossrefs

@proceedings{Aurer.B-2008a-CECIIS,
	Address = {Vara{\v z}din, Croatia},
	Booktitle = {Proceedings of the 19th Central European Conference on Information and Intelligent Systems (CECIIS 2008)},
	Editor = {Boris Aurer and Miroslav Ba{\v c}a and Kornelije Rabuzin},
	Isbn = {978-953-6071-04-3},
	Month = {24--26~} # sep,
	Publisher = {University of Zagreb},
	Title = {Proceedings of the 19th Central European Conference on Information and Intelligent Systems (CECIIS 2008)},
	Year = {2008}}

@proceedings{Babu.S-2010a-DBTest,
	Address = {Indianapolis, Indiana, USA},
	Booktitle = {Proceedings of the 3rd International Workshop on Testing Database Systems (DBTest 2010)},
	Editor = {Shivnath Babu and G. N. Paulley},
	Isbn = {978-1-4503-0190-9},
	Month = jun # {~7},
	Publisher = {ACM},
	Title = {Proceedings of the 3rd International Workshop on Testing Database Systems (DBTest 2010)},
	Year = {2010}}

@proceedings{Bailey.J-2007a-ADC,
	Address = {Ballarat, Victoria, Australia},
	Editor = {James Bailey and Alan Fekete},
	Month = jan # {~29--} # feb # {~2},
	Publisher = {Australian Computer Society},
	Booktitle = {Database Technologies 2007, Proceedings of the 18th Australasian Database Conference (ADC 2007)},
	Title = {Database Technologies 2007, Proceedings of the 18th Australasian Database Conference (ADC 2007)},
	Volume = {63},
	Series = {Conferences in Research and Practice in Information Technology},
	Year = {2007}}

@proceedings{Boyle.R-2004a-ITiCSE,
	Address = {Leeds, UK},
	Editor = {Roger D. Boyle and Martyn Clark and Amruth N. Kumar},
	Month = jun # {~28--30},
	Publisher = {ACM},
	Booktitle = {Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE 2004)},
	Title = {Proceedings of the 9th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education (ITiCSE 2004)},
	Year = {2004}}

@proceedings{Dann.W-2005a-SIGCSE,
	Address = {St. Louis, Missouri, USA},
	Editor = {Wanda Dann and Thomas L. Naps and Paul T. Tymann and Doug Baldwin},
	Month = feb # {~23--27},
	Publisher = {ACM},
	Booktitle = {Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education (SIGCSE 2005)},
	Title = {Proceedings of the 36th SIGCSE Technical Symposium on Computer Science Education (SIGCSE 2005)},
	Year = {2005}}

@proceedings{Gehrke.J-2015a-ICDE,
	Address = {Seoul, South Korea},
	Editor = {Johannes Gehrke and Wolfgang Lehner and Kyuseok Shim and Sang Kyun Cha and Guy M. Lohman},
	Month = apr # {~13--17},
	Publisher = {IEEE Computer Society},
	Booktitle = {Proceedings of the 31st IEEE International Conference on Data Engineering (ICDE 2015)},
	Title = {Proceedings of the 31st IEEE International Conference on Data Engineering (ICDE 2015)},
	Year = {2015}}

@proceedings{Giakoumakis.L-2008a-DBTest,
	Address = {Vancouver, British Columbia, Canada},
	Booktitle = {Proceedings of the 1st International Workshop on Testing Database Systems (DBTest 2008)},
	Editor = {Leo Giakoumakis and Donald Kossmann},
	Isbn = {978-1-60558-233-7},
	Month = jun # {~13},
	Publisher = {ACM},
	Title = {Proceedings of the 1st International Workshop on Testing Database Systems (DBTest 2008)},
	Year = {2008}}

@proceedings{Kinshuk-2004a-CELDA,
	Address = {Lisbon, Portugal},
	Editor = {{Kinshuk} and Demetrios G. Sampson and Pedro T. Isa{\'\i}as},
	Month = {15--17~} # dec,
	Publisher = {IADIS},
	Booktitle = {Proceedings of the IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA'04)},
	Title = {Proceedings of the IADIS International Conference on Cognition and Exploratory Learning in Digital Age (CELDA'04)},
	Year = {2004}}

@proceedings{OReilly.U-2005a-TLAD,
	Address = {Sunderland, UK},
	Editor = {Una O'Reilly and Richard Cooper},
	Month = jul,
	Publisher = {LTSN-ICS},
	Booktitle = {Proceedings of the 3rd HEA-ICS Workshop on Teaching Learning and Assessment in Databases (TLAD 2005)},
	Title = {Proceedings of the 3rd HEA-ICS Workshop on Teaching Learning and Assessment in Databases (TLAD 2005)},
	Year = {2005}}

@proceedings{ICIET-2010a-Proceedings,
	Address = {Karachi, Pakistan},
	Booktitle = {Proceedings of the 2010 International Conference on Information and Emerging Technologies},
	Month = {14--16~} # jun,
	Publisher = {IEEE Computer Society},
	Title = {Proceedings of the 2010 International Conference on Information and Emerging Technologies},
	Year = {2010}}

@proceedings{Laakso.M-2013a-KoliCalling,
	Address = {Koli, Finland},
	Editor = {Mikko-Jussi Laakso and {Simon}},
	Month = nov # {~14--17},
	Publisher = {ACM},
	Booktitle = {Proceedings of the 13th Koli Calling International Conference on Computing Education Research (Koli Calling '13)},
	Title = {Proceedings of the 13th Koli Calling International Conference on Computing Education Research (Koli Calling '13)},
	Year = {2013}}

@proceedings{Lewis.J-1998a-SIGCSE,
	Address = {Atlanta, Georgia, USA},
	Booktitle = {Proceedings of the 29th SIGCSE Technical Symposium on Computer Science Education (SIGCSE'98)},
	Editor = {John Lewis and Jane Prey and Daniel Joyce and John Impagliazzo},
	Isbn = {0-89791-994-7},
	Month = feb # {~26--} # mar # {~1},
	Publisher = {ACM},
	Title = {Proceedings of the 29th SIGCSE Technical Symposium on Computer Science Education (SIGCSE'98)},
	Year = {1998}}

@proceedings{Lo.E-2012a-DBTest,
	Address = {Scottsdale, Arizona, USA},
	Booktitle = {Proceedings of the 5th International Workshop on Testing Database Systems (DBTest 2012)},
	Editor = {Eric Lo and Florian Waas},
	Isbn = {978-1-4503-1429-9},
	Month = may # {~21},
	Publisher = {ACM},
	Title = {Proceedings of the 5th International Workshop on Testing Database Systems (DBTest 2012)},
	Year = {2012}}

@proceedings{Oliveira.J-2001a-QuaTIC,
	Address = {Lisbon, Portugal},
	Booktitle = {Proceedings of the 4th International Conference on the Quality of Information and Communications Technology (QuaTIC 2001)},
	Editor = {Jo{\~a}o Nuno Oliveira and Fernando {Brito e Abreu} and Mario Piattini and Carlos Barreiras},
	Month = mar # {~12--14},
	Series = {CEUR Workshop Proceedings},
	Title = {Proceedings of the 4th International Conference on the Quality of Information and Communications Technology (QuaTIC 2001)},
	Volume = {1284},
	Year = {2001}}

@proceedings{SISY-2003a-Proceedings,
	Address = {Subotica, Serbia},
	Booktitle = {Proceedings of the 1st Serbian-Hungarian Joint Symposium on Intelligent Systems (SISY 2003)},
	Month = {19--20~} # sep,
	Publisher = {IEEE Computer Society},
	Title = {Proceedings of the 1st Serbian-Hungarian Joint Symposium on Intelligent Systems (SISY 2003)},
	Year = {2003}}

@proceedings{Sondergaard.H-1997a-ACSE,
	Address = {Melbourne, Victoria, Australia},
	Booktitle = {Proceedings of the ACM SIGCSE 2nd Australasian Conference on Computer Science Education (ACSE 1997)},
	Editor = {Harald S{\o}ndergaard and A. John Hurst},
	Isbn = {0-89791-958-0},
	Publisher = {ACM},
	Title = {Proceedings of the ACM SIGCSE 2nd Australasian Conference on Computer Science Education (ACSE 1997)},
	Year = {1997}}

Ignore Space Show notes View Koli_2017/Koli_2017_Stanger.tex 0 → 100644
\documentclass[sigconf, review, anonymous, capitalise]{acmart} % capitalise option is for cleveref \usepackage{tcolorbox} \usepackage{listings} \usepackage{tikz} % \usepackage{flafter} \usepackage{booktabs} \usepackage{subfig} \usetikzlibrary{calc} \usetikzlibrary{graphs} \usetikzlibrary{shapes} \usetikzlibrary{positioning} \usetikzlibrary{arrows.meta} \lstloadlanguages{PHP} \lstset{language=PHP,basicstyle=\small\ttfamily} \newcommand{\anoninstitution}{Anonymous Institution} % \newcommand{\anoninstitution}{University of Otago} \newcommand{\anoncountry}{Anonymous Country} % \newcommand{\anoncountry}{New Zealand} % Copyright %\setcopyright{none} %\setcopyright{acmcopyright} %\setcopyright{acmlicensed} \setcopyright{rightsretained} %\setcopyright{usgov} %\setcopyright{usgovmixed} %\setcopyright{cagov} %\setcopyright{cagovmixed} % DOI \acmDOI{} % ISBN \acmISBN{} %Conference \acmConference[Koli Calling '17]{17th Koli Calling International Conference on Computing Education Research}{November 2017}{Koli, Finland} \acmYear{2017} \copyrightyear{2017} \acmPrice{} \title{Semi-automated grading of SQL \\ schemas using database unit testing} \author{Nigel Stanger} \orcid{orcid.org/0000-0003-3450-7443} \affiliation{ \institution{University of Otago} \department{Department of Information Science} \city{Dunedin} \country{New Zealand} } \email{nigel.stanger@otago.ac.nz} \begin{document} \begin{abstract} A key skill that students must learn when studying relational database concepts is how to design and implement a database schema in SQL. This skill is often tested using an assignment where students derive an SQL schema from a natural language specification. Grading of such assignments can be complex and time-consuming, and novice database students often lack the skills to evaluate whether their implementation accurately reflects the specification's requirements. In this paper we describe a novel semi-automated system for grading student-created SQL schemas, based on a unit testing model. The system verifies whether the schema conforms to a machine-readable version of the specification, and runs in two modes: a staff mode for grading, and a reduced functionality student mode that students can use to check whether their schema meets minimum requirements. Analysis of student performance over the period this system was in use shows evidence of improved grades as a result of students using the system. \end{abstract} \begin{CCSXML} <ccs2012> <concept> <concept_id>10003456.10003457.10003527.10003540</concept_id> <concept_desc>Social and professional topics~Student assessment</concept_desc> <concept_significance>500</concept_significance> </concept> <concept> <concept_id>10002951.10002952.10003197.10010822.10010823</concept_id> <concept_desc>Information systems~Structured Query Language</concept_desc> <concept_significance>300</concept_significance> </concept> <concept> <concept_id>10002951.10002952.10003190.10003206</concept_id> <concept_desc>Information systems~Integrity checking</concept_desc> <concept_significance>100</concept_significance> </concept> <concept> <concept_id>10010405.10010489.10010490</concept_id> <concept_desc>Applied computing~Computer-assisted instruction</concept_desc> <concept_significance>300</concept_significance> </concept> <concept> <concept_id>10011007.10011074.10011099.10011102.10011103</concept_id> <concept_desc>Software and its engineering~Software testing and debugging</concept_desc> <concept_significance>100</concept_significance> </concept> </ccs2012> \end{CCSXML} \ccsdesc[500]{Social and professional topics~Student assessment} \ccsdesc[300]{Information systems~Structured Query Language} \ccsdesc[100]{Information systems~Integrity checking} \ccsdesc[300]{Applied computing~Computer-assisted instruction} \ccsdesc[100]{Software and its engineering~Software testing and debugging} \keywords{automated grading, SQL, database schema, data definition language (DDL), student performance, unit testing} \maketitle \section{Introduction} Any introductory database course needs to cover several core concepts, such as what is a database, what is a logical data model, and how to create and interact with a database. Typically such courses focus on the relational model and its embodiment in SQL database management systems (DBMSs). This is because the relational model provides a sound theoretical framework for discussing key database concepts \cite{Date.C-2009a-SQL-and-Relational}, and because SQL DBMSs are so widely used. The shadow of SQL is so strong that even non-relational systems have adopted some form of SQL-like language in order to leverage existing knowledge (e.g., OQL \cite{Cattell.R-2000a-ODMG3}, HiveQL \cite{Apache-2017a-Hive}, and CQL \cite{Apache-2017a-CQL}). Courses that teach SQL usually include one or more assessments that test students' ability to create a database using SQL data definition (DDL) statements, and to interact with the database using SQL data manipulation (DML) statements. Manually grading the code submitted for such assessments can be a slow, tedious, and potentially error-prone process. Automated or semi-automated grading has been shown to improve turnaround time and consistency, and is generally received positively by students \cite{Douce.C-2005a-Automatic,Russell.G-2004a-Improving,Dekeyser.S-2007a-Computer,Prior.J-2004a-Backwash}. If the grading can be done in real time, the grading tool can even become part of a larger, interactive SQL learning environment (e.g., \cite{Kenny.C-2005a-Automated,Kleiner.C-2013a-Automated,Mitrovic.A-1998a-Learning,Russell.G-2004a-Improving,Sadiq.S-2004a-SQLator}). While there have been many prior efforts to automatically or semi-automatically grade SQL DML (see \cref{sec-literature}), there appear to have been no corresponding efforts to grade SQL \emph{DDL}. There are generally two main aspects that need to be considered when grading an SQL schema implementation. First, is the DDL code (i.e., \texttt{CREATE} statements) syntactically correct? This is already dealt with quite effectively by the syntax checkers built into every SQL DBMS (although it is fair to say that the errors produced by such checkers can sometimes be obscure and unhelpful). A student who submits code containing syntax errors cannot expect to score well! A related aspect is code style (e.g., naming, formatting, indentation), but we do not consider this here. Second, does the schema meet the requirements of the problem being solved? A database schema is normally designed and implemented within the context of a specific set of requirements, so verifying that the implemented SQL schema fulfills these requirements is an effective way to grade the implementation, and also provides a useful framework for providing feedback to students. The requirements for a database schema can be loosely divided into \emph{structure} (e.g., tables, columns, data types), \emph{integrity} (e.g., keys, constraints), and \emph{behavior} (e.g., sequences, triggers). In this paper we describe a novel system that semi-automates the grading of students' SQL schema implementations. The system takes as input a machine-readable specification of the assessment requirements and a live instance of a submitted student schema, and verifies whether the schema conforms to the specification. Rather than attempt to parse and check the \texttt{CREATE} statements directly, the system verifies the structure of the schema by issuing queries against the schema's metadata (catalog) to look for expected values such as table names, column names, etc. It verifies integrity constraints by attempting to insert known legal and illegal values. The system currently does not check behavioral aspects. The results of the tests are compared against the machine-readable specification. This process is effectively one of unit testing the schema using the specification as a framework. We use the PHPUnit database unit testing framework to carry out this process, albeit in a somewhat unorthodox way (see \cref{sec-design}). The remainder of the paper is structured as follows. In the next section we discuss related work, while \cref{sec-motivation} discusses the motivation and context for our approach. \Cref{sec-design} discusses the design of our system, and \cref{sec-evaluation} evaluates its impact. We discuss known issues and future work in \cref{sec-issues}, and conclude in \cref{sec-conclusion}. \section{Related work} \label{sec-literature} There have been many prior efforts to build systems to support students who are learning SQL. Early examples such as RDBI \cite{Dietrich.S-1993a-An-educational} and esql \cite{Kearns.R-1997a-A-teaching} were essentially just query execution environments that students could use to practice writing SQL queries. RDBI provided relatively little feedback about the correctness of a query, whereas esql could visualize the intermediate tables generated by each step of a query, enabling students to better understand the steps in its execution. Later systems became more responsive in terms of the feedback they provided to students. Systems like SQL-Tutor \cite{Mitrovic.A-1998a-Learning}, SQLator \cite{Sadiq.S-2004a-SQLator}, AsseSQL \cite{Prior.J-2004a-Backwash}, ActiveSQL\footnote{\url{https://db.grussell.org/}} \cite{Russell.G-2004a-Improving,Russell.G-2005a-Online}, SQLify \cite{Dekeyser.S-2007a-Computer}, and aSQLg \cite{Kleiner.C-2013a-Automated} all provided varying levels of syntactic and semantic feedback. Many of these took a more ``functional'' approach to checking SQL query code, i.e., verifying that the code written fulfilled the requirements of the problem, rather than focusing on the code itself. This was most commonly done by checking how much the result set of a student's query differed from the correct result set (e.g., SQLator's ``equivalence engine'' \cite{Sadiq.S-2004a-SQLator}). ActiveSQL could also detect ``hard-coded'' queries that produced the desired result, but which would fail if the data set changed \cite{Russell.G-2005a-Online}. The SQL Exploratorium \cite{Brusilovsky.P-2010a-Learning} took an interesting alternative approach, using parameterized query templates to generate the questions given to students. Given the relatively static nature of an SQL schema, we feel this verification style of approach is the most appropriate way of constructing an automated grading system for SQL DDL. As an aside, the concept of verifying a schema against its specification sounds like it should be a useful application of formal methods \cite{Spivey.J-1989a-An-introduction}, but work with formal methods and databases has historically focused either on \emph{generating} a valid schema from a specification (e.g., \cite{Vatanawood.W-2004a-Formal,Lukovic.I-2003a-Proceedings,Choppella.V-2006a-Constructing}), or on verifying schema transformation and evolution \cite{Bench-Capon.T-1998a-Report}. A formal methods approach could certainly be used to generate the model answer or rubric for grading an assessment, but this is effectively the inverse of what is needed to grade a student-written schema. Previous systems focused almost exclusively on SQL \emph{queries} using the \texttt{SELECT} statement (i.e., DML) rather than SQL \emph{schema definitions} (DDL). This focus is unsurprising given that \texttt{SELECT} is probably the most frequently used of any SQL statement, and also because it is relatively complex compared to most other SQL statements. Only a few of the systems we reviewed even mentioned schema definition. RDBI \cite{Dietrich.S-1993a-An-educational} supported a DDL, but only its own non-SQL language, while esql \cite{Kearns.R-1997a-A-teaching} simply passed anything that was not a \texttt{SELECT} statement through to the DBMS. \citeauthor{Gong.A-2015a-CS-121-Automation}'s ``CS 121 Automation Tool'' \cite{Gong.A-2015a-CS-121-Automation} focuses primarily on SQL DML statements, but appears to be extensible and could potentially be modified to support SQL DDL statements. Many previous systems have implemented some form of automated or semi-automated grading, e.g., SQLator \cite{Sadiq.S-2004a-SQLator}, AsseSQL \cite{Prior.J-2004a-Backwash}, ActiveSQL \cite{Russell.G-2004a-Improving,Russell.G-2005a-Online}, SQLify \cite{Dekeyser.S-2007a-Computer}, aSQLg \cite{Kleiner.C-2013a-Automated}, \citeauthor{Gong.A-2015a-CS-121-Automation}'s ``CS 121 Automation Tool'' \cite{Gong.A-2015a-CS-121-Automation}, and XDa-TA\footnote{\url{http://www.cse.iitb.ac.in/infolab/xdata/}} \cite{Bhangdiya.A-2015a-XDa-TA,Chandra.B-2015a-Data,Chandra.B-2016a-Partial}. Some (e.g., SQLator, AsseSQL) provided only correct/incorrect responses, while others (e.g, ActiveSQL, SQLify, aSQLg, XDa-TA) had more sophisticated schemes for assigning partial credit, such as ActiveSQL's accuracy score, SQLify's eight-level scale, and XDa-TA's ability to generate data sets designed to catch common errors. In either case, these systems were often able to automatically mark a significant fraction (e.g., about a third for SQLator \cite{Sadiq.S-2004a-SQLator}) of submitted queries as correct without human intervention, thus reducing marking load for teachers. To our knowledge there has been no work on automated grading of SQL DDL statements. While dealing with \texttt{CREATE} statements should be simpler than dealing with \texttt{SELECT} statements, the ability to at least semi-automate the grading of SQL schema definitions should reap rewards in terms of more consistent application of grading criteria, and faster turnaround time \cite{Douce.C-2005a-Automatic,Russell.G-2004a-Improving,Dekeyser.S-2007a-Computer,Prior.J-2004a-Backwash}. Another branch of related work is systems that actively aid students in learning SQL, e.g., SQL-Tutor \cite{Mitrovic.A-1998a-Learning}, SQLator \cite{Sadiq.S-2004a-SQLator}, ActiveSQL \cite{Russell.G-2004a-Improving}, and aSQLg \cite{Kleiner.C-2013a-Automated}. SQL-Tutor is a typical example of this category. It was an intelligent teaching system that provided students with a guided discovery learning environment for SQL queries, and used constraint-based modeling \cite{Ohlsson.S-1992a-Constraint-based,Ohlsson.S-2016a-Constraint-based} to provide feedback to students. \citeauthor{Kenny.C-2005a-Automated} \cite{Kenny.C-2005a-Automated} described a similar SQL learning system that incorporated an assessment of a student's previous progress. This enabled a more personalized and adaptive approach to student learning, where feedback was tailored according to student progress. There is relatively little prior work on unit testing of databases. Most authors working in this area have focused on testing database \emph{applications} rather than the database itself (e.g., \cite{Binnig.C-2008a-Multi-RQP,Chays.D-2008a-Query-based,Marcozzi.M-2012a-Test,Haller.K-2010a-Test}). \citeauthor{Ambler.S-2006a-Database} discusses how to test the functionality of a database \cite{Ambler.S-2006a-Database}, while \citeauthor{Farre.C-2008a-SVTe} describe how to test the ``correctness'' of a schema \cite{Farre.C-2008a-SVTe}, focusing mainly on consistency of constraints. Neither consider how to verify that a database schema conforms to the requirements of the original specification. \section{Motivation} \label{sec-motivation} Since 1989, our department has offered some form of mandatory database concepts course, typically one semester during the second year\footnote{\anoncountry\ Bachelor's degrees normally comprise three years of study.} of study, building on a short introduction to basic data management concepts in the first year. Typical of courses of this nature, it covered core topics such as the relational model, relational algebra, data integrity, SQL DDL and DML, and a varying mixture of other database topics such as transactions, concurrency control, triggers, and security. Assessment of SQL skills was typically carried out using a mixture of assignments and tests. From 2001 to 2003, we assessed students' SQL DDL skills with an in-lab practical examination. Students were given a fictional scenario specification, and had 100 minutes in which to modify a provided schema template with additional tables, constraints, etc. The test was generally easier to grade than the more ``realistic'' practical assignment that we had used in earlier years, as the scenario specification tended to be quite tightly specified and thus less open to (mis)interpretation. However, the test experience was quite stressful to students due to the limited timeframe and limited access to online references. We did not attempt to automate the grading of these tests. The most common approach we have used to assess students' SQL DDL skills was a practical assignment, where students had 3--4 weeks in which to implement a database schema based on a fictional scenario specification. The scenario posed that the student was a database developer involved in a larger project, and that the specification was the output of the requirements analysis phase. An entity-relationship diagram (ERD) of a typical scenario that we used (known as the ``BDL'' scenario) is shown in \cref{fig-ERD}. \begin{figure}[hb] \centering \includegraphics[width=0.85\columnwidth, keepaspectratio]{images/BDL_ERD.pdf} \caption{ERD of the ``BDL'' scenario used to assess SQL DDL skills (Information Engineering notation).} \label{fig-ERD} \end{figure} Prior to 2001, the specifications for the scenarios used in this assignment were deliberately somewhat loosely defined and often contained areas that were under-specified or ambiguous. At the time the course also included some data modeling content, so this approach enabled students to explore the different ways that a conceptual model could be converted into an implementation. This of course led to variation across student submissions, due to differing interpretations of the under-specified elements. Automation of grading was problematic, especially when students chose different database structures, or different names for tables and columns, than what was ``expected''. We therefore made no significant attempt to automate grading under this approach. By 2004 the data modeling content was being migrated into other courses, so we took the opportunity to tighten up the scenario specifications to reduce ambiguity. In 2013 we took the further step of ``freezing'' the specification, i.e., students were not permitted to make arbitrary changes to the specification without strong justification, and even then only if the changes did not impact how client programs interacted with the database. For example, if a column needed to be restricted to a discrete set of values, a student could---if they wished---choose to go beyond the specification and enforce this requirement using a separate lookup table rather than a \texttt{CHECK} constraint, as long as this did not change the originally specified structure of the table. This meant that students could still exercise some flexibility in their schema implementations where appropriate, and ensured that the assignment was not just a mechanical translation exercise. The in-scenario rationale for freezing the specification was that other developers were independently using the same specification to code end-user applications. Any significant variation from the specification would therefore break those applications. This approach tested not only students' ability to write SQL DDL, but also their ability to correctly interpret and convert a natural language database specification into a corresponding SQL schema. This approach seemed effective, but maintaining consistent grading standards across a large number of submissions (on the order of 50--70) was difficult, due to the often large number of distinct gradable elements implied by the specification. This required a complex and highly-detailed rubric to be constructed so that no element was missed, and the grading process consequently took a significant amount of time, especially when feedback to students was required. In 2012 a significant change to the structure of the course resulted in higher grading workloads and increased time pressure, prompting greater interest in the possibility of at least semi-automating the grading of this assignment. This was now more feasible than in earlier years due to the more tightly constrained nature of the project specification. Another motivation for automated grading was that it can sometimes be difficult for database novices to know whether they are on the right track while implementing a specification. If a limited version of the grading tool was available to students before the submission deadline, it could be used to provide feedback on their progress. We decided to define a minimum set of requirements for the assignment: students' SQL code should be syntactically correct, and it should include all tables and columns---with correct names and appropriate data types---as detailed in the specification. If a student satisfied these minimum requirements, they were guaranteed to score at least 50\%. They could check whether their schema met these requirements prior to the deadline by submitting their schema for checking through a web application. That way they (and we) could be more certain that at least the core structure of their schema was correct. After the deadline, teaching staff used a console application to check for correct implementation of other aspects of the schema, such as integrity constraints. We trialled a prototype of the staff-facing application in 2012, and rolled out the student-facing application in 2013. \section{System design} \label{sec-design} The architecture of our system is shown in \cref{fig-architecture}. Its core function is to check whether a student's schema conforms to the assignment specification, by automatically comparing their schema submission against a machine-readable version of the specification. This is essentially a unit testing approach, so we decided to build the system around an existing unit testing framework. \begin{figure}[h] \sffamily \begin{tikzpicture}[every node/.style={draw, minimum height=7.5mm, inner sep=1em}] \node (console) {\shortstack{Console app \\ \footnotesize(staff mode)}}; \coordinate[below=5mm of console.south] (console port); \node[anchor=north west, minimum width=6cm] (driver) at ($(console.south west) - (0,5mm)$) {Main driver}; \node[anchor=south east] (web) at ($(driver.north east) + (0,5mm)$) {\shortstack{Web app \\ \footnotesize(student mode)}}; \coordinate[below=5mm of web.south] (web port); \node[below=5mm of driver] (phpunit) {PHPUnit}; \node[left=5mm of phpunit] (spec) {\shortstack{Assign. \\ spec.}}; \node[right=5mm of phpunit] (reporting) {Reporting}; \coordinate[above=5mm of reporting.north] (reporting port); \node[cylinder, shape border rotate=90, below=5mm of phpunit, aspect=0.1] (database) {Database}; \path (database.before top) -- (database.after top) coordinate[midway] (dbtop); \node[right=5mm of database] (schema) {\shortstack{Student's \\ schema}}; \graph { [edges={draw, arrows={{Stealth}-{Stealth}}}] {(console), (web)} -> {(console port), (web port)}, (driver) -> (phpunit), (phpunit) -> (dbtop), }; \graph { [edges={draw, arrows={-{Stealth}}}] (reporting) -> (reporting port), (spec) -> (phpunit), (schema) -> (database), }; \end{tikzpicture} \caption{System architecture.} \label{fig-architecture} \end{figure} There are surprisingly few frameworks for performing unit tests that interact with a database, probably due to the complexities involved. In conventional application unit testing it is relatively simple to create mocked interfaces for testing purposes. With a database, however, we need to create tables, populate them with appropriate test data, verify the state of the database after each test has run, and clean up the database for each new test \cite{Bergmann.S-2017a-PHPUnit}. Cleaning up is particularly crucial, as the order of tests is often not deterministic, i.e., the tests are not guaranteed to run in exactly the same order each time. Tests that change the state of the database may therefore affect the results of later tests in unpredictable ways. We are aware of only four unit testing frameworks that provide specific support for database unit tests: DbUnit for Java,\footnote{http://dbunit.sourceforge.net/} DbUnit.NET,\footnote{http://dbunit-net.sourceforge.net/} Test::DBUnit for Perl,\footnote{http://search.cpan.org/~adrianwit/Test-DBUnit-0.20/lib/Test/DBUnit.pm} and PHPUnit.\footnote{https://phpunit.de/} We chose to implement the system in PHP, as it enabled us to quickly prototype the system and simplified development of the student-facing web application. A similar approach could be taken with any of the other frameworks, however. Our database teaching was mainly based around Oracle, but the system could be adapted relatively easily for use with any DBMS supported by PHP's PDO extension. This would require implementing an additional layer to abstract the internal details of how each DBMS structures its catalog tables. The core of the system is the \textsf{Main driver} component (see \cref{fig-architecture}). This can execute in either \emph{student mode}, which runs only a subset of the available tests, or in \emph{staff mode}, which runs all available tests (the nature of these tests will be discussed in more detail shortly). The mode is determined by the client application, as shown in \cref{fig-architecture}. Currently student mode is accessed through a web application, while staff mode is accessed through a console application, but each could easily be implemented in a variety of different ways. The main driver uses the \textsf{Reporting} module to generate test output in either HTML or plain text, as appropriate. The assignment specification (\textsf{Assign.\ spec.} in \cref{fig-architecture}) is encoded as a collection of subclasses of PHPUnit's \texttt{TestCase} class. Each subclass specifies the properties of one database table. \Cref{fig-test-class} shows a fragment of the class corresponding to the \textsf{Product} table in \cref{fig-ERD}. The methods of this class return various properties of the table: \begin{description} \item[\texttt{getTableName()}] returns the expected name of the table. \item[\texttt{getColumnList()}] returns an array of column specifications, keyed by expected column name. Each column specification includes a generic data type (text, number, date, or binary), a list of corresponding acceptable SQL data types (e.g., \texttt{VARCHAR}, \texttt{DECIMAL}), whether the column permits nulls, and a known legal value for general testing. Where applicable, it may include minimum and maximum column lengths (precision for numeric types), and the number of decimal places (scale). Underflow and overflow values, and lists of known legal and illegal values can also be specified to test the boundary conditions of integrity constraints. \item[\texttt{getPKColumnList()}] returns an array of column names that comprise the primary key of the table. \item[\texttt{getFKColumnList()}] returns an array of foreign key specifications (where applicable), keyed by the name of the referenced table. Each specification contains an array of column names that comprise that foreign key. \end{description} \begin{figure} \begin{lstlisting} public function getTableName() { return 'PRODUCT'; } public function getColumnList() { return array( 'PRODUCT_CODE' => array( 'generic_type' => 'NUMBER', 'sql_type' => array('NUMBER', 'INTEGER'), 'min_length' => 8, 'max_length' => 8, 'decimals' => 0, 'test_value' => 87654321, 'nullable' => false), 'DESCRIPTION' => array( ... ), 'STOCK_COUNT' => array( 'generic_type' => 'NUMBER', 'sql_type' => array('NUMBER', 'INTEGER'), 'min_length' => 5, 'max_length' => 6, 'decimals' => 0, 'underflow' => -1, 'overflow' => 100000, 'legal_values' => array(0, 99999), 'test_value' => 456, 'nullable' => false), 'RESTOCK_LEVEL' => array( ... ), 'MINIMUM_LEVEL' => array( ... ), 'LIST_PRICE' => array( 'generic_type' => 'NUMBER', 'sql_type' => array('NUMBER', 'INTEGER'), 'min_length' => 7, 'max_length' => 8, 'decimals' => 2, 'underflow' => -0.01, 'overflow' => 100000.00, 'legal_values' => array(0, 99999.99), 'test_value' => 123.99, 'nullable' => false), 'ASSEMBLY_MANUAL' => array( 'generic_type' => 'BINARY', 'sql_type' => array('BLOB'), 'test_value' => "NULL", 'nullable' => true), 'ASSEMBLY_PROGRAM' => array( ... ) ); } public function getPKColumnList() { return array( 'PRODUCT_CODE' ); } public function getFKColumnList() { return array(); // no FKs in this table } \end{lstlisting} \caption{Fragment of the \textsf{Product} table specification.} \label{fig-test-class} \end{figure} Each table specification also requires the definition of two distinct sets of tests that will be run on the database. The first set verifies the structural elements of the table (columns, data types, etc.), thus verifying that a schema meets the minimum requirements of the assignment. This is done by issuing queries against the metadata (catalog) of the schema for elements like tables, columns, and data types. When in student mode, only this set of tests is run. An empty data fixture (specified using a separate XML document) is also required to support this set of tests. The second set of tests verifies the integrity elements of the table, i.e., its constraints. The only integrity constraints that are explicitly tested are primary and foreign keys, which are again verified using queries against the schema's metadata, and nullability (\texttt{NOT NULL}), which is tested by attempting to insert a null. The remaining constraints are tested by attempting to insert lists of known legal and illegal values, an approach consistent with normal unit testing practice. When in staff mode, both this set of tests and the set of structural tests are run. A known-legal data fixture is also required to support this set of tests. \begin{figure} \includegraphics[width=0.95\columnwidth,keepaspectratio]{images/web_output.png} \caption{Example of student mode output (web app) for the \textsf{Staff} table.} \label{fig-student-output} \end{figure} The way that the system runs the tests is somewhat unusual, in two ways. First, database unit testing frameworks are generally designed around the concept of testing database-backed \emph{applications}, rather than testing the database itself. Second, in typical unit testing, the tests are essentially standalone code units that are automatically executed in an order determined by the unit testing framework. The framework resolves dependencies among tests and handles collation of test results internally, reporting only success, failure, or ``error'' back to the client. In fact, most unit testing frameworks appear to be designed to be called from a specific tool (e.g., the \texttt{phpunit} command line tool), rather than embedded as a library into other programs. Our system effectively inverts (or perhaps subverts) this approach. The main driver explicitly creates test suites itself and executes them directly, listening for the results of each test and collating them. This is because we need to be able to control the order in which tests are executed. If the structural tests fail, there is little point in running the integrity tests, as they will only generate a stream of errors. Similarly, if a column is missing, there is no point in running the data type and length tests. (Note that while PHPUnit supports grouping of tests, dependencies can only be defined between individual tests, not between groups.) It is quite feasible to add table properties and tests beyond those already mentioned, as the specification is encoded as ordinary PHP code and is therefore constrained only by what is possible in PHP. All a teacher needs to do is add custom properties to the table specification, then add tests that use those custom properties. Custom tests are easily registered with the appropriate test set (structure or integrity) using PHPUnit's \texttt{@group} annotation. Students can check their schema (\textsf{Student's schema} in \cref{fig-architecture}) by loading it into their personal database account (i.e., creating the tables), then entering their database login credentials into a web application (\textsf{Web app} in \cref{fig-architecture}). This calls the main driver in student mode and accesses the student's schema directly using the provided credentials. Only the structural set of tests is run, and the output is displayed in the web browser. \Cref{fig-student-output} shows an example of the output produced by student mode for the \textsf{Staff} table in \cref{fig-ERD}. A teacher can check further aspects of a student's schema using the \textsf{Console app} shown in \cref{fig-architecture}. To ensure a clean testing environment, the teacher does not connect directly to the student's database account, as the student may (for example) continue to use it for other coursework. Instead, the teacher loads the student's submitted schema code under a separate account used only for grading purposes, and erases the schema before moving to the next submission. The login credentials for the grading account are specified in the console application's configuration file. Assuming that there are no syntax errors in the student's code,\footnote{If there are errors, then the student has clearly not met the minimum requirements!} the teacher then runs the console application, which calls the main driver in staff mode. The main driver connects to the pre-configured grading account, runs all the available tests, and displays the output in the terminal window. \Cref{fig-staff-output} shows an example of the output produced by staff mode for the \textsf{Product} table in \cref{fig-ERD}. % ANSI terminal colours for Terminal.app; see https://en.wikipedia.org/wiki/ANSI_escape_code#Colors % grey 203, 204, 205 % green 37 188 36 % red 194, 54, 33 \definecolor{test grey}{rgb}{0.796,0.800,0.804} \definecolor{test green}{rgb}{0.145,0.737,0.141} \definecolor{test red}{rgb}{0.761,0.212,0.129} \tcbset{boxsep=0pt, boxrule=0pt, arc=0pt, left=0pt, right=0pt, top=0.5pt, bottom=0.5pt} \newlength{\dothskip} \setlength{\dothskip}{0.72cm} \newlength{\dotvskip} \setlength{\dotvskip}{-1.25ex} \newlength{\codeskip} \setlength{\codeskip}{-0.5ex} \begin{figure} \ttfamily\scriptsize \begin{tabbing} 0123\=\kill \tcbox[colback=test grey]{NOTE: Checking structure of table Product.} \\[\codeskip] TEST: [[ Product ]] \\ \> \textcolor{test green}{+ OK} \\ \tcbox[colback=test green]{+++ PASSED: Table Product exists.} \\[\codeskip] TEST: [[ Product.Product\_code ]] \\ \> \textcolor{test green}{+ OK} \\[\dotvskip] \hspace{\dothskip}\vdots \\ \tcbox[colback=test green]{+++ PASSED: Table Product contains all the expected columns.} \\[\codeskip] TEST: [[ Product.Product\_code: data type is NUMBER \| INTEGER ]] \\ \> \textcolor{test green}{+ OK} \\[\dotvskip] \hspace{\dothskip}\vdots \\ \tcbox[colback=test green]{+++ PASSED: All columns of table Product have data types compatible with the}\\[\codeskip] \tcbox[colback=test green]{specification.} \\[\codeskip] TEST: [[ Product.Product\_code precision and scale = 8 (with scale 0) ]] \\ \> \textcolor{test green}{+ OK} \\[\dotvskip] \hspace{\dothskip}\vdots \\ \tcbox[colback=test green]{+++ PASSED: All columns of table Product have lengths compatible with the} \\[\codeskip] \tcbox[colback=test green]{specification.} \\[\codeskip] TEST: [[ Product PK ]] \\ \> \textcolor{test green}{+ OK} \\ \tcbox[colback=test green]{+++ PASSED: Primary key of table Product exists.} \\[\codeskip] TEST: [[ Product PK: Product\_code ]] \\ \> \textcolor{test green}{+ OK} \\ \tcbox[colback=test green]{+++ PASSED: Primary key of table Product includes (only) the expected} \\[\codeskip] \tcbox[colback=test green]{columns.} \\[\dotvskip] \hspace{\dothskip}\vdots \\ \tcbox[colback=test grey]{NOTE: Testing constraints of table Product.} \\[\codeskip] TEST: [[ Product.Stock\_count accepts ``0'' ]] \\ \> \textcolor{test green}{+ OK} \\ TEST: [[ Product.Stock\_count accepts ``99999'' ]] \\ \> \textcolor{test green}{+ OK} \\ TEST: [[ Product.Restock\_level accepts ``0'' ]] \\ \> \textcolor{test red}{- FAILED! Column Product.Restock\_level won't accept legal value 0.} \\ \textcolor{test red}{Failed asserting that false is true.} \\ TEST: [[ Product.Restock\_level accepts ``99999'' ]] \\ \> \textcolor{test green}{+ OK} \\ TEST: [[ Product.Minimum\_level accepts ``0'' ]] \\ \> \textcolor{test red}{- FAILED! Column Product.Minimum\_level won't accept legal value 0.} \\ \textcolor{test red}{Failed asserting that false is true.} \\ TEST: [[ Product.Minimum\_level accepts ``653'' ]] \\ \> \textcolor{test green}{+ OK} \\[\dotvskip] \hspace{\dothskip}\vdots \\ \tcbox[colback=test red, coltext=test grey]{--- FAILED: 2 of 8 legal values tested were rejected by a CHECK constraint.} \end{tabbing} \vskip-1ex \caption{Example of staff mode output (console app) for the \textsf{Product} table.} \label{fig-staff-output} \end{figure} \section{Evaluation} \label{sec-evaluation} Unfortunately, the system was conceived as a practical solution to a perceived teaching issue, rather than as a research project with formal evaluation in mind. We therefore did not carry out any user evaluations with students that used the system. From the perspective of the teaching staff, we found that the total amount of time taken to grade the relevant assignment was reduced only a little, as the system only semi-automated the grading process, and we still needed to convert the system's output into corresponding grades and meaningful feedback (see \cref{sec-issues} for further discussion of this). There were also a surprising number of submissions that did not meet the minimum requirements and thus had to be manually graded. On a more positive note, the system did automatically ensure that all gradable elements were checked, which improved consistency, and made the subjective experience of grading much less painful. However, we were able to analyze whether use of the system had any impact on student performance in the relevant assignment, as we had extensive historical data of student grades. We collated data for the period 2009--2016 (there are no data for 2017 because the course has been discontinued; see \cref{sec-issues}), which encompassed several different permutations of scenario and available system modes, as summarized in \cref{tab-data}. The assignment counted for 15\% of a student's total grade in 2009 and 2010, and 10\% in subsequent years. The grade distributions for the assignment in each year are shown in \cref{fig-distributions}. \begin{table} \caption{Historical characteristics of the database implementation assignment, 2009--2016.} \label{tab-data} \begin{tabular}{rrrrll} \toprule & Class & Median & Mean & & Modes \\ Year & size & GPA\textsuperscript{\emph{a}} & (\%)\textsuperscript{\emph{b}} & Scenario & used \\ \midrule 2009 & 46 & --\textsuperscript{\emph{c}} & 77.5 & ``postgrad'' & -- \\ 2010 & 68 & 3.4 & 73.4 & ``student records'' & -- \\ 2011 & 64 & 3.9 & 71.8 & ``used cars'' & -- \\ \midrule 2012 & 75 & 3.4 & 69.2 & ``BDL'' & staff \\ 2013 & 77 & 3.2 & 84.3 & ``student records'' & both \\ \midrule 2014 & 49 & 3.4 & 77.6 & ``used cars'' & both \\ 2015 & 71 & 3.0 & 69.2 & ``used cars'' & neither \\ 2016 & 75 & 3.5 & 71.0 & ``BDL'' & staff \\ \bottomrule \multicolumn{6}{l}{\footnotesize \textsuperscript{\emph{a}} On a 9-point scale where C-- = 1, A+ = 9. Value is for the specified year only.} \\ \multicolumn{6}{l}{\footnotesize \textsuperscript{\emph{b}} For students who submitted the assignment.} \\ \multicolumn{6}{l}{\footnotesize \textsuperscript{\emph{c}} Complete GPA data for 2009 were not available.} \\ \end{tabular} \end{table} \begin{figure} \includegraphics[width=\columnwidth,keepaspectratio]{grades_all_years.pdf} \caption{Comparison of grade distributions for the database implementation assignment, 2009--2016.} \label{fig-distributions} \end{figure} The horizontal rule in \cref{tab-data} between 2011 and 2012 marks a significant reorganization of the course's curriculum, and also a switch from first to second semester.\footnote{First semester at the \anoninstitution\ runs from March to June, second semester from July to October.} (2012 was also the year we deployed the first prototype of staff mode.) The horizontal rule between 2013 and 2014 marks a shift from second semester back to first semester. The system was not used at all in 2015 due to different staff teaching the course, and student mode was not made available in 2016 due to technical issues delaying its deployment beyond the point where it would be useful. These differences have provided us with a natural experiment with some interesting points for comparison. The mean grade for the assignment drifted slowly downwards from 2009 to 2012. This reversed dramatically in 2013, however, the year we first deployed student mode. As can be seen in \cref{fig-distributions}, the grades are not normally distributed (they typically have negative skew), so we used a Mann-Whitney \emph{U} test to determine whether the increase in mean from 2012 to 2013 was statistically significant. This showed that the increase was highly significant (\(p \approx 10^{-9}\)). The 2013 mean was also significantly higher than both 2010 (\(p \approx 0.0002\)) and 2011 (\(p \approx 10^{-6}\)), but not significantly higher than 2009. The mean decreased significantly again in 2014 (\(p \approx 0.0012\)), the second year that the system was used, and even more dramatically in 2015 (\(p \approx 0.0005\)), when the system was not used at all. The increase from 2015 to 2016 was not significant. Even more interesting, if we compare the performance between the years that student mode was not available (2009--2012 and 2015--2016, mean 71.6\%) and the years it was (2013--2014, mean 81.7\%), there is again a highly statistically significant increase in the mean (\(p \approx 10^{-8}\)). This difference is strikingly illustrated in \cref{fig-distribution-comparison}, which compares the grade distributions for both periods. During the period that student mode was available, the lowest grade awarded was 46\%, contrasted with a much longer tail of lower grades when student mode was not available. This strongly suggests that the introduction of student mode had a positive impact on students' ability to complete the assignment more effectively. \begin{figure} \centering \subfloat[]{\includegraphics[trim=0 0 0 72,scale=0.6]{grades_without_student_mode.pdf}} \qquad \subfloat[]{\includegraphics[trim=0 0 0 72,scale=0.6]{grades_with_student_mode.pdf}} \caption{Comparison of grade distributions (a) without student mode, (b) with student mode.} \label{fig-distribution-comparison} \end{figure*} There are some potential confounding factors to consider, however. First, not only was 2013 the first year that student mode was available, it was also the first year that the assignment specification was ``frozen'' (as discussed in \cref{sec-motivation}). It could be argued that this improved grades due to students having less flexibility, and thus less opportunity for misinterpretation, than in previous years. However, the assignment specification was also ``frozen'' in all subsequent years, and there is considerable variation in the grades achieved over this period, especially in 2015. It therefore seems unlikely that this was a factor in student performance. Second, the observed effect may be due to the imposition of minimum requirements from 2013 onwards. Since any schema that met the minimum requirements would score at least 50\%, we could reasonably expect this to have a positive effect on the mean. If we compare with the period where minimum requirements were not enforced, we do find a significant increase in the mean, from 72.4\% to 75.6\% (\(p \approx 0.047\)). We argue, however, that this effect would be minimal in the absence of a convenient mechanism for students to check conformance with the minimum requirements. Indeed, for 2013--2014 when student mode was available, the mean (81.7\%) was significantly higher (\(p \approx 10^{-9}\)) than for 2015--2016 (70.2\%) when student mode was not available. This is further supported by no significant difference between the means for 2009--2012 and 2015--2016. Third, the switch to second semester in 2012--2013 could have negatively impacted students' performance by increasing the length of time between their exposure to basic data management concepts in first year, and their entry into the second year database course. In effect, they had longer to forget relevant material they learned in first year. If so, we could reasonably expect the grades in second semester offerings of the course to be lower. However, mean grades for second semester offerings (76.9\%) were in fact significantly \emph{higher} (\(p \approx 0.015\)) than those for first semester offerings (72.9\%). This should not be surprising, given that 2013 (second semester) had the highest grades overall. This effectively rules out semester changes as a factor. Fourth, perhaps the years with higher grades used less complex---and therefore easier---scenarios. To test this, we computed the following database complexity metrics for each of the four scenarios used: database complexity index (DCI) \cite{Sinha.B-2014a-Estimation}; referential degree (RD), depth of referential tree (DRT), and number of attributes (NA) \cite{Calero.C-2001a-Database,Piattini.M-2001a-Table}; database complexity (DC) \cite{Pavlic.M-2008a-Database}; and ``Software Metric Analyzer for Relational Database Systems'' (SMARtS) \cite{Jamil.B-2010a-SMARtS}. The results are shown in \cref{tab-metrics}. All but the DRT metric clearly showed the ``BDL'', ``used cars'', and ``student records'' scenarios to be of comparable complexity, while the ``postgrad'' scenario was noticeably less complex. It therefore seems unlikely that scenario complexity is a factor in student performance. It is also interesting to note that the ``used cars'' scenario was used in both 2014 and 2015, and yet the 2015 grades were significantly \emph{lower} than those for 2014. The only clear difference here is that our system was not used in 2015. \begin{table} \caption{Database complexity metrics for the scenarios used over the period 2009--2016.} \label{tab-metrics} \begin{tabular}{lrrrrrr} \toprule Scenario & DCI & RD & NA & DRT & DC & SMARtS \\ \midrule ``postgrad'' & 277 & 9 & 32 & 7 & 37 & 28.75 \\ ``student records'' & 367 & 12 & 43 & 9 & 61 & 38.25 \\ ``BDL'' & 370 & 11 & 50 & 8 & 59 & 40.75 \\ ``used cars'' & 380 & 13 & 46 & 7 & 53 & 36.50 \\ \bottomrule \end{tabular} \end{table} Fifth, class size could be a factor. We might plausibly expect a smaller class to have a more collegial atmosphere that promotes better learning. However, if we look at the sizes of the classes in \cref{tab-data}, we can see no discernible pattern between class size and performance. Indeed, both the best (2013) and worst (2012, 2015) performances came from classes of similar size (75, 77, and 71, respectively). Sixth, it could be that better performance occurred in years where the students were just more capable in general. We obtained annual GPA data for the students enrolled in each year, and computed the median as an indication of the general capability of the class. Looking at \cref{tab-data}, we can immediately see that the year with the best results (2013) was also the year with the second-lowest median GPA (3.2). Contrast this with the poorer performance in 2012, where the median GPA was 3.4. In both years that student mode was available, median GPA was lower than or the same as in most other years, yet performance was better than in years with higher median GPA. This argues against the idea that we simply had a class full of very capable students in the better performing years. Seventh, the timing of the assignment varied across the period, occurring either early (2012--2014), halfway (2009--2010), or late (2011, 2015--2016) in the semester, depending on the ordering of course content. The mean grade for early timing (77.0\%) was significantly higher than for both halfway (72.2\%, \(p \approx 0.019\)) and late (71.8\%, \(p \approx 0.013\)), while there was no significant difference between the halfway and late means. This suggests that scheduling the assignment early in the semester may have a positive effect on grades, and the period of early timing does overlap the period that student mode was available. However, as noted earlier there was a highly significant difference in mean between 2012 and 2013. There is also a smaller, but still significant difference in mean between 2012 and 2014 (\(p \approx 0.0015\)). We therefore conclude that while scheduling the assignment early in the semester may have had some positive effect on student performance, it does not explain all of the positive effect seen in 2013 and 2014. Finally, perhaps the different weightings of the assignment (15\% in 2009--2010 vs.\ 10\% in 2011--2016) affected student motivation. It could be argued that the higher weighting in 2009--2010 provided a greater incentive for students to work more, as the potential reward was greater. If so, we should expect better performance in 2009--2010. Indeed, we do find this: the mean for 2009--2010 is 75.1\%, while that for 2011--2016 is 73.9\%, a statistically significant decrease (\(p \approx 0.034\)). However, since this change occurred well before our system was even conceived of, let alone implemented, it cannot be a factor in the improved performance seen in 2013 and 2014. The sum of this evidence provides a strong argument in favor of student mode being the main factor in improving student grade performance in the database implementation assignment during 2013 and 2014. The most plausible explanation for this effect is that the direct feedback provided by the student mode web application enabled students to more effectively correct the structural aspects of their schema to meet the minimum requirements, giving them more time to work on the integrity aspects of their schema, and thus gain a higher grade. \section{Known issues \& future work} \label{sec-issues} While the use of student mode in 2013 and 2014 appears to have had a beneficial effect on student performance in the database implementation assignment, there are some outstanding issues with the system that still need to be addressed. Ideally, our system would automatically assign appropriate marks and generate meaningful feedback, and write both of these directly into a student management database. Currently, however, it only semi-automates the grading process, and it is still up to the teacher to interpret the test results, assign appropriate marks, and write feedback. Automatically generating appropriate marks is not particularly difficult, and we have already implemented the core functionality required to assign mark penalties to different kinds of error. It would be a relatively simple extension to calculate marks based on these penalties and write them directly into a student management database. Automatically generating meaningful feedback is problematic, however, due to the way that PHPUnit reports test failures. There is currently no way to control or suppress the messages generated by PHPUnit test assertions, most likely because PHPUnit is not designed to be used as an embedded library (as discussed in \cref{sec-design}). Individual tests can specify a meaningful message to be displayed, but PHPUnit will still generate somewhat obscure messages like ``Failed asserting that 0 matches expected 1'', which can be confusing for students. One particularly tricky example we encountered was if a student failed to specify a precision for a numeric column, e.g., they declared a column as just \texttt{NUMERIC} rather than \texttt{NUMERIC(5)}. Most DBMSs will in this situation assign the maximum precision (e.g., 38 significant digits for Oracle). The student then sees a message ``Failed asserting that 38 matches expected 5'', and has no idea where the ``38'' came from. This issue of unsuppressable assertion messages significantly constrained our ability to provide fully automatic, meaningful feedback to students. It may be that the only way to address this issue is to change the way PHPUnit works internally. A second issue is that any schema that fails to meet the minimum requirements will result in a large number of errors due to missing tables. This could be alleviated by more careful exception handling in the main driver. Another issue is how students made use of the student mode web application. In 2013, students tended to misuse it as a ``schema compiler'': they would submit their schema, fix only the first issue reported (rather than attempting to fix as many as possible), immediately re-submit, and so on. We had not expected so high a request rate, so the system was only single-threaded and thus not designed to handle concurrent requests. Consequently the web application suffered from waits and timeouts. An initial workaround was to log all student access and warn those who were abusing the system. We later throttled the system by enforcing a short delay between attempts by the same student. Things improved in 2014 with careful management of student expectations, but longer term, the system should be redesigned to handle concurrent loads. The system currently does not support behavioral aspects such as sequences, triggers, or stored procedures, as these are usually DBMS-specific in nature. It would not be difficult to add support for these, however, which could be included in the DBMS abstraction layer proposed in \cref{sec-design}. Looking forward, it would be interesting to extend our system to help facilitate the teaching of SQL DDL, rather than just focusing on assignment grading. This has been done previously with several systems for SQL DML, as discussed in \cref{sec-literature} (e.g., \cite{Kenny.C-2005a-Automated,Kleiner.C-2013a-Automated,Mitrovic.A-1998a-Learning,Russell.G-2004a-Improving,Sadiq.S-2004a-SQLator}). From 2017, we no longer have a dedicated second year database course. The main introduction to database concepts and SQL now occurs as part of our new (in 2017) first year ``Foundations of Information Systems'' course. This course attracted about 160 students in first semester, and about 100 students in second semester. Classes of this size further strengthen the argument for automated grading, and we are currently exploring whether our system could be used in this course in 2018. If this is feasible, we intend to carry out user evaluations with the students. \section{Conclusion} \label{sec-conclusion} In this paper we have described a novel system that semi-automates the process of grading SQL data definition code. To our knowledge, all prior work in this area has focused only on data manipulation statements such as \texttt{SELECT}, so extending this to data definition code is a key contribution of our system. The system also takes a novel approach to checking an SQL schema: rather than attempting to parse the SQL DDL code directly, it instead verifies that the schema conforms to a machine-readable version of the original specification. Finally, our system uses a unit testing approach to carry out the verification, which has not been used before in this context. While no user evaluations of the system were carried out, analysis of student performance in a relevant database assessment shows a highly statistically significant increase in mean grade in the two years that the system was available for use by students. While we cannot be absolutely certain, analysis of all the factors involved strongly suggests that use of our system by students had a positive impact on their performance in this assessment. This is an encouraging result, which we will explore further in future work. \bibliographystyle{ACM-Reference-Format} \bibliography{Koli_2017_Stanger} \end{document}

Ignore Space Show notes View

Koli_2017/Koli_2017_Stanger.tex 0 → 100644

\documentclass[sigconf, review, anonymous, capitalise]{acmart}
% capitalise option is for cleveref

\usepackage{tcolorbox}
\usepackage{listings}
\usepackage{tikz}
% \usepackage{flafter}
\usepackage{booktabs}
\usepackage{subfig}

\usetikzlibrary{calc}
\usetikzlibrary{graphs}
\usetikzlibrary{shapes}
\usetikzlibrary{positioning}
\usetikzlibrary{arrows.meta}

\lstloadlanguages{PHP}
\lstset{language=PHP,basicstyle=\small\ttfamily}

\newcommand{\anoninstitution}{Anonymous Institution}
% \newcommand{\anoninstitution}{University of Otago}
\newcommand{\anoncountry}{Anonymous Country}
% \newcommand{\anoncountry}{New Zealand}

% Copyright
%\setcopyright{none}
%\setcopyright{acmcopyright}
%\setcopyright{acmlicensed}
\setcopyright{rightsretained}
%\setcopyright{usgov}
%\setcopyright{usgovmixed}
%\setcopyright{cagov}
%\setcopyright{cagovmixed}

% DOI
\acmDOI{}

% ISBN
\acmISBN{}

%Conference
\acmConference[Koli Calling '17]{17th Koli Calling International Conference on Computing Education Research}{November 2017}{Koli, Finland} 
\acmYear{2017}
\copyrightyear{2017}

\acmPrice{}

\title{Semi-automated grading of SQL \\ schemas using database unit testing}
\author{Nigel Stanger}
\orcid{orcid.org/0000-0003-3450-7443}
\affiliation{
    \institution{University of Otago}
    \department{Department of Information Science}
    \city{Dunedin}
    \country{New Zealand}
}
\email{nigel.stanger@otago.ac.nz}

\begin{document}

\begin{abstract}
    A key skill that students must learn when studying relational database concepts is how to design and implement a database schema in SQL. This skill is often tested using an assignment where students derive an SQL schema from a natural language specification. Grading of such assignments can be complex and time-consuming, and novice database students often lack the skills to evaluate whether their implementation accurately reflects the specification's requirements. In this paper we describe a novel semi-automated system for grading student-created SQL schemas, based on a unit testing model. The system verifies whether the schema conforms to a machine-readable version of the specification, and runs in two modes: a staff mode for grading, and a reduced functionality student mode that students can use to check whether their schema meets minimum requirements. Analysis of student performance over the period this system was in use shows evidence of improved grades as a result of students using the system.
\end{abstract}

\begin{CCSXML}
<ccs2012>
<concept>
<concept_id>10003456.10003457.10003527.10003540</concept_id>
<concept_desc>Social and professional topics~Student assessment</concept_desc>
<concept_significance>500</concept_significance>
</concept>
<concept>
<concept_id>10002951.10002952.10003197.10010822.10010823</concept_id>
<concept_desc>Information systems~Structured Query Language</concept_desc>
<concept_significance>300</concept_significance>
</concept>
<concept>
<concept_id>10002951.10002952.10003190.10003206</concept_id>
<concept_desc>Information systems~Integrity checking</concept_desc>
<concept_significance>100</concept_significance>
</concept>
<concept>
<concept_id>10010405.10010489.10010490</concept_id>
<concept_desc>Applied computing~Computer-assisted instruction</concept_desc>
<concept_significance>300</concept_significance>
</concept>
<concept>
<concept_id>10011007.10011074.10011099.10011102.10011103</concept_id>
<concept_desc>Software and its engineering~Software testing and debugging</concept_desc>
<concept_significance>100</concept_significance>
</concept>
</ccs2012>
\end{CCSXML}

\ccsdesc[500]{Social and professional topics~Student assessment}
\ccsdesc[300]{Information systems~Structured Query Language}
\ccsdesc[100]{Information systems~Integrity checking}
\ccsdesc[300]{Applied computing~Computer-assisted instruction}
\ccsdesc[100]{Software and its engineering~Software testing and debugging}

\keywords{automated grading, SQL, database schema, data definition language (DDL), student performance, unit testing}

\maketitle

\section{Introduction}

Any introductory database course needs to cover several core concepts, such as what is a database, what is a logical data model, and how to create and interact with a database. Typically such courses focus on the relational model and its embodiment in SQL database management systems (DBMSs). This is because the relational model provides a sound theoretical framework for discussing key database concepts \cite{Date.C-2009a-SQL-and-Relational}, and because SQL DBMSs are so widely used. The shadow of SQL is so strong that even non-relational systems have adopted some form of SQL-like language in order to leverage existing knowledge (e.g., OQL \cite{Cattell.R-2000a-ODMG3}, HiveQL \cite{Apache-2017a-Hive}, and CQL \cite{Apache-2017a-CQL}).

Courses that teach SQL usually include one or more assessments that test students' ability to create a database using SQL data definition (DDL) statements, and to interact with the database using SQL data manipulation (DML) statements. Manually grading the code submitted for such assessments can be a slow, tedious, and potentially error-prone process. Automated or semi-automated grading has been shown to improve turnaround time and consistency, and is generally received positively by students \cite{Douce.C-2005a-Automatic,Russell.G-2004a-Improving,Dekeyser.S-2007a-Computer,Prior.J-2004a-Backwash}. If the grading can be done in real time, the grading tool can even become part of a larger, interactive SQL learning environment (e.g.,  \cite{Kenny.C-2005a-Automated,Kleiner.C-2013a-Automated,Mitrovic.A-1998a-Learning,Russell.G-2004a-Improving,Sadiq.S-2004a-SQLator}).

While there have been many prior efforts to automatically or semi-automatically grade SQL DML (see \cref{sec-literature}), there appear to have been no corresponding efforts to grade SQL \emph{DDL}. There are generally two main aspects that need to be considered when grading an SQL schema implementation. First, is the DDL code (i.e., \texttt{CREATE} statements) syntactically correct? This is already dealt with quite effectively by the syntax checkers built into every SQL DBMS (although it is fair to say that the errors produced by such checkers can sometimes be obscure and unhelpful). A student who submits code containing syntax errors cannot expect to score well! A related aspect is code style (e.g., naming, formatting, indentation), but we do not consider this here.
    
Second, does the schema meet the requirements of the problem being solved? A database schema is normally designed and implemented within the context of a specific set of requirements, so verifying that the implemented SQL schema fulfills these requirements is an effective way to grade the implementation, and also provides a useful framework for providing feedback to students. The requirements for a database schema can be loosely divided into \emph{structure} (e.g., tables, columns, data types), \emph{integrity} (e.g., keys, constraints), and \emph{behavior} (e.g., sequences, triggers).

In this paper we describe a novel system that semi-automates the grading of students' SQL schema implementations. The system takes as input a machine-readable specification of the assessment requirements and a live instance of a submitted student schema, and verifies whether the schema conforms to the specification. Rather than attempt to parse and check the \texttt{CREATE} statements directly, the system verifies the structure of the schema by issuing queries against the schema's metadata (catalog) to look for expected values such as table names, column names, etc. It verifies integrity constraints by attempting to insert known legal and illegal values. The system currently does not check behavioral aspects. The results of the tests are compared against the machine-readable specification. This process is effectively one of unit testing the schema using the specification as a framework. We use the PHPUnit database unit testing framework to carry out this process, albeit in a somewhat unorthodox way (see \cref{sec-design}).

The remainder of the paper is structured as follows. In the next section we discuss related work, while \cref{sec-motivation} discusses the motivation and context for our approach. \Cref{sec-design} discusses the design of our system, and \cref{sec-evaluation} evaluates its impact. We discuss known issues and future work in \cref{sec-issues}, and conclude in \cref{sec-conclusion}.

\section{Related work}
\label{sec-literature}

There have been many prior efforts to build systems to support students who are learning SQL. Early examples such as RDBI \cite{Dietrich.S-1993a-An-educational} and esql \cite{Kearns.R-1997a-A-teaching} were essentially just query execution environments that students could use to practice writing SQL queries. RDBI provided relatively little feedback about the correctness of a query, whereas esql could visualize the intermediate tables generated by each step of a query, enabling students to better understand the steps in its execution.

Later systems became more responsive in terms of the feedback they provided to students. Systems like SQL-Tutor \cite{Mitrovic.A-1998a-Learning}, SQLator \cite{Sadiq.S-2004a-SQLator}, AsseSQL \cite{Prior.J-2004a-Backwash}, ActiveSQL\footnote{\url{https://db.grussell.org/}} \cite{Russell.G-2004a-Improving,Russell.G-2005a-Online}, SQLify \cite{Dekeyser.S-2007a-Computer}, and aSQLg \cite{Kleiner.C-2013a-Automated} all provided varying levels of syntactic and semantic feedback. Many of these took a more ``functional'' approach to checking SQL query code, i.e., verifying that the code written fulfilled the requirements of the problem, rather than focusing on the code itself. This was most commonly done by checking how much the result set of a student's query differed from the correct result set (e.g., SQLator's ``equivalence engine'' \cite{Sadiq.S-2004a-SQLator}). ActiveSQL could also detect ``hard-coded'' queries that produced the desired result, but which would fail if the data set changed \cite{Russell.G-2005a-Online}. The SQL Exploratorium \cite{Brusilovsky.P-2010a-Learning} took an interesting alternative approach, using parameterized query templates to generate the questions given to students.

Given the relatively static nature of an SQL schema, we feel this verification style of approach is the most appropriate way of constructing an automated grading system for SQL DDL. As an aside, the concept of verifying a schema against its specification sounds like it should be a useful application of formal methods \cite{Spivey.J-1989a-An-introduction}, but work with formal methods and databases has historically focused either on \emph{generating} a valid schema from a specification (e.g., \cite{Vatanawood.W-2004a-Formal,Lukovic.I-2003a-Proceedings,Choppella.V-2006a-Constructing}), or on verifying schema transformation and evolution \cite{Bench-Capon.T-1998a-Report}. A formal methods approach could certainly be used to generate the model answer or rubric for grading an assessment, but this is effectively the inverse of what is needed to grade a student-written schema.

Previous systems focused almost exclusively on SQL \emph{queries} using the \texttt{SELECT} statement (i.e., DML) rather than SQL \emph{schema definitions} (DDL). This focus is unsurprising given that \texttt{SELECT} is probably the most frequently used of any SQL statement, and also because it is relatively complex compared to most other SQL statements. Only a few of the systems we reviewed even mentioned schema definition. RDBI \cite{Dietrich.S-1993a-An-educational} supported a DDL, but only its own non-SQL language, while esql \cite{Kearns.R-1997a-A-teaching} simply passed anything that was not a \texttt{SELECT} statement through to the DBMS. \citeauthor{Gong.A-2015a-CS-121-Automation}'s ``CS 121 Automation Tool'' \cite{Gong.A-2015a-CS-121-Automation} focuses primarily on SQL DML statements, but appears to be extensible and could potentially be modified to support SQL DDL statements.

Many previous systems have implemented some form of automated or semi-automated grading, e.g., SQLator \cite{Sadiq.S-2004a-SQLator}, AsseSQL \cite{Prior.J-2004a-Backwash}, ActiveSQL \cite{Russell.G-2004a-Improving,Russell.G-2005a-Online}, SQLify \cite{Dekeyser.S-2007a-Computer}, aSQLg \cite{Kleiner.C-2013a-Automated}, \citeauthor{Gong.A-2015a-CS-121-Automation}'s ``CS 121 Automation Tool'' \cite{Gong.A-2015a-CS-121-Automation}, and XDa-TA\footnote{\url{http://www.cse.iitb.ac.in/infolab/xdata/}} \cite{Bhangdiya.A-2015a-XDa-TA,Chandra.B-2015a-Data,Chandra.B-2016a-Partial}. Some (e.g., SQLator, AsseSQL) provided only correct/incorrect responses, while others (e.g, ActiveSQL, SQLify, aSQLg, XDa-TA) had more sophisticated schemes for assigning partial credit, such as ActiveSQL's accuracy score, SQLify's eight-level scale, and XDa-TA's ability to generate data sets designed to catch common errors. In either case, these systems were often able to automatically mark a significant fraction (e.g., about a third for SQLator \cite{Sadiq.S-2004a-SQLator}) of submitted queries as correct without human intervention, thus reducing marking load for teachers.

To our knowledge there has been no work on automated grading of SQL DDL statements. While dealing with \texttt{CREATE} statements should be simpler than dealing with \texttt{SELECT} statements, the ability to at least semi-automate the grading of SQL schema definitions should reap rewards in terms of more consistent application of grading criteria, and faster turnaround time \cite{Douce.C-2005a-Automatic,Russell.G-2004a-Improving,Dekeyser.S-2007a-Computer,Prior.J-2004a-Backwash}.

Another branch of related work is systems that actively aid students in learning SQL, e.g., SQL-Tutor \cite{Mitrovic.A-1998a-Learning}, SQLator \cite{Sadiq.S-2004a-SQLator}, ActiveSQL \cite{Russell.G-2004a-Improving}, and aSQLg \cite{Kleiner.C-2013a-Automated}. SQL-Tutor is a typical example of this category. It was an intelligent teaching system that provided students with a guided discovery learning environment for SQL queries, and used constraint-based modeling \cite{Ohlsson.S-1992a-Constraint-based,Ohlsson.S-2016a-Constraint-based} to provide feedback to students. \citeauthor{Kenny.C-2005a-Automated} \cite{Kenny.C-2005a-Automated} described a similar SQL learning system that incorporated an assessment of a student's previous progress. This enabled a more personalized and adaptive approach to student learning, where feedback was tailored according to student progress.

There is relatively little prior work on unit testing of databases. Most authors working in this area have focused on testing database \emph{applications} rather than the database itself (e.g., \cite{Binnig.C-2008a-Multi-RQP,Chays.D-2008a-Query-based,Marcozzi.M-2012a-Test,Haller.K-2010a-Test}). \citeauthor{Ambler.S-2006a-Database} discusses how to test the functionality of a database \cite{Ambler.S-2006a-Database}, while \citeauthor{Farre.C-2008a-SVTe} describe how to test the ``correctness'' of a schema \cite{Farre.C-2008a-SVTe}, focusing mainly on consistency of constraints. Neither consider how to verify that a database schema conforms to the requirements of the original specification.

\section{Motivation}
\label{sec-motivation}

Since 1989, our department has offered some form of mandatory database concepts course, typically one semester during the second year\footnote{\anoncountry\ Bachelor's degrees normally comprise three years of study.} of study, building on a short introduction to basic data management concepts in the first year. Typical of courses of this nature, it covered core topics such as the relational model, relational algebra, data integrity, SQL DDL and DML, and a varying mixture of other database topics such as transactions, concurrency control, triggers, and security. Assessment of SQL skills was typically carried out using a mixture of assignments and tests.

From 2001 to 2003, we assessed students' SQL DDL skills with an in-lab practical examination. Students were given a fictional scenario specification, and had 100 minutes in which to modify a provided schema template with additional tables, constraints, etc. The test was generally easier to grade than the more ``realistic'' practical assignment that we had used in earlier years, as the scenario specification tended to be quite tightly specified and thus less open to (mis)interpretation. However, the test experience was quite stressful to students due to the limited timeframe and limited access to online references. We did not attempt to automate the grading of these tests.

The most common approach we have used to assess students' SQL DDL skills was a practical assignment, where students had 3--4 weeks in which to implement a database schema based on a fictional scenario specification. The scenario posed that the student was a database developer involved in a larger project, and that the specification was the output of the requirements analysis phase. An entity-relationship diagram (ERD) of a typical scenario that we used (known as the ``BDL'' scenario) is shown in \cref{fig-ERD}.
 
 
\begin{figure}[hb]
    \centering
    \includegraphics[width=0.85\columnwidth, keepaspectratio]{images/BDL_ERD.pdf} 
    \caption{ERD of the ``BDL'' scenario used to assess SQL DDL skills (Information Engineering notation).}
    \label{fig-ERD}
\end{figure}

Prior to 2001, the specifications for the scenarios used in this assignment were deliberately somewhat loosely defined and often contained areas that were under-specified or ambiguous. At the time the course also included some data modeling content, so this approach enabled students to explore the different ways that a conceptual model could be converted into an implementation. This of course led to variation across student submissions, due to differing interpretations of the under-specified elements. Automation of grading was problematic, especially when students chose different database structures, or different names for tables and columns, than what was ``expected''. We therefore made no significant attempt to automate grading under this approach.

By 2004 the data modeling content was being migrated into other courses, so we took the opportunity to tighten up the scenario specifications to reduce ambiguity. In 2013 we took the further step of ``freezing'' the specification, i.e., students were not permitted to make arbitrary changes to the specification without strong justification, and even then only if the changes did not impact how client programs interacted with the database. For example, if a column needed to be restricted to a discrete set of values, a student could---if they wished---choose to go beyond the specification and enforce this requirement using a separate lookup table rather than a \texttt{CHECK} constraint, as long as this did not change the originally specified structure of the table. This meant that students could still exercise some flexibility in their schema implementations where appropriate, and ensured that the assignment was not just a mechanical translation exercise.

The in-scenario rationale for freezing the specification was that other developers were independently using the same specification to code end-user applications. Any significant variation from the specification would therefore break those applications. This approach tested not only students' ability to write SQL DDL, but also their ability to correctly interpret and convert a natural language database specification into a corresponding SQL schema.

This approach seemed effective, but maintaining consistent grading standards across a large number of submissions (on the order of 50--70) was difficult, due to the often large number of distinct gradable elements implied by the specification. This required a complex and highly-detailed rubric to be constructed so that no element was missed, and the grading process consequently took a significant amount of time, especially when feedback to students was required. In 2012 a significant change to the structure of the course resulted in higher grading workloads and increased time pressure, prompting greater interest in the possibility of at least semi-automating the grading of this assignment. This was now more feasible than in earlier years due to the more tightly constrained nature of the project specification.

Another motivation for automated grading was that it can sometimes be difficult for database novices to know whether they are on the right track while implementing a specification. If a limited version of the grading tool was available to students before the submission deadline, it could be used to provide feedback on their progress.

We decided to define a minimum set of requirements for the assignment: students' SQL code should be syntactically correct, and it should include all tables and columns---with correct names and appropriate data types---as detailed in the specification. If a student satisfied these minimum requirements, they were guaranteed to score at least 50\%. They could check whether their schema met these requirements prior to the deadline by submitting their schema for checking through a web application. That way they (and we) could be more certain that at least the core structure of their schema was correct. After the deadline, teaching staff used a console application to check for correct implementation of other aspects of the schema, such as integrity constraints.

We trialled a prototype of the staff-facing application in 2012, and rolled out the student-facing application in 2013.

\section{System design}
\label{sec-design}

The architecture of our system is shown in \cref{fig-architecture}. Its core function is to check whether a student's schema conforms to the assignment specification, by automatically comparing their schema submission against a machine-readable version of the specification. This is essentially a unit testing approach, so we decided to build the system around an existing unit testing framework.

\begin{figure}[h]
    \sffamily
    \begin{tikzpicture}[every node/.style={draw, minimum height=7.5mm, inner sep=1em}]
        \node (console) {\shortstack{Console app \\ \footnotesize(staff mode)}};
        \coordinate[below=5mm of console.south] (console port);
        
        \node[anchor=north west, minimum width=6cm] (driver) at ($(console.south west) - (0,5mm)$) {Main driver};
        
        \node[anchor=south east] (web) at ($(driver.north east) + (0,5mm)$) {\shortstack{Web app \\ \footnotesize(student mode)}};
        \coordinate[below=5mm of web.south] (web port);
        
        \node[below=5mm of driver] (phpunit) {PHPUnit};
        
        \node[left=5mm of phpunit] (spec) {\shortstack{Assign. \\ spec.}};
        
        \node[right=5mm of phpunit] (reporting) {Reporting};
        \coordinate[above=5mm of reporting.north] (reporting port);
        
        \node[cylinder, shape border rotate=90, below=5mm of phpunit, aspect=0.1] (database) {Database};
        
        \path (database.before top) -- (database.after top) coordinate[midway] (dbtop);
        
        \node[right=5mm of database] (schema) {\shortstack{Student's \\ schema}};
        
        \graph { [edges={draw, arrows={{Stealth}-{Stealth}}}]
            {(console), (web)} -> {(console port), (web port)},
            (driver) -> (phpunit),
            (phpunit) -> (dbtop),
        };
        
        \graph { [edges={draw, arrows={-{Stealth}}}]
            (reporting) -> (reporting port),
            (spec) -> (phpunit),
            (schema) -> (database),
        };
    \end{tikzpicture}
    \caption{System architecture.}
    \label{fig-architecture}
\end{figure}

There are surprisingly few frameworks for performing unit tests that interact with a database, probably due to the complexities involved. In conventional application unit testing it is relatively simple to create mocked interfaces for testing purposes. With a database, however, we need to create tables, populate them with appropriate test data, verify the state of the database after each test has run, and clean up the database for each new test \cite{Bergmann.S-2017a-PHPUnit}. Cleaning up is particularly crucial, as the order of tests is often not deterministic, i.e., the tests are not guaranteed to run in exactly the same order each time. Tests that change the state of the database may therefore affect the results of later tests in unpredictable ways.

We are aware of only four unit testing frameworks that provide specific support for database unit tests: DbUnit for Java,\footnote{http://dbunit.sourceforge.net/} DbUnit.NET,\footnote{http://dbunit-net.sourceforge.net/} Test::DBUnit for Perl,\footnote{http://search.cpan.org/~adrianwit/Test-DBUnit-0.20/lib/Test/DBUnit.pm} and PHPUnit.\footnote{https://phpunit.de/} We chose to implement the system in PHP, as it enabled us to quickly prototype the system and simplified development of the student-facing web application. A similar approach could be taken with any of the other frameworks, however.

Our database teaching was mainly based around Oracle, but the system could be adapted relatively easily for use with any DBMS supported by PHP's PDO extension. This would require implementing an additional layer to abstract the internal details of how each DBMS structures its catalog tables.

The core of the system is the \textsf{Main driver} component (see \cref{fig-architecture}). This can execute in either \emph{student mode}, which runs only a subset of the available tests, or in \emph{staff mode}, which runs all available tests (the nature of these tests will be discussed in more detail shortly). The mode is determined by the client application, as shown in \cref{fig-architecture}. Currently student mode is accessed through a web application, while staff mode is accessed through a console application, but each could easily be implemented in a variety of different ways. The main driver uses the \textsf{Reporting} module to generate test output in either HTML or plain text, as appropriate.

The assignment specification (\textsf{Assign.\ spec.} in \cref{fig-architecture}) is encoded as a collection of subclasses of PHPUnit's \texttt{TestCase} class. Each subclass specifies the properties of one database table. \Cref{fig-test-class} shows a fragment of the class corresponding to the \textsf{Product} table in \cref{fig-ERD}. The methods of this class return various properties of the table:

\begin{description}
    \item[\texttt{getTableName()}] returns the expected name of the table.
    \item[\texttt{getColumnList()}] returns an array of column specifications, keyed by expected column name. Each column specification includes a generic data type (text, number, date, or binary), a list of corresponding acceptable SQL data types (e.g., \texttt{VARCHAR}, \texttt{DECIMAL}), whether the column permits nulls, and a known legal value for general testing. Where applicable, it may include minimum and maximum column lengths (precision for numeric types), and the number of decimal places (scale). Underflow and overflow values, and lists of known legal and illegal values can also be specified to test the boundary conditions of integrity constraints.
    \item[\texttt{getPKColumnList()}] returns an array of column names that comprise the primary key of the table.
    \item[\texttt{getFKColumnList()}] returns an array of foreign key specifications (where applicable), keyed by the name of the referenced table. Each specification contains an array of column names that comprise that foreign key. 
\end{description}

\begin{figure}
    \begin{lstlisting}
public function getTableName() {
  return 'PRODUCT';
}

public function getColumnList() {
  return array(
    'PRODUCT_CODE' => array(
      'generic_type' => 'NUMBER',
      'sql_type' => array('NUMBER', 'INTEGER'),
      'min_length' => 8,
      'max_length' => 8,
      'decimals' => 0,
      'test_value' => 87654321,
      'nullable' => false),
    'DESCRIPTION' => array( ... ),
    'STOCK_COUNT' => array(
      'generic_type' => 'NUMBER',
      'sql_type' => array('NUMBER', 'INTEGER'),
      'min_length' => 5,
      'max_length' => 6,
      'decimals' => 0,
      'underflow' => -1,
      'overflow' => 100000,
      'legal_values' => array(0, 99999),
      'test_value' => 456,
      'nullable' => false),
    'RESTOCK_LEVEL' => array( ... ),
    'MINIMUM_LEVEL' => array( ... ),
    'LIST_PRICE' => array(
      'generic_type' => 'NUMBER',
      'sql_type' => array('NUMBER', 'INTEGER'),
      'min_length' => 7,
      'max_length' => 8,
      'decimals' => 2,
      'underflow' => -0.01,
      'overflow' => 100000.00,
      'legal_values' => array(0, 99999.99),
      'test_value' => 123.99,
      'nullable' =>  false),
    'ASSEMBLY_MANUAL' => array(
      'generic_type' => 'BINARY',
      'sql_type' => array('BLOB'),
      'test_value'    =>  "NULL",
      'nullable' => true),
    'ASSEMBLY_PROGRAM' => array( ... )
  );
}

public function getPKColumnList() {
  return array( 'PRODUCT_CODE' );
}

public function getFKColumnList() {
  return array();   // no FKs in this table
}   \end{lstlisting}
    \caption{Fragment of the \textsf{Product} table specification.}
    \label{fig-test-class}
\end{figure}

Each table specification also requires the definition of two distinct sets of tests that will be run on the database. The first set verifies the structural elements of the table (columns, data types, etc.), thus verifying that a schema meets the minimum requirements of the assignment. This is done by issuing queries against the metadata (catalog) of the schema for elements like tables, columns, and data types. When in student mode, only this set of tests is run. An empty data fixture (specified using a separate XML document) is also required to support this set of tests.

The second set of tests verifies the integrity elements of the table, i.e., its constraints. The only integrity constraints that are explicitly tested are primary and foreign keys, which are again verified using queries against the schema's metadata, and nullability (\texttt{NOT NULL}), which is tested by attempting to insert a null. The remaining constraints are tested by attempting to insert lists of known legal and illegal values, an approach consistent with normal unit testing practice. When in staff mode, both this set of tests and the set of structural tests are run. A known-legal data fixture is also required to support this set of tests.

\begin{figure}
    \includegraphics[width=0.95\columnwidth,keepaspectratio]{images/web_output.png}
    \caption{Example of student mode output (web app) for the \textsf{Staff} table.}
    \label{fig-student-output}
\end{figure}

The way that the system runs the tests is somewhat unusual, in two ways. First, database unit testing frameworks are generally designed around the concept of testing database-backed \emph{applications}, rather than testing the database itself. Second, in typical unit testing, the tests are essentially standalone code units that are automatically executed in an order determined by the unit testing framework. The framework resolves dependencies among tests and handles collation of test results internally, reporting only success, failure, or ``error'' back to the client. In fact, most unit testing frameworks appear to be designed to be called from a specific tool (e.g., the \texttt{phpunit} command line tool), rather than embedded as a library into other programs.

Our system effectively inverts (or perhaps subverts) this approach. The main driver explicitly creates test suites itself and executes them directly, listening for the results of each test and collating them. This is because we need to be able to control the order in which tests are executed. If the structural tests fail, there is little point in running the integrity tests, as they will only generate a stream of errors. Similarly, if a column is missing, there is no point in running the data type and length tests. (Note that while PHPUnit supports grouping of tests, dependencies can only be defined between individual tests, not between groups.)

It is quite feasible to add table properties and tests beyond those already mentioned, as the specification is encoded as ordinary PHP code and is therefore constrained only by what is possible in PHP. All a teacher needs to do is add custom properties to the table specification, then add tests that use those custom properties. Custom tests are easily registered with the appropriate test set (structure or integrity) using PHPUnit's \texttt{@group} annotation.

Students can check their schema (\textsf{Student's schema} in \cref{fig-architecture}) by loading it into their personal database account (i.e., creating the tables), then entering their database login credentials into a web application (\textsf{Web app} in \cref{fig-architecture}). This calls the main driver in student mode and accesses the student's schema directly using the provided credentials. Only the structural set of tests is run, and the output is displayed in the web browser. \Cref{fig-student-output} shows an example of the output produced by student mode for the \textsf{Staff} table in \cref{fig-ERD}.

A teacher can check further aspects of a student's schema using the \textsf{Console app} shown in \cref{fig-architecture}. To ensure a clean testing environment, the teacher does not connect directly to the student's database account, as the student may (for example) continue to use it for other coursework. Instead, the teacher loads the student's submitted schema code under a separate account used only for grading purposes, and erases the schema before moving to the next submission. The login credentials for the grading account are specified in the console application's configuration file.

Assuming that there are no syntax errors in the student's code,\footnote{If there are errors, then the student has clearly not met the minimum requirements!} the teacher then runs the console application, which calls the main driver in staff mode. The main driver connects to the pre-configured grading account, runs all the available tests, and displays the output in the terminal window. \Cref{fig-staff-output} shows an example of the output produced by staff mode for the \textsf{Product} table in \cref{fig-ERD}.

% ANSI terminal colours for Terminal.app; see https://en.wikipedia.org/wiki/ANSI_escape_code#Colors
% grey 203, 204, 205
% green 37 188 36
% red 194, 54, 33
\definecolor{test grey}{rgb}{0.796,0.800,0.804}
\definecolor{test green}{rgb}{0.145,0.737,0.141}
\definecolor{test red}{rgb}{0.761,0.212,0.129}

\tcbset{boxsep=0pt, boxrule=0pt, arc=0pt, left=0pt, right=0pt, top=0.5pt, bottom=0.5pt}

\newlength{\dothskip}
\setlength{\dothskip}{0.72cm}
\newlength{\dotvskip}
\setlength{\dotvskip}{-1.25ex}
\newlength{\codeskip}
\setlength{\codeskip}{-0.5ex}

\begin{figure}
    \ttfamily\scriptsize
    \begin{tabbing}
        0123\=\kill
        \tcbox[colback=test grey]{NOTE: Checking structure of table Product.} \\[\codeskip]
        TEST: [[ Product ]]  \\
        \>  \textcolor{test green}{+ OK} \\
        \tcbox[colback=test green]{+++ PASSED: Table Product exists.} \\[\codeskip]
        TEST: [[ Product.Product\_code ]]  \\
        \>  \textcolor{test green}{+ OK} \\[\dotvskip]
        \hspace*{\dothskip}\vdots \\
        \tcbox[colback=test green]{+++ PASSED: Table Product contains all the expected columns.} \\[\codeskip]
        TEST: [[ Product.Product\_code: data type is NUMBER | INTEGER ]]  \\
        \>  \textcolor{test green}{+ OK} \\[\dotvskip]
        \hspace*{\dothskip}\vdots \\
        \tcbox[colback=test green]{+++ PASSED: All columns of table Product have data types compatible with the}\\[\codeskip]
        \tcbox[colback=test green]{specification.} \\[\codeskip]
        TEST: [[ Product.Product\_code precision and scale = 8 (with scale 0) ]]  \\
        \>  \textcolor{test green}{+ OK} \\[\dotvskip]
        \hspace*{\dothskip}\vdots \\
        \tcbox[colback=test green]{+++ PASSED: All columns of table Product have lengths compatible with the} \\[\codeskip]
        \tcbox[colback=test green]{specification.} \\[\codeskip]
        TEST: [[ Product PK ]]  \\
        \>  \textcolor{test green}{+ OK} \\
        \tcbox[colback=test green]{+++ PASSED: Primary key of table Product exists.} \\[\codeskip]
        TEST: [[ Product PK: Product\_code ]]  \\
        \>  \textcolor{test green}{+ OK} \\
        \tcbox[colback=test green]{+++ PASSED: Primary key of table Product includes (only) the expected} \\[\codeskip]
        \tcbox[colback=test green]{columns.} \\[\dotvskip]
        \hspace*{\dothskip}\vdots \\
        \tcbox[colback=test grey]{NOTE: Testing constraints of table Product.} \\[\codeskip]
        TEST: [[ Product.Stock\_count accepts ``0'' ]]  \\
        \>  \textcolor{test green}{+ OK} \\
        TEST: [[ Product.Stock\_count accepts ``99999'' ]]  \\
        \>  \textcolor{test green}{+ OK} \\
        TEST: [[ Product.Restock\_level accepts ``0'' ]]  \\
        \>  \textcolor{test red}{- FAILED! Column Product.Restock\_level won't accept legal value 0.} \\
        \textcolor{test red}{Failed asserting that false is true.} \\
        TEST: [[ Product.Restock\_level accepts ``99999'' ]]  \\
        \>  \textcolor{test green}{+ OK} \\
        TEST: [[ Product.Minimum\_level accepts ``0'' ]]  \\
        \>  \textcolor{test red}{- FAILED! Column Product.Minimum\_level won't accept legal value 0.} \\
        \textcolor{test red}{Failed asserting that false is true.} \\
        TEST: [[ Product.Minimum\_level accepts ``653'' ]]  \\
        \>  \textcolor{test green}{+ OK} \\[\dotvskip]
        \hspace*{\dothskip}\vdots \\
        \tcbox[colback=test red, coltext=test grey]{--- FAILED: 2 of 8 legal values tested were rejected by a CHECK constraint.}
    \end{tabbing}
    \vskip-1ex
    \caption{Example of staff mode output (console app) for the \textsf{Product} table.}
    \label{fig-staff-output}
\end{figure}

\section{Evaluation}
\label{sec-evaluation}

Unfortunately, the system was conceived as a practical solution to a perceived teaching issue, rather than as a research project with formal evaluation in mind. We therefore did not carry out any user evaluations with students that used the system.

From the perspective of the teaching staff, we found that the total amount of time taken to grade the relevant assignment was reduced only a little, as the system only semi-automated the grading process, and we still needed to convert the system's output into corresponding grades and meaningful feedback (see \cref{sec-issues} for further discussion of this). There were also a surprising number of submissions that did not meet the minimum requirements and thus had to be manually graded. On a more positive note, the system did automatically ensure that all gradable elements were checked, which improved consistency, and made the subjective experience of grading much less painful.

However, we were able to analyze whether use of the system had any impact on student performance in the relevant assignment, as we had extensive historical data of student grades. We collated data for the period 2009--2016 (there are no data for 2017 because the course has been discontinued; see \cref{sec-issues}), which encompassed several different permutations of scenario and available system modes, as summarized in \cref{tab-data}. The assignment counted for 15\% of a student's total grade in 2009 and 2010, and 10\% in subsequent years. The grade distributions for the assignment in each year are shown in \cref{fig-distributions}.

\begin{table}
    \caption{Historical characteristics of the database implementation assignment, 2009--2016.}
    \label{tab-data}
    \begin{tabular}{rrrrll}
        \toprule
                &   Class   &   Median                          &   Mean                            &               &   Modes   \\
        Year    &   size    &   GPA\textsuperscript{\emph{a}}   &   (\%)\textsuperscript{\emph{b}}  &   Scenario    &   used    \\
        \midrule
        2009    &   46  &   --\textsuperscript{\emph{c}} &   77.5    &   ``postgrad''        &   --  \\
        2010    &   68  &   3.4 &   73.4    &   ``student records'' &   --  \\
        2011    &   64  &   3.9 &   71.8    &   ``used cars''       &   --  \\
        \midrule
        2012    &   75  &   3.4 &   69.2    &   ``BDL''             &   staff   \\
        2013    &   77  &   3.2 &   84.3    &   ``student records'' &   both \\
        \midrule
        2014    &   49  &   3.4 &   77.6    &   ``used cars''       &   both \\
        2015    &   71  &   3.0 &   69.2    &   ``used cars''       &   neither  \\
        2016    &   75  &   3.5 &   71.0    &   ``BDL''             &   staff   \\
        \bottomrule
        \multicolumn{6}{l}{\footnotesize \textsuperscript{\emph{a}} On a 9-point scale where C-- = 1, A+ = 9. Value is for the specified year only.}    \\
        \multicolumn{6}{l}{\footnotesize \textsuperscript{\emph{b}} For students who submitted the assignment.}    \\
        \multicolumn{6}{l}{\footnotesize \textsuperscript{\emph{c}} Complete GPA data for 2009 were not available.}    \\
    \end{tabular}
\end{table}

\begin{figure}
    \includegraphics[width=\columnwidth,keepaspectratio]{grades_all_years.pdf}
    \caption{Comparison of grade distributions for the database implementation assignment, 2009--2016.}
    \label{fig-distributions}
\end{figure}

The horizontal rule in \cref{tab-data} between 2011 and 2012 marks a significant reorganization of the course's curriculum, and also a switch from first to second semester.\footnote{First semester at the \anoninstitution\ runs from March to June, second semester from July to October.} (2012 was also the year we deployed the first prototype of staff mode.) The horizontal rule between 2013 and 2014 marks a shift from second semester back to first semester. The system was not used at all in 2015 due to different staff teaching the course, and student mode was not made available in 2016 due to technical issues delaying its deployment beyond the point where it would be useful. These differences have provided us with a natural experiment with some interesting points for comparison.

The mean grade for the assignment drifted slowly downwards from 2009 to 2012. This reversed dramatically in 2013, however, the year we first deployed student mode. As can be seen in \cref{fig-distributions}, the grades are not normally distributed (they typically have negative skew), so we used a Mann-Whitney \emph{U} test to determine whether the increase in mean from 2012 to 2013 was statistically significant. This showed that the increase was highly significant ($p \approx 10^{-9}$). The 2013 mean was also significantly higher than both 2010 ($p \approx 0.0002$) and 2011 ($p \approx 10^{-6}$), but not significantly higher than 2009. The mean decreased significantly again in 2014 ($p \approx 0.0012$), the second year that the system was used, and even more dramatically in 2015 ($p \approx 0.0005$), when the system was not used at all. The increase from 2015 to 2016 was not significant.

Even more interesting, if we compare the performance between the years that student mode was not available (2009--2012 and 2015--2016, mean 71.6\%) and the years it was (2013--2014, mean 81.7\%), there is again a highly statistically significant increase in the mean ($p \approx 10^{-8}$). This difference is strikingly illustrated in \cref{fig-distribution-comparison}, which compares the grade distributions for both periods. During the period that student mode was available, the lowest grade awarded was 46\%, contrasted with a much longer tail of lower grades when student mode was not available. This strongly suggests that the introduction of student mode had a positive impact on students' ability to complete the assignment more effectively.

\begin{figure*}
    \centering
    \subfloat[]{\includegraphics[trim=0 0 0 72,scale=0.6]{grades_without_student_mode.pdf}} \qquad
    \subfloat[]{\includegraphics[trim=0 0 0 72,scale=0.6]{grades_with_student_mode.pdf}}
    \caption{Comparison of grade distributions (a) without student mode, (b) with student mode.}
    \label{fig-distribution-comparison}
\end{figure*}

There are some potential confounding factors to consider, however. First, not only was 2013 the first year that student mode was available, it was also the first year that the assignment specification was ``frozen'' (as discussed in \cref{sec-motivation}). It could be argued that this improved grades due to students having less flexibility, and thus less opportunity for misinterpretation, than in previous years. However, the assignment specification was also ``frozen'' in all subsequent years, and there is considerable variation in the grades achieved over this period, especially in 2015. It therefore seems unlikely that this was a factor in student performance.

Second, the observed effect may be due to the imposition of minimum requirements from 2013 onwards. Since any schema that met the minimum requirements would score at least 50\%, we could reasonably expect this to have a positive effect on the mean. If we compare with the period where minimum requirements were not enforced, we do find a significant increase in the mean, from 72.4\% to 75.6\% ($p \approx 0.047$). We argue, however, that this effect would be minimal in the absence of a convenient mechanism for students to check conformance with the minimum requirements. Indeed, for 2013--2014 when student mode was available, the mean (81.7\%) was significantly higher ($p \approx 10^{-9}$) than for 2015--2016 (70.2\%) when student mode was not available. This is further supported by no significant difference between the means for 2009--2012 and 2015--2016.

Third, the switch to second semester in 2012--2013 could have negatively impacted students' performance by increasing the length of time between their exposure to basic data management concepts in first year, and their entry into the second year database course. In effect, they had longer to forget relevant material they learned in first year. If so, we could reasonably expect the grades in second semester offerings of the course to be lower. However, mean grades for second semester offerings (76.9\%) were in fact significantly \emph{higher} ($p \approx 0.015$) than those for first semester offerings (72.9\%). This should not be surprising, given that 2013 (second semester) had the highest grades overall. This effectively rules out semester changes as a factor.

Fourth, perhaps the years with higher grades used less complex---and therefore easier---scenarios. To test this, we computed the following database complexity metrics for each of the four scenarios used: database complexity index (DCI) \cite{Sinha.B-2014a-Estimation}; referential degree (RD), depth of referential tree (DRT), and number of attributes (NA) \cite{Calero.C-2001a-Database,Piattini.M-2001a-Table}; database complexity (DC) \cite{Pavlic.M-2008a-Database}; and ``Software Metric Analyzer for Relational Database Systems'' (SMARtS) \cite{Jamil.B-2010a-SMARtS}. The results are shown in \cref{tab-metrics}. All but the DRT metric clearly showed the ``BDL'', ``used cars'', and ``student records'' scenarios to be of comparable complexity, while the ``postgrad'' scenario was noticeably less complex. It therefore seems unlikely that scenario complexity is a factor in student performance. It is also interesting to note that the ``used cars'' scenario was used in both 2014 and 2015, and yet the 2015 grades were significantly \emph{lower} than those for 2014. The only clear difference here is that our system was not used in 2015.

\begin{table}
    \caption{Database complexity metrics for the scenarios used over the period 2009--2016.}
    \label{tab-metrics}
    \begin{tabular}{lrrrrrr}
        \toprule
            Scenario            &   DCI &   RD  &   NA  &   DRT &   DC  &   SMARtS  \\
            \midrule
            ``postgrad''        &   277 &   9   &   32  &   7   &   37  &   28.75   \\
            ``student records'' &   367 &   12  &   43  &   9   &   61  &   38.25   \\
            ``BDL''             &   370 &   11  &   50  &   8   &   59  &   40.75   \\
            ``used cars''       &   380 &   13  &   46  &   7   &   53  &   36.50    \\
        \bottomrule
    \end{tabular}
\end{table}

Fifth, class size could be a factor. We might plausibly expect a smaller class to have a more collegial atmosphere that promotes better learning. However, if we look at the sizes of the classes in \cref{tab-data}, we can see no discernible pattern between class size and performance. Indeed, both the best (2013) and worst (2012, 2015) performances came from classes of similar size (75, 77, and 71, respectively).

Sixth, it could be that better performance occurred in years where the students were just more capable in general. We obtained annual GPA data for the students enrolled in each year, and computed the median as an indication of the general capability of the class. Looking at \cref{tab-data}, we can immediately see that the year with the best results (2013) was also the year with the second-lowest median GPA (3.2). Contrast this with the poorer performance in 2012, where the median GPA was 3.4. In both years that student mode was available, median GPA was lower than or the same as in most other years, yet performance was better than in years with higher median GPA. This argues against the idea that we simply had a class full of very capable students in the better performing years.

Seventh, the timing of the assignment varied across the period, occurring either early (2012--2014), halfway (2009--2010), or late (2011, 2015--2016) in the semester, depending on the ordering of course content. The mean grade for early timing (77.0\%) was significantly higher than for both halfway (72.2\%, $p \approx 0.019$) and late (71.8\%, $p \approx 0.013$), while there was no significant difference between the halfway and late means. This suggests that scheduling the assignment early in the semester may have a positive effect on grades, and the period of early timing does overlap the period that student mode was available. However, as noted earlier there was a highly significant difference in mean between 2012 and 2013. There is also a smaller, but still significant difference in mean between 2012 and 2014 ($p \approx 0.0015$). We therefore conclude that while scheduling the assignment early in the semester may have had some positive effect on student performance, it does not explain all of the positive effect seen in 2013 and 2014.

Finally, perhaps the different weightings of the assignment (15\% in 2009--2010 vs.\ 10\% in 2011--2016) affected student motivation. It could be argued that the higher weighting in 2009--2010 provided a greater incentive for students to work more, as the potential reward was greater. If so, we should expect better performance in 2009--2010. Indeed, we do find this: the mean for 2009--2010 is 75.1\%, while that for 2011--2016 is 73.9\%, a statistically significant decrease ($p \approx 0.034$). However, since this change occurred well before our system was even conceived of, let alone implemented, it cannot be a factor in the improved performance seen in 2013 and 2014.

The sum of this evidence provides a strong argument in favor of student mode being the main factor in improving student grade performance in the database implementation assignment during 2013 and 2014. The most plausible explanation for this effect is that the direct feedback provided by the student mode web application enabled students to more effectively correct the structural aspects of their schema to meet the minimum requirements, giving them more time to work on the integrity aspects of their schema, and thus gain a higher grade.

\section{Known issues \& future work}
\label{sec-issues}

While the use of student mode in 2013 and 2014 appears to have had a beneficial effect on student performance in the database implementation assignment, there are some outstanding issues with the system that still need to be addressed.

Ideally, our system would automatically assign appropriate marks and generate meaningful feedback, and write both of these directly into a student management database. Currently, however, it only semi-automates the grading process, and it is still up to the teacher to interpret the test results, assign appropriate marks, and write feedback.

Automatically generating appropriate marks is not particularly difficult, and we have already implemented the core functionality required to assign mark penalties to different kinds of error. It would be a relatively simple extension to calculate marks based on these penalties and write them directly into a student management database.

Automatically generating meaningful feedback is problematic, however, due to the way that PHPUnit reports test failures. There is currently no way to control or suppress the messages generated by PHPUnit test assertions, most likely because PHPUnit is not designed to be used as an embedded library (as discussed in \cref{sec-design}). Individual tests can specify a meaningful message to be displayed, but PHPUnit will still generate somewhat obscure messages like ``Failed asserting that 0 matches expected 1'', which can be confusing for students.

One particularly tricky example we encountered was if a student failed to specify a precision for a numeric column, e.g., they declared a column as just \texttt{NUMERIC} rather than \texttt{NUMERIC(5)}. Most DBMSs will in this situation assign the maximum precision (e.g., 38 significant digits for Oracle). The student then sees a message ``Failed asserting that 38 matches expected 5'', and has no idea where the ``38'' came from.

This issue of unsuppressable assertion messages significantly constrained our ability to provide fully automatic, meaningful feedback to students. It may be that the only way to address this issue is to change the way PHPUnit works internally.

A second issue is that any schema that fails to meet the minimum requirements will result in a large number of errors due to missing tables.  This could be alleviated by more careful exception handling in the main driver.

Another issue is how students made use of the student mode web application. In 2013, students tended to misuse it as a ``schema compiler'': they would submit their schema, fix only the first issue reported (rather than attempting to fix as many as possible), immediately re-submit, and so on. We had not expected so high a request rate, so the system was only single-threaded and thus not designed to handle concurrent requests. Consequently the web application suffered from waits and timeouts. An initial workaround was to log all student access and warn those who were abusing the system. We later throttled the system by enforcing a short delay between attempts by the same student. Things improved in 2014 with careful management of student expectations, but longer term, the system should be redesigned to handle concurrent loads.

The system currently does not support behavioral aspects such as sequences, triggers, or stored procedures, as these are usually DBMS-specific in nature. It would not be difficult to add support for these, however, which could be included in the DBMS abstraction layer proposed in \cref{sec-design}.

Looking forward, it would be interesting to extend our system to help facilitate the teaching of SQL DDL, rather than just focusing on assignment grading. This has been done previously with several systems for SQL DML, as discussed in \cref{sec-literature} (e.g., \cite{Kenny.C-2005a-Automated,Kleiner.C-2013a-Automated,Mitrovic.A-1998a-Learning,Russell.G-2004a-Improving,Sadiq.S-2004a-SQLator}).

From 2017, we no longer have a dedicated second year database course. The main introduction to database concepts and SQL now occurs as part of our new (in 2017) first year ``Foundations of Information Systems'' course. This course attracted about 160 students in first semester, and about 100 students in second semester. Classes of this size further strengthen the argument for automated grading, and we are currently exploring whether our system could be used in this course in 2018. If this is feasible,  we intend to carry out user evaluations with the students.

\section{Conclusion}
\label{sec-conclusion}

In this paper we have described a novel system that semi-automates the process of grading SQL data definition code. To our knowledge, all prior work in this area has focused only on data manipulation statements such as \texttt{SELECT}, so extending this to data definition code is a key contribution of our system. The system also takes a novel approach to checking an SQL schema: rather than attempting to parse the SQL DDL code directly, it instead verifies that the schema conforms to a machine-readable version of the original specification. Finally, our system uses a unit testing approach to carry out the verification, which has not been used before in this context.

While no user evaluations of the system were carried out, analysis of student performance in a relevant database assessment shows a highly statistically significant increase in mean grade in the two years that the system was available for use by students. While we cannot be absolutely certain, analysis of all the factors involved strongly suggests that use of our system by students had a positive impact on their performance in this assessment. This is an encouraging result, which we will explore further in future work.

\bibliographystyle{ACM-Reference-Format}
\bibliography{Koli_2017_Stanger}

\end{document}

Ignore Space Show notes View Koli_2017/acmart.cls 0 → 100644
%% %% This is file `acmart.cls', %% generated with the docstrip utility. %% %% The original source files were: %% %% acmart.dtx (with options: `class') %% %% IMPORTANT NOTICE: %% %% For the copyright see the source file. %% %% Any modified versions of this file must be renamed %% with new filenames distinct from acmart.cls. %% %% For distribution of the original source see the terms %% for copying and modification in the file acmart.dtx. %% %% This generated file may be distributed as long as the %% original source files, as listed above, are part of the %% same distribution. (The sources need not necessarily be %% in the same archive or directory.) %% \CharacterTable %% {Upper-case \A\B\C\D\E\F\G\H\I\J\K\L\M\N\O\P\Q\R\S\T\U\V\W\X\Y\Z %% Lower-case \a\b\c\d\e\f\g\h\i\j\k\l\m\n\o\p\q\r\s\t\u\v\w\x\y\z %% Digits \0\1\2\3\4\5\6\7\8\9 %% Exclamation \! Double quote \" Hash (number) \# %% Dollar \$ Percent \% Ampersand \& %% Acute accent \' Left paren \( Right paren \) %% Asterisk \* Plus \+ Comma \, %% Minus \- Point \. Solidus \/ %% Colon \: Semicolon \; Less than \< %% Equals \= Greater than \> Question mark \? %% Commercial at \@ Left bracket \[ Backslash \\ %% Right bracket \] Circumflex \^ Underscore \_ %% Grave accent \` Left brace \{ Vertical bar \\| %% Right brace \} Tilde \~} \NeedsTeXFormat{LaTeX2e} \ProvidesClass{acmart} [2017/07/09 v1.43 Typesetting articles for Association of Computing Machinery] \def\@classname{acmart} \InputIfFileExists{acmart-preload-hook.tex}{% \ClassWarning{\@classname}{% I am loading acmart-preload-hook.tex. You are fully responsible for any problems from now on.}}{} \RequirePackage{xkeyval} \RequirePackage{xstring} \define@choicekey+{acmart.cls}{format}[\ACM@format\ACM@format@nr]{% manuscript, acmsmall, acmlarge, acmtog, sigconf, siggraph, sigplan, sigchi, sigchi-a}[manuscript]{}{% \ClassError{\@classname}{The option format must be manuscript, acmsmall, acmlarge, acmtog, sigconf, siggraph, sigplan, sigchi or sigchi-a}} \def\@DeclareACMFormat#1{\DeclareOptionX{#1}{\setkeys{acmart.cls}{format=#1}}} \@DeclareACMFormat{manuscript} \@DeclareACMFormat{acmsmall} \@DeclareACMFormat{acmlarge} \@DeclareACMFormat{acmtog} \@DeclareACMFormat{sigconf} \@DeclareACMFormat{siggraph} \@DeclareACMFormat{sigplan} \@DeclareACMFormat{sigchi} \@DeclareACMFormat{sigchi-a} \ExecuteOptionsX{format} \define@boolkey+{acmart.cls}[@ACM@]{screen}[true]{% \if@ACM@screen \PackageInfo{\@classname}{Using screen mode}% \else \PackageInfo{\@classname}{Not using screen mode}% \fi}{\PackageError{\@classname}{Option screen can be either true or false}} \ExecuteOptionsX{screen=false} \define@boolkey+{acmart.cls}[@ACM@]{review}[true]{% \if@ACM@review \PackageInfo{\@classname}{Using review mode}% \else \PackageInfo{\@classname}{Not using review mode}% \fi}{\PackageError{\@classname}{Option review can be either true or false}} \ExecuteOptionsX{review=false} \define@boolkey+{acmart.cls}[@ACM@]{authorversion}[true]{% \if@ACM@authorversion \PackageInfo{\@classname}{Using authorversion mode}% \else \PackageInfo{\@classname}{Not using authorversion mode}% \fi}{\PackageError{\@classname}{Option authorversion can be either true or false}} \ExecuteOptionsX{authorversion=false} \define@boolkey+{acmart.cls}[@ACM@]{natbib}[true]{% \if@ACM@natbib \PackageInfo{\@classname}{Explicitly selecting natbib mode}% \else \PackageInfo{\@classname}{Explicitly deselecting natbib mode}% \fi}{\PackageError{\@classname}{Option natbib can be either true or false}} \ExecuteOptionsX{natbib=true} \define@boolkey+{acmart.cls}[@ACM@]{anonymous}[true]{% \if@ACM@anonymous \PackageInfo{\@classname}{Using anonymous mode}% \else \PackageInfo{\@classname}{Not using anonymous mode}% \fi}{\PackageError{\@classname}{Option anonymous can be either true or false}} \ExecuteOptionsX{anonymous=false} \define@boolkey+{acmart.cls}[@ACM@]{timestamp}[true]{% \if@ACM@timestamp \PackageInfo{\@classname}{Using timestamp mode}% \else \PackageInfo{\@classname}{Not using timestamp mode}% \fi}{\PackageError{\@classname}{Option timestamp can be either true or false}} \ExecuteOptionsX{timestamp=false} \define@boolkey+{acmart.cls}[@ACM@]{authordraft}[true]{% \if@ACM@authordraft \PackageInfo{\@classname}{Using authordraft mode}% \@ACM@timestamptrue \@ACM@reviewtrue \else \PackageInfo{\@classname}{Not using authordraft mode}% \fi}{\PackageError{\@classname}{Option authordraft can be either true or false}} \ExecuteOptionsX{authordraft=false} \def\ACM@fontsize{} \DeclareOptionX{9pt}{\edef\ACM@fontsize{\CurrentOption}} \DeclareOptionX{10pt}{\edef\ACM@fontsize{\CurrentOption}} \DeclareOptionX{11pt}{\edef\ACM@fontsize{\CurrentOption}} \DeclareOptionX{12pt}{\edef\ACM@fontsize{\CurrentOption}} \DeclareOptionX{draft}{\PassOptionsToClass{\CurrentOption}{amsart}} \DeclareOptionX{}{\PassOptionsToClass{\CurrentOption}{amsart}} \ProcessOptionsX \ClassInfo{\@classname}{Using format \ACM@format, number \ACM@format@nr} \newif\if@ACM@manuscript \newif\if@ACM@journal \newif\if@ACM@sigchiamode \ifnum\ACM@format@nr=5\relax % siggraph \ClassWarning{\@classname}{The format `siggraph' is now obsolete. I am switching to sigconf.} \setkeys{acmart.cls}{format=sigconf} \fi \ifnum\ACM@format@nr=0\relax \@ACM@manuscripttrue \else \@ACM@manuscriptfalse \fi \@ACM@sigchiamodefalse \ifcase\ACM@format@nr \relax % manuscript \@ACM@journaltrue \or % acmsmall \@ACM@journaltrue \or % acmlarge \@ACM@journaltrue \or % acmtog \@ACM@journaltrue \or % sigconf \@ACM@journalfalse \or % siggraph \@ACM@journalfalse \or % sigplan \@ACM@journalfalse \or % sigchi \@ACM@journalfalse \or % sigchi-a \@ACM@journalfalse \@ACM@sigchiamodetrue \fi \ifx\ACM@fontsize\@empty \ifcase\ACM@format@nr \relax % manuscript \def\ACM@fontsize{9pt}% \or % acmsmall \def\ACM@fontsize{10pt}% \or % acmlarge \def\ACM@fontsize{10pt}% \or % acmtog \def\ACM@fontsize{9pt}% \or % sigconf \def\ACM@fontsize{9pt}% \or % siggraph \def\ACM@fontsize{9pt}% \or % sigplan \def\ACM@fontsize{10pt}% \or % sigchi \def\ACM@fontsize{10pt}% \or % sigchi-a \def\ACM@fontsize{10pt}% \fi \fi \ClassInfo{\@classname}{Using fontsize \ACM@fontsize} \LoadClass[\ACM@fontsize, reqno]{amsart} \RequirePackage{microtype} \RequirePackage{totpages} \RequirePackage{environ} \if@ACM@manuscript \RequirePackage{setspace} \onehalfspacing \fi \RequirePackage{textcase} \if@ACM@natbib \RequirePackage{natbib} \renewcommand{\bibsection}{% \section{\refname}% \phantomsection\addcontentsline{toc}{section}{\refname}% } \renewcommand{\bibfont}{\bibliofont} \renewcommand\setcitestyle[1]{ \@for\@tempa:=#1\do {\def\@tempb{round}\ifx\@tempa\@tempb \renewcommand\NAT@open{(}\renewcommand\NAT@close{)}\fi \def\@tempb{square}\ifx\@tempa\@tempb \renewcommand\NAT@open{[}\renewcommand\NAT@close{]}\fi \def\@tempb{angle}\ifx\@tempa\@tempb \renewcommand\NAT@open{$<$}\renewcommand\NAT@close{$>$}\fi \def\@tempb{curly}\ifx\@tempa\@tempb \renewcommand\NAT@open{\{}\renewcommand\NAT@close{\}}\fi \def\@tempb{semicolon}\ifx\@tempa\@tempb \renewcommand\NAT@sep{;}\fi \def\@tempb{colon}\ifx\@tempa\@tempb \renewcommand\NAT@sep{;}\fi \def\@tempb{comma}\ifx\@tempa\@tempb \renewcommand\NAT@sep{,}\fi \def\@tempb{authoryear}\ifx\@tempa\@tempb \NAT@numbersfalse\fi \def\@tempb{numbers}\ifx\@tempa\@tempb \NAT@numberstrue\NAT@superfalse\fi \def\@tempb{super}\ifx\@tempa\@tempb \NAT@numberstrue\NAT@supertrue\fi \def\@tempb{nobibstyle}\ifx\@tempa\@tempb \let\bibstyle=\@gobble\fi \def\@tempb{bibstyle}\ifx\@tempa\@tempb \let\bibstyle=\@citestyle\fi \def\@tempb{sort}\ifx\@tempa\@tempb \def\NAT@sort{\@ne}\fi \def\@tempb{nosort}\ifx\@tempa\@tempb \def\NAT@sort{\z@}\fi \def\@tempb{compress}\ifx\@tempa\@tempb \def\NAT@cmprs{\@ne}\fi \def\@tempb{nocompress}\ifx\@tempa\@tempb \def\NAT@cmprs{\@z}\fi \def\@tempb{sort&compress}\ifx\@tempa\@tempb \def\NAT@sort{\@ne}\def\NAT@cmprs{\@ne}\fi \def\@tempb{mcite}\ifx\@tempa\@tempb \let\NAT@merge\@ne\fi \def\@tempb{merge}\ifx\@tempa\@tempb \@ifnum{\NAT@merge<\tw@}{\let\NAT@merge\tw@}{}\fi \def\@tempb{elide}\ifx\@tempa\@tempb \@ifnum{\NAT@merge<\thr@@}{\let\NAT@merge\thr@@}{}\fi \def\@tempb{longnamesfirst}\ifx\@tempa\@tempb \NAT@longnamestrue\fi \def\@tempb{nonamebreak}\ifx\@tempa\@tempb \def\NAT@nmfmt#1{\mbox{\NAT@up#1}}\fi \expandafter\NAT@find@eq\@tempa=\relax\@nil \if\@tempc\relax\else \expandafter\NAT@rem@eq\@tempc \def\@tempb{open}\ifx\@tempa\@tempb \xdef\NAT@open{\@tempc}\fi \def\@tempb{close}\ifx\@tempa\@tempb \xdef\NAT@close{\@tempc}\fi \def\@tempb{aysep}\ifx\@tempa\@tempb \xdef\NAT@aysep{\@tempc}\fi \def\@tempb{yysep}\ifx\@tempa\@tempb \xdef\NAT@yrsep{\@tempc}\fi \def\@tempb{notesep}\ifx\@tempa\@tempb \xdef\NAT@cmt{\@tempc}\fi \def\@tempb{citesep}\ifx\@tempa\@tempb \xdef\NAT@sep{\@tempc}\fi \fi }% \NAT@@setcites } \renewcommand\citestyle[1]{% \ifcsname bibstyle@#1\endcsname% \csname bibstyle@#1\endcsname\let\bibstyle\@gobble% \else% \@latex@error{Undefined `#1' citestyle}% \fi }% \fi \newcommand{\bibstyle@acmauthoryear}{% \setcitestyle{% authoryear,% open={[},close={]},citesep={;},% aysep={},yysep={,},% notesep={, }}} \newcommand{\bibstyle@acmnumeric}{% \setcitestyle{% numbers,sort&compress,% open={[},close={]},citesep={,},% notesep={, }}} \if@ACM@natbib \citestyle{acmnumeric} \fi \def\@startsection#1#2#3#4#5#6{% \if@noskipsec \leavevmode \fi \par \@tempskipa #4\relax \@afterindenttrue \ifdim \@tempskipa <\z@ \@tempskipa -\@tempskipa \@afterindentfalse \fi \if@nobreak \everypar{}% \else \addpenalty\@secpenalty\addvspace\@tempskipa \fi \@ifstar {\@ssect{#3}{#4}{#5}{#6}}% {\@dblarg{\@sect{#1}{#2}{#3}{#4}{#5}{#6}}}} \def\@sect#1#2#3#4#5#6[#7]#8{% \edef\@toclevel{\ifnum#2=\@m 0\else\number#2\fi}% \ifnum #2>\c@secnumdepth \let\@svsec\@empty \else \refstepcounter{#1}% \protected@edef\@svsec{\@seccntformat{#1}\relax}% \fi \@tempskipa #5\relax \ifdim \@tempskipa>\z@ \begingroup #6{% \@hangfrom{\hskip #3\relax\@svsec}% \interlinepenalty \@M #8\@@par}% \endgroup \csname #1mark\endcsname{#7}% \ifnum #2>\c@secnumdepth \else \@tochangmeasure{\csname the#1\endcsname}% \fi \addcontentsline{toc}{#1}{% \ifnum #2>\c@secnumdepth \else \protect\numberline{\csname the#1\endcsname}% \fi #7}% \else \def\@svsechd{% #6{\hskip #3\relax \@svsec #8}% \csname #1mark\endcsname{#7}% \ifnum #2>\c@secnumdepth \else \@tochangmeasure{\csname the#1\endcsname\space}% \fi \addcontentsline{toc}{#1}{% \ifnum #2>\c@secnumdepth \else \protect\numberline{\csname the#1\endcsname}% \fi #7}}% \fi \@xsect{#5}} \def\@xsect#1{% \@tempskipa #1\relax \ifdim \@tempskipa>\z@ \par \nobreak \vskip \@tempskipa \@afterheading \else \@nobreakfalse \global\@noskipsectrue \everypar{% \if@noskipsec \global\@noskipsecfalse {\setbox\z@\lastbox}% \clubpenalty\@M \begingroup \@svsechd \endgroup \unskip \@tempskipa #1\relax \hskip -\@tempskipa \else \clubpenalty \@clubpenalty \everypar{}% \fi}% \fi \ignorespaces} \def\@seccntformat#1{\csname the#1\endcsname\quad} \def\@ssect#1#2#3#4#5{% \@tempskipa #3\relax \ifdim \@tempskipa>\z@ \begingroup #4{% \@hangfrom{\hskip #1}% \interlinepenalty \@M #5\@@par}% \endgroup \else \def\@svsechd{#4{\hskip #1\relax #5}}% \fi \@xsect{#3}} \def\@starttoc#1#2{\begingroup\makeatletter \setTrue{#1}% \par\removelastskip\vskip\z@skip \@startsection{section}\@M\z@{\linespacing\@plus\linespacing}% {.5\linespacing}{\centering\contentsnamefont}{#2}% \@input{\jobname.#1}% \if@filesw \@xp\newwrite\csname tf@#1\endcsname \immediate\@xp\openout\csname tf@#1\endcsname \jobname.#1\relax \fi \global\@nobreakfalse \endgroup \addvspace{32\p@\@plus14\p@}% } \def\l@section{\@tocline{1}{0pt}{1pc}{2pc}{}} \def\l@subsection{\@tocline{2}{0pt}{1pc}{3pc}{}} \def\l@subsubsection{\@tocline{2}{0pt}{1pc}{5pc}{}} \let\@footnotemark@nolink\@footnotemark \let\@footnotetext@nolink\@footnotetext \RequirePackage[bookmarksnumbered,unicode]{hyperref} \pdfstringdefDisableCommands{% \def\unskip{}% \def\textbullet{- }% \def\textrightarrow{ -> }% \def\footnotemark{}% } \urlstyle{rm} \ifcase\ACM@format@nr \relax % manuscript \or % acmsmall \or % acmlarge \or % acmtog \or % sigconf \or % siggraph \or % sigplan \urlstyle{sf} \or % sigchi \or % sigchi-a \urlstyle{sf} \fi \if@ACM@screen \hypersetup{colorlinks, linkcolor=ACMRed, citecolor=ACMPurple, urlcolor=ACMDarkBlue, filecolor=ACMDarkBlue} \else \hypersetup{hidelinks} \fi \RequirePackage{cleveref} \if@ACM@natbib \let\citeN\cite \let\cite\citep \let\citeANP\citeauthor \let\citeNN\citeyearpar \let\citeyearNP\citeyear \let\citeyear\citeyearpar \let\citeNP\citealt \DeclareRobustCommand\citeA {\begingroup\NAT@swafalse \let\NAT@ctype\@ne\NAT@partrue\NAT@fullfalse\NAT@open\NAT@citetp}% \providecommand\newblock{}% \else \AtBeginDocument{% \let\shortcite\cite% \providecommand\citename[1]{#1}} \fi \newcommand\shortcite[2][]{% \ifNAT@numbers\cite[#1]{#2}\else\citeyear[#1]{#2}\fi} \def\bibliographystyle#1{% \ifx\@begindocumenthook\@undefined\else \expandafter\AtBeginDocument \fi {\if@filesw \immediate\write\@auxout{\string\bibstyle{#1}}% \fi}} \RequirePackage{graphicx, xcolor} \definecolor[named]{ACMBlue}{cmyk}{1,0.1,0,0.1} \definecolor[named]{ACMYellow}{cmyk}{0,0.16,1,0} \definecolor[named]{ACMOrange}{cmyk}{0,0.42,1,0.01} \definecolor[named]{ACMRed}{cmyk}{0,0.90,0.86,0} \definecolor[named]{ACMLightBlue}{cmyk}{0.49,0.01,0,0} \definecolor[named]{ACMGreen}{cmyk}{0.20,0,1,0.19} \definecolor[named]{ACMPurple}{cmyk}{0.55,1,0,0.15} \definecolor[named]{ACMDarkBlue}{cmyk}{1,0.58,0,0.21} \if@ACM@authordraft \RequirePackage{draftwatermark} \SetWatermarkFontSize{0.5in} \SetWatermarkColor[gray]{.9} \SetWatermarkText{\parbox{12em}{\centering Unpublished working draft\\ Not for distribution}} \fi \RequirePackage{geometry} \ifcase\ACM@format@nr \relax % manuscript \geometry{letterpaper,head=13pt, marginparwidth=6pc,heightrounded}% \or % acmsmall \geometry{twoside=true, includeheadfoot, head=13pt, foot=2pc, paperwidth=6.75in, paperheight=10in, top=58pt, bottom=44pt, inner=46pt, outer=46pt, marginparwidth=2pc,heightrounded }% \or % acmlarge \geometry{twoside=true, head=13pt, foot=2pc, paperwidth=8.5in, paperheight=11in, includeheadfoot, top=78pt, bottom=114pt, inner=81pt, outer=81pt, marginparwidth=4pc,heightrounded }% \or % acmtog \geometry{twoside=true, head=13pt, foot=2pc, paperwidth=8.5in, paperheight=11in, includeheadfoot, columnsep=24pt, top=52pt, bottom=75pt, inner=52pt, outer=52pt, marginparwidth=2pc,heightrounded }% \or % sigconf \geometry{twoside=true, head=13pt, paperwidth=8.5in, paperheight=11in, includeheadfoot, columnsep=2pc, top=57pt, bottom=73pt, inner=54pt, outer=54pt, marginparwidth=2pc,heightrounded }% \or % siggraph \geometry{twoside=true, head=13pt, paperwidth=8.5in, paperheight=11in, includeheadfoot, columnsep=2pc, top=57pt, bottom=73pt, inner=54pt, outer=54pt, marginparwidth=2pc,heightrounded }% \or % sigplan \geometry{twoside=true, head=13pt, paperwidth=8.5in, paperheight=11in, includeheadfoot=false, columnsep=2pc, top=1in, bottom=1in, inner=0.75in, outer=0.75in, marginparwidth=2pc,heightrounded }% \or % sigchi \geometry{twoside=true, head=13pt, paperwidth=8.5in, paperheight=11in, includeheadfoot, columnsep=2pc, top=66pt, bottom=73pt, inner=54pt, outer=54pt, marginparwidth=2pc,heightrounded }% \or % sigchi-a \geometry{twoside=false, head=13pt, paperwidth=11in, paperheight=8.5in, includeheadfoot, marginparsep=72pt, marginparwidth=170pt, columnsep=20pt, top=72pt, bottom=72pt, left=314pt, right=72pt }% \@mparswitchfalse \reversemarginpar \fi \setlength\parindent{10\p@} \setlength\parskip{\z@} \ifcase\ACM@format@nr \relax % manuscript \or % acmsmall \or % acmlarge \or % acmtog \setlength\parindent{9\p@}% \or % sigconf \or % siggraph \or % sigplan \or % sigchi \or % sigchi-a \fi \setlength\normalparindent{\parindent} \def\copyrightpermissionfootnoterule{\kern-3\p@ \hrule \@width \columnwidth \kern 2.6\p@} \RequirePackage{manyfoot} \SelectFootnoteRule[2]{copyrightpermission} \DeclareNewFootnote{copyrightpermission} \def\footnoterule{\kern-3\p@ \hrule \@width 4pc \kern 2.6\p@} \def\endminipage{% \par \unskip \ifvoid\@mpfootins\else \vskip\skip\@mpfootins \normalcolor \unvbox\@mpfootins \fi \@minipagefalse %% added 24 May 89 \color@endgroup \egroup \expandafter\@iiiparbox\@mpargs{\unvbox\@tempboxa}} \def\@makefntext{\noindent\@makefnmark} \if@ACM@sigchiamode \long\def\@footnotetext#1{\marginpar{% \reset@font\small \interlinepenalty\interfootnotelinepenalty \protected@edef\@currentlabel{% \csname p@footnote\endcsname\@thefnmark }% \color@begingroup \@makefntext{% \rule\z@\footnotesep\ignorespaces#1\@finalstrut\strutbox}% \color@endgroup}}% \fi \long\def\@mpfootnotetext#1{% \global\setbox\@mpfootins\vbox{% \unvbox\@mpfootins \reset@font\footnotesize \hsize\columnwidth \@parboxrestore \protected@edef\@currentlabel {\csname p@mpfootnote\endcsname\@thefnmark}% \color@begingroup\centering \@makefntext{% \rule\z@\footnotesep\ignorespaces#1\@finalstrut\strutbox}% \color@endgroup}} \def\@makefnmark{\hbox{\@textsuperscript{\normalfont\@thefnmark}}} \def\@textbottom{\vskip \z@ \@plus 1pt} \let\@texttop\relax \RequirePackage{iftex} \ifPDFTeX \input{glyphtounicode} \pdfglyphtounicode{f_f}{FB00} \pdfglyphtounicode{f_f_i}{FB03} \pdfglyphtounicode{f_f_l}{FB04} \pdfglyphtounicode{f_i}{FB01} \pdfglyphtounicode{t_t}{00740074} \pdfglyphtounicode{f_t}{00660074} \pdfglyphtounicode{T_h}{00540068} \pdfgentounicode=1 \fi \RequirePackage{cmap} \newif\if@ACM@newfonts \@ACM@newfontstrue \IfFileExists{libertine.sty}{}{\ClassWarning{\@classname}{You do not have libertine package installed. Please upgrade your TeX}\@ACM@newfontsfalse} \IfFileExists{zi4.sty}{}{\ClassWarning{\@classname}{You do not have zi4 package installed. Please upgrade your TeX}\@ACM@newfontsfalse} \IfFileExists{newtxmath.sty}{}{\ClassWarning{\@classname}{You do not have newtxmath package installed. Please upgrade your TeX}\@ACM@newfontsfalse} \if@ACM@newfonts \RequirePackage[tt=false, type1=true]{libertine} \RequirePackage[varqu]{zi4} \RequirePackage[libertine]{newtxmath} \RequirePackage[T1]{fontenc} \fi \if@ACM@sigchiamode \renewcommand{\familydefault}{\sfdefault} \fi \RequirePackage{caption, float} \captionsetup[table]{position=top} \if@ACM@journal \captionsetup{labelfont={sf, small}, textfont={sf, small}, margin=\z@} \captionsetup[figure]{name={Fig.}} \else \captionsetup{labelfont={bf}, textfont={bf}, labelsep=colon, margin=\z@} \ifcase\ACM@format@nr \relax % manuscript \or % acmsmall \or % acmlarge \or % acmtog \or % sigconf \or % siggraph \captionsetup{textfont={it}} \or % sigplan \captionsetup{labelfont={bf}, textfont={normalfont}, labelsep=period, margin=\z@} \or % sigchi \captionsetup[figure]{labelfont={bf, small}, textfont={bf, small}} \or % sigchi-a \captionsetup[figure]{labelfont={bf, small}, textfont={bf, small}} \fi \fi \newfloat{sidebar}{}{sbar} \floatname{sidebar}{Sidebar} \renewenvironment{sidebar}{\Collect@Body\@sidebar}{} \long\def\@sidebar#1{\bgroup\captionsetup{type=sidebar}% \marginpar{\small#1}\egroup} \newenvironment{marginfigure}{\Collect@Body\@marginfigure}{} \long\def\@marginfigure#1{\bgroup\captionsetup{type=figure}% \marginpar{\centering\small#1}\egroup} \newenvironment{margintable}{\Collect@Body\@margintable}{} \long\def\@margintable#1{\bgroup\captionsetup{type=table}% \marginpar{\centering\small#1}\egroup} \newdimen\fulltextwidth \fulltextwidth=\dimexpr(\textwidth+\marginparwidth+\marginparsep) \if@ACM@sigchiamode \def\@dblfloat{\bgroup\columnwidth=\fulltextwidth \let\@endfloatbox\@endwidefloatbox \def\@fpsadddefault{\def\@fps{tp}}% \@float} \fi \if@ACM@sigchiamode \def\end@dblfloat{% \end@float\egroup} \fi \def\@endwidefloatbox{% \par\vskip\z@skip \@minipagefalse \outer@nobreak \egroup \color@endbox \global\setbox\@currbox=\vbox{\moveleft \dimexpr(\fulltextwidth-\textwidth)\box\@currbox}% \wd\@currbox=\textwidth } \ifcase\ACM@format@nr \relax % manuscript \or % acmsmall \or % acmlarge \or % acmtog \or % sigconf \or % siggraph \or % sigplan \def\labelenumi{\theenumi.} \def\labelenumii{\theenumii.} \def\labelenumiii{\theenumiii.} \def\labelenumiv{\theenumiv.} \or % sigchi \or % sigchi-a \fi \newdimen\@ACM@labelwidth \AtBeginDocument{% \setlength\labelsep{4pt} \setlength{\@ACM@labelwidth}{6.5pt} %% First-level list: when beginning after the first line of an %% indented paragraph or ending before an indented paragraph, labels %% should not hang to the left of the preceding/following text. \setlength\leftmargini{\z@} \addtolength\leftmargini{\parindent} \addtolength\leftmargini{2\labelsep} \addtolength\leftmargini{\@ACM@labelwidth} %% Second-level and higher lists. \setlength\leftmarginii{\z@} \addtolength\leftmarginii{0.5\labelsep} \addtolength\leftmarginii{\@ACM@labelwidth} \setlength\leftmarginiii{\leftmarginii} \setlength\leftmarginiv{\leftmarginiii} \setlength\leftmarginv{\leftmarginiv} \setlength\leftmarginvi{\leftmarginv} \@listi} \newskip\listisep \listisep\smallskipamount \def\@listI{\leftmargin\leftmargini \labelwidth\leftmargini \advance\labelwidth-\labelsep \listparindent\z@ \topsep\listisep} \let\@listi\@listI \def\@listii{\leftmargin\leftmarginii \labelwidth\leftmarginii \advance\labelwidth-\labelsep \topsep\z@skip} \def\@listiii{\leftmargin\leftmarginiii \labelwidth\leftmarginiii \advance\labelwidth-\labelsep} \def\@listiv{\leftmargin\leftmarginiv \labelwidth\leftmarginiv \advance\labelwidth-\labelsep} \def\@listv{\leftmargin\leftmarginv \labelwidth\leftmarginv \advance\labelwidth-\labelsep} \def\@listvi{\leftmargin\leftmarginvi \labelwidth\leftmarginvi \advance\labelwidth-\labelsep} \renewcommand{\descriptionlabel}[1]{\upshape\bfseries #1} \renewenvironment{description}{\list{}{% \labelwidth\@ACM@labelwidth \let\makelabel\descriptionlabel}% }{ \endlist } \let\enddescription=\endlist % for efficiency \define@choicekey+{ACM}{acmJournal}[\@journalCode\@journalCode@nr]{% CIE,% CSUR,% IMWUT,% JACM,% JDIQ,% JEA,% JERIC,% JETC,% JOCCH,% PACMHCI,% PACMPL,% POMACS,% TAAS,% TACCESS,% TACO,% TALG,% TALLIP,% TAP,% TCPS,% TEAC,% TECS,% TIIS,% TISSEC,% TIST,% TKDD,% TMIS,% TOCE,% TOCHI,% TOCL,% TOCS,% TOCT,% TODAES,% TODS,% TOG,% TOIS,% TOIT,% TOMACS,% TOMM,% TOMPECS,% TOMS,% TOPC,% TOPS,% TOPLAS,% TOS,% TOSEM,% TOSN,% TRETS,% TSAS,% TSC,% TSLP,% TWEB% }{% \ifcase\@journalCode@nr \relax % CIE \def\@journalName{ACM Computers in Entertainment}% \def\@journalNameShort{ACM Comput. Entertain.}% \def\@permissionCodeOne{1544-3574}% \or % CSUR \def\@journalName{ACM Computing Surveys}% \def\@journalNameShort{ACM Comput. Surv.}% \def\@permissionCodeOne{0360-0300}% \or % IMWUT \def\@journalName{Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies}% \def\@journalNameShort{Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.}% \def\@permissionCodeOne{2474-9567}% \or % JACM \def\@journalName{Journal of the ACM}% \def\@journalNameShort{J. ACM}% \def\@permissionCodeOne{0004-5411}% \or % JDIQ \def\@journalName{ACM Journal of Data and Information Quality}% \def\@journalNameShort{ACM J. Data Inform. Quality}% \def\@permissionCodeOne{1936-1955}% \or % JEA \def\@journalName{ACM Journal of Experimental Algorithmics}% \def\@journalNameShort{ACM J. Exp. Algor.}% \def\@permissionCodeOne{1084-6654}% \or % JERIC \def\@journalName{ACM Journal of Educational Resources in Computing}% \def\@journalNameShort{ACM J. Edu. Resources in Comput.}% \def\@permissionCodeOne{1073-0516}% \or % JETC \def\@journalName{ACM Journal on Emerging Technologies in Computing Systems}% \def\@journalNameShort{ACM J. Emerg. Technol. Comput. Syst.}% \def\@permissionCodeOne{1550-4832}% \or % JOCCH \def\@journalName{ACM Journal on Computing and Cultural Heritage}% \def\@journalName{ACM J. Comput. Cult. Herit.}% \or % PACMHCI \def\@journalName{Proceedings of the ACM on Human-Computer Interaction}% \def\@journalName{Proc. ACM Hum.-Comput. Interact.}% \def\@permissionCodeOne{2573-0142}% \or % PACMPL \def\@journalName{Proceedings of the ACM on Programming Languages}% \def\@journalName{Proc. ACM Program. Lang.}% \def\@permissionCodeOne{2475-1421}% \or % POMACS \def\@journalName{Proceedings of the ACM on Measurement and Analysis of Computing Systems}% \def\@journalName{Proc. ACM Meas. Anal. Comput. Syst.}% \def\@permissionCodeOne{2476-1249}% \or % TAAS \def\@journalName{ACM Transactions on Autonomous and Adaptive Systems}% \def\@journalNameShort{ACM Trans. Autonom. Adapt. Syst.}% \def\@permissionCodeOne{1556-4665}% \or % TACCESS \def\@journalName{ACM Transactions on Accessible Computing}% \def\@journalNameShort{ACM Trans. Access. Comput.}% \def\@permissionCodeOne{1936-7228}% \or % TACO \def\@journalName{ACM Transactions on Architecture and Code Optimization}% \or % TALG \def\@journalName{ACM Transactions on Algorithms}% \def\@journalNameShort{ACM Trans. Algor.}% \def\@permissionCodeOne{1549-6325}% \or % TALLIP \def\@journalName{ACM Transactions on Asian and Low-Resource Language Information Processing}% \def\@journalNameShort{ACM Trans. Asian Low-Resour. Lang. Inf. Process.}% \def\@permissionCodeOne{2375-4699}% \or % TAP \def\@journalName{ACM Transactions on Applied Perception}% \or % TCPS \def\@journalName{ACM Transactions on Cyber-Physical Systems}% \or % TEAC \def\@journalName{ACM Transactions on Economics and Computation}% \or % TECS \def\@journalName{ACM Transactions on Embedded Computing Systems}% \def\@journalNameShort{ACM Trans. Embedd. Comput. Syst.}% \def\@permissionCodeOne{1539-9087}% \or % TIIS \def\@journalName{ACM Transactions on Interactive Intelligent Systems}% \def\@journalNameShort{ACM Trans. Interact. Intell. Syst.}% \def\@permissionCodeOne{2160-6455}% \or % TISSEC \def\@journalName{ACM Transactions on Information and System Security}% \def\@journalNameShort{ACM Trans. Info. Syst. Sec.}% \def\@permissionCodeOne{1094-9224}% \or % TIST \def\@journalName{ACM Transactions on Intelligent Systems and Technology}% \def\@journalNameShort{ACM Trans. Intell. Syst. Technol.}% \def\@permissionCodeOne{2157-6904}% \or % TKDD \def\@journalName{ACM Transactions on Knowledge Discovery from Data}% \def\@journalNameShort{ACM Trans. Knowl. Discov. Data.}% \def\@permissionCodeOne{1556-4681}% \or % TMIS \def\@journalName{ACM Transactions on Management Information Systems}% \def\@journalNameShort{ACM Trans. Manag. Inform. Syst.}% \def\@permissionCodeOne{2158-656X}% \or % TOCE \def\@journalName{ACM Transactions on Computing Education}% \def\@journalNameShort{ACM Trans. Comput. Educ.}% \def\@permissionCodeOne{1946-6226}% \or % TOCHI \def\@journalName{ACM Transactions on Computer-Human Interaction}% \def\@journalNameShort{ACM Trans. Comput.-Hum. Interact.}% \def\@permissionCodeOne{1073-0516}% \or % TOCL \def\@journalName{ACM Transactions on Computational Logic}% \def\@journalNameShort{ACM Trans. Comput. Logic}% \def\@permissionCodeOne{1529-3785}% \or % TOCS \def\@journalName{ACM Transactions on Computer Systems}% \def\@journalNameShort{ACM Trans. Comput. Syst.}% \def\@permissionCodeOne{0734-2071}% \or % TOCT \def\@journalName{ACM Transactions on Computation Theory}% \def\@journalNameShort{ACM Trans. Comput. Theory}% \def\@permissionCodeOne{1942-3454}% \or % TODAES \def\@journalName{ACM Transactions on Design Automation of Electronic Systems}% \def\@journalNameShort{ACM Trans. Des. Autom. Electron. Syst.}% \def\@permissionCodeOne{1084-4309}% \or % TODS \def\@journalName{ACM Transactions on Database Systems}% \def\@journalNameShort{ACM Trans. Datab. Syst.}% \def\@permissionCodeOne{0362-5915}% \or % TOG \def\@journalName{ACM Transactions on Graphics}% \def\@journalNameShort{ACM Trans. Graph.}% \def\@permissionCodeOne{0730-0301} \or % TOIS \def\@journalName{ACM Transactions on Information Systems}% \def\@journalName{ACM Transactions on Information Systems}% \def\@permissionCodeOne{1046-8188}% \or % TOIT \def\@journalName{ACM Transactions on Internet Technology}% \def\@journalNameShort{ACM Trans. Internet Technol.}% \def\@permissionCodeOne{1533-5399}% \or % TOMACS \def\@journalName{ACM Transactions on Modeling and Computer Simulation}% \def\@journalName{ACM Transactions on Modeling and Computer Simulation}% \def\@journalNameShort{ACM Trans. Model. Comput. Simul.}% \or % TOMM \def\@journalName{ACM Transactions on Multimedia Computing, Communications and Applications}% \def\@journalNameShort{ACM Trans. Multimedia Comput. Commun. Appl.}% \def\@permissionCodeOne{1551-6857}% \def\@permissionCodeTwo{0100}% \or % TOMPECS \def\@journalName{ACM Transactions on Modeling and Performance Evaluation of Computing Systems}% \def\@journalNameShort{ACM Trans. Model. Perform. Eval. Comput. Syst.}% \def\@permissionCodeOne{2376-3639}% \or % TOMS \def\@journalName{ACM Transactions on Mathematical Software}% \def\@journalNameShort{ACM Trans. Math. Softw.}% \def\@permissionCodeOne{0098-3500}% \or % TOPC \def\@journalName{ACM Transactions on Parallel Computing}% \def\@journalNameShort{ACM Trans. Parallel Comput.}% \def\@permissionCodeOne{1539-9087}% \or % TOPS \def\@journalName{ACM Transactions on Privacy and Security}% \def\@journalNameShort{ACM Trans. Priv. Sec.}% \def\@permissionCodeOne{2471-2566}% \or % TOPLAS \def\@journalName{ACM Transactions on Programming Languages and Systems}% \def\@journalNameShort{ACM Trans. Program. Lang. Syst.}% \def\@permissionCodeOne{0164-0925}% \or % TOS \def\@journalName{ACM Transactions on Storage}% \def\@journalNameShort{ACM Trans. Storage}% \def\@permissionCodeOne{1553-3077}% \or % TOSEM \def\@journalName{ACM Transactions on Software Engineering and Methodology}% \def\@journalNameShort{ACM Trans. Softw. Eng. Methodol.}% \def\@permissionCodeOne{1049-331X}% \or % TOSN \def\@journalName{ACM Transactions on Sensor Networks}% \def\@journalNameShort{ACM Trans. Sensor Netw.}% \def\@permissionCodeOne{1550-4859}% \or % TRETS \def\@journalName{ACM Transactions on Reconfigurable Technology and Systems}% \def\@journalNameShort{ACM Trans. Reconfig. Technol. Syst.}% \def\@permissionCodeOne{1936-7406}% \or % TSAS \def\@journalName{ACM Transactions on Spatial Algorithms and Systems}% \def\@journalNameShort{ACM Trans. Spatial Algorithms Syst.}% \def\@permissionCodeOne{2374-0353}% \or % TSC \def\@journalName{ACM Transactions on Social Computing}% \def\@journalNameShort{ACM Trans. Soc. Comput.}% \def\@permissionCodeOne{2469-7818}% \or % TSLP \def\@journalName{ACM Transactions on Speech and Language Processing}% \def\@journalNameShort{ACM Trans. Speech Lang. Process.}% \def\@permissionCodeOne{1550-4875}% \or % TWEB \def\@journalName{ACM Transactions on the Web}% \def\@journalNameShort{ACM Trans. Web}% \def\@permissionCodeOne{1559-1131}% \fi \ClassInfo{\@classname}{Using journal code \@journalCode}% }{% \ClassError{\@classname}{Incorrect journal #1}% }% \def\acmJournal#1{\setkeys{ACM}{acmJournal=#1}} \def\@journalCode@nr{0} \def\@journalName{}% \def\@journalNameShort{\@journalName}% \def\@permissionCodeOne{XXXX-XXXX}% \def\@permissionCodeTwo{}% \newcommand\acmConference[4][]{% \gdef\acmConference@shortname{#1}% \gdef\acmConference@name{#2}% \gdef\acmConference@date{#3}% \gdef\acmConference@venue{#4}% \ifx\acmConference@shortname\@empty \gdef\acmConference@shortname{#2}% \fi} \acmConference[Conference'17]{ACM Conference}{July 2017}{Washington, DC, USA} \def\subtitle#1{\def\@subtitle{#1}} \subtitle{} \newcount\num@authorgroups \num@authorgroups=0\relax \newif\if@insideauthorgroup \@insideauthorgroupfalse \renewcommand\author[2][]{% \if@insideauthorgroup\else \global\advance\num@authorgroups by 1\relax \global\@insideauthorgrouptrue \fi \ifx\addresses\@empty \if@ACM@anonymous \gdef\addresses{\@author{Anonymous Author(s)}}% \gdef\authors{Anonymous Author(s)}% \else \gdef\addresses{\@author{#2}}% \gdef\authors{#2}% \fi \else \if@ACM@anonymous\else \g@addto@macro\addresses{\and\@author{#2}}% \g@addto@macro\authors{\and#2}% \fi \fi \if@ACM@anonymous \ifx\shortauthors\@empty \gdef\shortauthors{Anon.}% \fi \else \def\@tempa{#1}% \ifx\@tempa\@empty \ifx\shortauthors\@empty \gdef\shortauthors{#2}% \else \g@addto@macro\shortauthors{\and#2}% \fi \else \ifx\shortauthors\@empty \gdef\shortauthors{#1}% \else \g@addto@macro\shortauthors{\and#1}% \fi \fi \fi} \newcommand{\affiliation}[2][]{% \global\@insideauthorgroupfalse \if@ACM@anonymous\else \g@addto@macro\addresses{\affiliation{#1}{#2}}% \fi} \define@boolkey+{@ACM@affiliation@}[@ACM@affiliation@]{obeypunctuation}% [true]{}{\ClassError{\@classname}{obeypunctuation must be true or false}} \def\additionalaffiliation#1{\authornote{\@additionalaffiliation{#1}}} \def\@additionalaffiliation#1{\bgroup \def\position##1{\ignorespaces}% \def\institution##1{##1\ignorespaces}% \def\department{\@ifnextchar[{\@department}{\@department[]}}% \def\@department[##1]##2{\unskip, ##2\ignorespaces}% \let\streetaddress\position \let\city\position \let\state\position \let\postcode\position \let\country\position Also with #1\unskip.\egroup} \renewcommand{\email}[2][]{% \if@ACM@anonymous\else \g@addto@macro\addresses{\email{#1}{#2}}% \fi} \def\orcid#1{\unskip\ignorespaces} \def\@titlenotes{} \def\titlenote#1{% \g@addto@macro\@title{\footnotemark}% \if@ACM@anonymous \g@addto@macro\@titlenotes{% \stepcounter{footnote}\footnotetext{Title note}}% \else \g@addto@macro\@titlenotes{\stepcounter{footnote}\footnotetext{#1}}% \fi} \def\@subtitlenotes{} \def\subtitlenote#1{% \g@addto@macro\@subtitle{\footnotemark}% \if@ACM@anonymous \g@addto@macro\@subtitlenotes{% \stepcounter{footnote}\footnotetext{Subtitle note}}% \else \g@addto@macro\@subtitlenotes{% \stepcounter{footnote}\footnotetext{#1}}% \fi} \def\@authornotes{} \def\authornote#1{% \if@ACM@anonymous\else \g@addto@macro\addresses{\@authornotemark}% \g@addto@macro\@authornotes{% \stepcounter{footnote}\footnotetext{#1}}% \fi} \newcommand\authornotemark[1][\relax]{% \ifx#1\relax\relax\relax \g@addto@macro\addresses{\@authornotemark}% \else \g@addto@macro\addresses{\@@authornotemark{#1}}% \fi} \def\acmVolume#1{\def\@acmVolume{#1}} \acmVolume{1} \def\acmNumber#1{\def\@acmNumber{#1}} \acmNumber{1} \def\acmArticle#1{\def\@acmArticle{#1}} \acmArticle{1} \def\acmArticleSeq#1{\def\@acmArticleSeq{#1}} \acmArticleSeq{\@acmArticle} \def\acmYear#1{\def\@acmYear{#1}} \acmYear{\the\year} \def\acmMonth#1{\def\@acmMonth{#1}} \acmMonth{\the\month} \def\@acmPubDate{\ifcase\@acmMonth\or January\or February\or March\or April\or May\or June\or July\or August\or September\or October\or November\or December\fi~\@acmYear} \def\acmPrice#1{\def\@acmPrice{#1}} \acmPrice{15.00} \def\acmSubmissionID#1{\def\@acmSubmissionID{#1}} \acmSubmissionID{} \def\acmISBN#1{\def\@acmISBN{#1}} \acmISBN{978-x-xxxx-xxxx-x/YY/MM} \def\acmDOI#1{\def\@acmDOI{#1}} \acmDOI{10.1145/nnnnnnn.nnnnnnn} \newif\if@ACM@badge \@ACM@badgefalse \newlength\@ACM@badge@width \setlength\@ACM@badge@width{5pc} \newlength\@ACM@title@width \newlength\@ACM@badge@skip \setlength\@ACM@badge@skip{1pc} \newcommand\acmBadgeR[2][]{\@ACM@badgetrue \def\@acmBadgeR@url{#1}% \def\@acmBadgeR@image{#2}} \def\@acmBadgeR@url{} \def\@acmBadgeR@image{} \newcommand\acmBadgeL[2][]{\@ACM@badgetrue \def\@acmBadgeL@url{#1}% \def\@acmBadgeL@image{#2}} \def\@acmBadgeL@url{} \def\@acmBadgeL@image{} \def\startPage#1{\def\@startPage{#1}} \startPage{} \def\terms#1{\ClassWarning{\@classname}{The command \string\terms{} is obsolete. I am going to ignore it}} \def\keywords#1{\def\@keywords{#1}} \keywords{} \renewenvironment{abstract}{\Collect@Body\@saveabstract}{} \long\def\@saveabstract#1{\long\gdef\@abstract{#1}} \@saveabstract{} \long\def\@lempty{} \define@boolkey+{@ACM@topmatter@}[@ACM@]{printccs}[true]{% \if@ACM@printccs \ClassInfo{\@classname}{Printing CCS}% \else \ClassInfo{\@classname}{Suppressing CCS}% \fi}{\ClassError{\@classname}{printccs must be true or false}} \define@boolkey+{@ACM@topmatter@}[@ACM@]{printacmref}[true]{% \if@ACM@printacmref \ClassInfo{\@classname}{Printing bibformat}% \else \ClassInfo{\@classname}{Suppressing bibformat}% \fi}{\ClassError{\@classname}{printacmref must be true or false}} \define@boolkey+{@ACM@topmatter@}[@ACM@]{printfolios}[true]{% \if@ACM@printfolios \ClassInfo{\@classname}{Printing folios}% \else \ClassInfo{\@classname}{Suppressing folios}% \fi}{\ClassError{\@classname}{printfolios must be true or false}} \define@cmdkey{@ACM@topmatter@}[@ACM@]{authorsperrow}[0]{% \IfInteger{#1}{\ClassInfo{\@classname}{Setting authorsperrow to #1}}{\ClassWarning{\@classname}{Parameter authorsperrow must be numerical. Ignoring the input #1}\gdef\@ACM@authorsperrow{0}}} \def\settopmatter#1{\setkeys{@ACM@topmatter@}{#1}} \settopmatter{printccs=true, printacmref=true} \if@ACM@manuscript \settopmatter{printfolios=true} \else \if@ACM@journal \settopmatter{printfolios=true} \else \settopmatter{printfolios=false} \fi \fi \settopmatter{authorsperrow=0} \def\@received{} \newcommand\received[2][]{\def\@tempa{#1}% \ifx\@tempa\@empty \ifx\@received\@empty \gdef\@received{Received #2}% \else \g@addto@macro{\@received}{; revised #2}% \fi \else \ifx\@received\@empty \gdef\@received{#1 #2}% \else \g@addto@macro{\@received}{; #1 #2}% \fi \fi} \AtEndDocument{% \ifx\@received\@empty\else \par\bigskip\noindent\small\normalfont\@received\par \fi} \RequirePackage{comment} \excludecomment{CCSXML} \let\@concepts\@empty \newcommand\ccsdesc[2][100]{% \ccsdesc@parse#1~#2~~\ccsdesc@parse@end} \RequirePackage{textcomp} \def\ccsdesc@parse#1~#2~#3~{% \expandafter\ifx\csname CCS@General@#2\endcsname\relax \expandafter\gdef\csname CCS@General@#2\endcsname{\textbullet\ \textbf{#2}}% \expandafter\gdef\csname CCS@Punctuation@#2\endcsname{; }% \expandafter\gdef\csname CCS@Specific@#2\endcsname{}% \g@addto@macro{\@concepts}{\csname CCS@General@#2\endcsname \csname CCS@Punctuation@#2\endcsname \csname CCS@Specific@#2\endcsname}% \fi \ifx#3\relax\relax\else \expandafter\gdef\csname CCS@Punctuation@#2\endcsname{ \textrightarrow\ }% \expandafter\g@addto@macro\expandafter{\csname CCS@Specific@#2\endcsname}{% \ifnum#1>499\textbf{#3}; \else \ifnum#1>299\textit{#3}; \else #3; \fi\fi}% \fi \ccsdesc@parse@finish} \def\ccsdesc@parse@finish#1\ccsdesc@parse@end{} \newif\if@printcopyright \@printcopyrighttrue \newif\if@printpermission \@printpermissiontrue \newif\if@acmowned \@acmownedtrue \define@choicekey{ACM@}{acmcopyrightmode}[% \acm@copyrightinput\acm@copyrightmode]{none,% acmcopyright,acmlicensed,rightsretained,% usgov,usgovmixed,cagov,cagovmixed,% licensedusgovmixed,% licensedcagov,licensedcagovmixed,othergov,licensedothergov}{% \@printpermissiontrue \@printcopyrighttrue \@acmownedtrue \ifnum\acm@copyrightmode=0\relax % none \@printpermissionfalse \@printcopyrightfalse \@acmownedfalse \fi \ifnum\acm@copyrightmode=2\relax % acmlicensed \@acmownedfalse \fi \ifnum\acm@copyrightmode=3\relax % rightsretained \@acmownedfalse \acmPrice{}% \fi \ifnum\acm@copyrightmode=4\relax % usgov \@printpermissiontrue \@printcopyrightfalse \@acmownedfalse \acmPrice{}% \fi \ifnum\acm@copyrightmode=6\relax % cagov \@acmownedfalse \fi \ifnum\acm@copyrightmode=8\relax % licensedusgovmixed \@acmownedfalse \fi \ifnum\acm@copyrightmode=9\relax % licensedcagov \@acmownedfalse \fi \ifnum\acm@copyrightmode=10\relax % licensedcagovmixed \@acmownedfalse \fi \ifnum\acm@copyrightmode=11\relax % othergov \@acmownedtrue \fi \ifnum\acm@copyrightmode=12\relax % licensedothergov \@acmownedfalse \fi} \def\setcopyright#1{\setkeys{ACM@}{acmcopyrightmode=#1}} \setcopyright{acmcopyright} \def\@copyrightowner{% \ifcase\acm@copyrightmode\relax % none \or % acmcopyright Association for Computing Machinery. \or % acmlicensed Copyright held by the owner/author(s). Publication rights licensed to Association for Computing Machinery. \or % rightsretained Copyright held by the owner/author(s). \or % usgov \or % usgovmixed Association for Computing Machinery. \or % cagov Crown in Right of Canada. \or %cagovmixed Association for Computing Machinery. \or %licensedusgovmixed Copyright held by the owner/author(s). Publication rights licensed to Association for Computing Machinery. \or % licensedcagov Crown in Right of Canada. Publication rights licensed to Association for Computing Machinery. \or %licensedcagovmixed Copyright held by the owner/author(s). Publication rights licensed to Association for Computing Machinery. \or % othergov Association for Computing Machinery. \or % licensedothergov Copyright held by the owner/author(s). Publication rights licensed to Association for Computing Machinery. \fi} \def\@formatdoi#1{\url{https://doi.org/#1}} \def\@copyrightpermission{% \ifcase\acm@copyrightmode\relax % none \or % acmcopyright Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and\hspace{.5pt}/or a fee. Request permissions from permissions@acm.org. \or % acmlicensed Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and\hspace{.5pt}/or a fee. Request permissions from permissions@acm.org. \or % rightsretained Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner\hspace{.5pt}/author(s). \or % usgov This paper is authored by an employee(s) of the United States Government and is in the public domain. Non-exclusive copying or redistribution is allowed, provided that the article citation is given and the authors and agency are clearly identified as its source. \or % usgovmixed ACM acknowledges that this contribution was authored or co-authored by an employee, contractor, or affiliate of the United States government. As such, the United States government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for government purposes only. \or % cagov This article was authored by employees of the Government of Canada. As such, the Canadian government retains all interest in the copyright to this work and grants to ACM a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, provided that clear attribution is given both to the authors and the Canadian government agency employing them. Permission to make digital or hard copies for personal or classroom use is granted. Copies must bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the Canadian Government must be honored. To copy otherwise, distribute, republish, or post, requires prior specific permission and\hspace{.5pt}/or a fee. Request permissions from permissions@acm.org. \or % cagovmixed ACM acknowledges that this contribution was co-authored by an affiliate of the national government of Canada. As such, the Crown in Right of Canada retains an equal interest in the copyright. Reprints must include clear attribution to ACM and the author's government agency affiliation. Permission to make digital or hard copies for personal or classroom use is granted. Copies must bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. To copy otherwise, distribute, republish, or post, requires prior specific permission and\hspace{.5pt}/or a fee. Request permissions from permissions@acm.org. \or % licensedusgovmixed Publication rights licensed to ACM\@. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only. \or % licensedcagov This article was authored by employees of the Government of Canada. As such, the Canadian government retains all interest in the copyright to this work and grants to ACM a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, provided that clear attribution is given both to the authors and the Canadian government agency employing them. Permission to make digital or hard copies for personal or classroom use is granted. Copies must bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the Canadian Government must be honored. To copy otherwise, distribute, republish, or post, requires prior specific permission and\hspace{.5pt}/or a fee. Request permissions from permissions@acm.org. \or % licensedcagovmixed Publication rights licensed to ACM\@. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the national government of Canada. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only. \or % othergov ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only. \or % licensedothergov Publication rights licensed to ACM\@. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only. \fi} \def\copyrightyear#1{\def\@copyrightyear{#1}} \copyrightyear{\@acmYear} \def\@teaserfigures{} \newenvironment{teaserfigure}{\Collect@Body\@saveteaser}{} \long\def\@saveteaser#1{\g@addto@macro\@teaserfigures{\@teaser{#1}}} \renewcommand{\thanks}[1]{% \@ifnotempty{#1}{% \if@ACM@anonymous \g@addto@macro\thankses{\thanks{A note}}% \else \g@addto@macro\thankses{\thanks{#1}}% \fi}} \newbox\mktitle@bx \def\maketitle{% \if@ACM@anonymous % Anonymize omission of \author-s \ifnum\num@authorgroups=0\author{}\fi \fi \begingroup \let\@footnotemark\@footnotemark@nolink \let\@footnotetext\@footnotetext@nolink \renewcommand\thefootnote{\@fnsymbol\c@footnote}% \global\@topnum\z@ % this prevents floats from falling % at the top of page 1 \global\@botnum\z@ % we do not want them to be on bottom either \hsize=\textwidth \def\@makefnmark{\hbox{\@textsuperscript{\@thefnmark}}}% \@mktitle\if@ACM@sigchiamode\else\@mkauthors\fi\@mkteasers \@printtopmatter \if@ACM@sigchiamode\@mkauthors\fi \setcounter{footnote}{0}% \def\@makefnmark{\hbox{\@textsuperscript{\normalfont\@thefnmark}}}% \@titlenotes \@subtitlenotes \@authornotes \let\@makefnmark\relax \let\@thefnmark\relax \let\@makefntext\noindent \ifx\@empty\thankses\else \footnotetextcopyrightpermission{% \def\par{\let\par\@par}\parindent\z@\@setthanks}% \fi \footnotetextcopyrightpermission{% \if@ACM@authordraft \raisebox{-2ex}[\z@][\z@]{\makebox[0pt][l]{\large\bfseries Unpublished working draft. Not for distribution}}% \color[gray]{0.9}% \fi \parindent\z@\parskip0.1\baselineskip \if@ACM@authorversion\else \if@printpermission\@copyrightpermission\par\fi \fi \if@ACM@manuscript\else \if@ACM@journal\else % Print the conference information {\itshape \acmConference@shortname, \acmConference@date, \acmConference@venue}\par \fi \fi \if@printcopyright \copyright\ \@copyrightyear\ \@copyrightowner\\ \else \@copyrightyear.\ \fi \if@ACM@manuscript Manuscript submitted to ACM\\ \else \if@ACM@authorversion This is the author's version of the work. It is posted here for your personal use. Not for redistribution. The definitive Version of Record was published in \if@ACM@journal \emph{\@journalName}% \else \emph{Proceedings of \acmConference@name, \acmConference@date}% \fi \ifx\@acmDOI\@empty . \else , \@formatdoi{\@acmDOI}. \fi\\ \else \if@ACM@journal \@permissionCodeOne/\@acmYear/\@acmMonth-ART\@acmArticle \ifx\@acmPrice\@empty\else\ \$\@acmPrice\fi\\ \@formatdoi{\@acmDOI}% \else % Conference ACM~ISBN~\@acmISBN \ifx\@acmPrice\@empty.\else\dots\$\@acmPrice\fi\\ \@formatdoi{\@acmDOI}% \fi \fi \fi} \endgroup \setcounter{footnote}{0}% \@mkabstract \if@ACM@printccs \ifx\@concepts\@empty\else\bgroup {\@specialsection{CCS Concepts}% \@concepts\par}\egroup \fi \fi \ifx\@keywords\@empty\else\bgroup {\if@ACM@journal \@specialsection{Additional Key Words and Phrases}% \else \@specialsection{Keywords}% \fi \@keywords}\par\egroup \fi \andify\authors \andify\shortauthors \global\let\authors=\authors \global\let\shortauthors=\shortauthors \if@ACM@printacmref \@mkbibcitation \fi \hypersetup{pdfauthor={\authors}, pdftitle={\@title}, pdfsubject={\@concepts}, pdfkeywords={\@keywords}}% \@printendtopmatter \@afterindentfalse \@afterheading } \def\@specialsection#1{% \ifcase\ACM@format@nr \relax % manuscript \par\medskip\small\noindent#1: % \or % acmsmall \par\medskip\small\noindent#1: % \or % acmlarge \par\medskip\small\noindent#1: % \or % acmtog \par\medskip\small\noindent#1: % \or % sigconf \section{#1}% \or % siggraph \section{#1}% \or % sigplan \paragraph{#1}% \or % sigchi \section{#1}% \or % sigchi-a \section{#1}% \fi} \def\@printtopmatter{% \ifx\@startPage\@empty \gdef\@startPage{1}% \else \setcounter{page}{\@startPage}% \fi \thispagestyle{firstpagestyle}% \noindent \ifcase\ACM@format@nr \relax % manuscript \box\mktitle@bx\par \noindent\hrulefill\par \or % acmsmall \box\mktitle@bx\par \noindent\hrulefill\par \or % acmlarge \box\mktitle@bx\par \noindent\hrulefill\par \or % acmtog \twocolumn[\box\mktitle@bx]% \or % sigconf \twocolumn[\box\mktitle@bx]% \or % siggraph \twocolumn[\box\mktitle@bx]% \or % sigplan \twocolumn[\box\mktitle@bx]% \or % sigchi \twocolumn[\box\mktitle@bx]% \or % sigchi-a \par\box\mktitle@bx\par\bigskip \if@ACM@badge \marginpar{\noindent \ifx\@acmBadgeL@image\@empty\else \href{\@acmBadgeL@url}{% \includegraphics[width=\@ACM@badge@width]{\@acmBadgeL@image}}% \hskip\@ACM@badge@skip \fi \ifx\@acmBadgeR@image\@empty\else \href{\@acmBadgeR@url}{% \includegraphics[width=\@ACM@badge@width]{\@acmBadgeR@image}}% \fi}% \fi \fi } \def\@mktitle{% \ifcase\ACM@format@nr \relax % manuscript \@mktitle@i \or % acmsmall \@mktitle@i \or % acmlarge \@mktitle@i \or % acmtog \@mktitle@i \or % sigconf \@mktitle@iii \or % siggraph \@mktitle@iii \or % sigplan \@mktitle@iii \or % sigchi \@mktitle@iii \or % sigchi-a \@mktitle@iv \fi } \def\@titlefont{% \ifcase\ACM@format@nr \relax % manuscript \LARGE\bfseries\sffamily \or % acmsmall \LARGE\bfseries\sffamily \or % acmlarge \LARGE\bfseries\sffamily \or % acmtog \Huge\sffamily \or % sigconf \Huge\sffamily\bfseries \or % siggraph \Huge\sffamily\bfseries \or % sigplan \Huge\bfseries \or % sigchi \Huge\sffamily\bfseries \or % sigchi-a \Huge\bfseries \fi} \def\@subtitlefont{\normalsize \ifcase\ACM@format@nr \relax % manuscript \mdseries \or % acmsmall \mdseries \or % acmlarge \mdseries \or % acmtog \LARGE \or % sigconf \LARGE\mdseries \or % siggraph \LARGE\mdseries \or % sigplan \LARGE\mdseries \or % sigchi \LARGE\mdseries \or % sigchi-a \mdseries \fi} \def\@mktitle@i{\hsize=\textwidth \@ACM@title@width=\hsize \ifx\@acmBadgeL@image\@empty\else \advance\@ACM@title@width by -\@ACM@badge@width \advance\@ACM@title@width by -\@ACM@badge@skip \fi \ifx\@acmBadgeR@image\@empty\else \advance\@ACM@title@width by -\@ACM@badge@width \advance\@ACM@title@width by -\@ACM@badge@skip \fi \setbox\mktitle@bx=\vbox{\noindent\@titlefont \ifx\@acmBadgeL@image\@empty\else \raisebox{-.5\baselineskip}[\z@][\z@]{\href{\@acmBadgeL@url}{% \includegraphics[width=\@ACM@badge@width]{\@acmBadgeL@image}}}% \hskip\@ACM@badge@skip \fi \parbox[t]{\@ACM@title@width}{\raggedright \@titlefont\noindent \@title \ifx\@subtitle\@empty\else \par\noindent{\@subtitlefont\@subtitle} \fi}% \ifx\@acmBadgeR@image\@empty\else \hskip\@ACM@badge@skip \raisebox{-.5\baselineskip}[\z@][\z@]{\href{\@acmBadgeR@url}{% \includegraphics[width=\@ACM@badge@width]{\@acmBadgeR@image}}}% \fi \par\bigskip}}% \def\@mktitle@iii{\hsize=\textwidth \setbox\mktitle@bx=\vbox{\@titlefont\centering \@ACM@title@width=\hsize \if@ACM@badge \advance\@ACM@title@width by -2\@ACM@badge@width \advance\@ACM@title@width by -2\@ACM@badge@skip \parbox[b]{\@ACM@badge@width}{\strut \ifx\@acmBadgeL@image\@empty\else \raisebox{-.5\baselineskip}[\z@][\z@]{\href{\@acmBadgeL@url}{% \includegraphics[width=\@ACM@badge@width]{\@acmBadgeL@image}}}% \fi}% \hskip\@ACM@badge@skip \fi \parbox[t]{\@ACM@title@width}{\centering\@titlefont \@title \ifx\@subtitle\@empty\else \par\noindent{\@subtitlefont\@subtitle} \fi }% \if@ACM@badge \hskip\@ACM@badge@skip \parbox[b]{\@ACM@badge@width}{\strut \ifx\@acmBadgeR@image\@empty\else \raisebox{-.5\baselineskip}[\z@][\z@]{\href{\@acmBadgeR@url}{% \includegraphics[width=\@ACM@badge@width]{\@acmBadgeR@image}}}% \fi}% \fi \par\bigskip}}% \def\@mktitle@iv{\hsize=\textwidth \setbox\mktitle@bx=\vbox{\raggedright\leftskip5pc\@titlefont \noindent\leavevmode\leaders\hrule height 2pt\hfill\kern0pt\par \noindent\@title \ifx\@subtitle\@empty\else \par\noindent\@subtitlefont\@subtitle \fi \par\bigskip}}% \newbox\@ACM@commabox \def\@ACM@addtoaddress#1{% \ifvmode\else \if@ACM@affiliation@obeypunctuation\else \setbox\@ACM@commabox=\hbox{, }% \unskip\cleaders\copy\@ACM@commabox\hskip\wd\@ACM@commabox \fi\fi #1} \def\streetaddress#1{\unskip\ignorespaces} \def\postcode#1{\unskip\ignorespaces} \if@ACM@journal \def\position#1{\unskip\ignorespaces} \def\institution#1{#1\ignorespaces} \def\city#1{\unskip\ignorespaces} \def\state#1{\unskip\ignorespaces} \newcommand\department[2][0]{} \def\country#1{\if@ACM@affiliation@obeypunctuation\else, \fi#1\ignorespaces} \else \def\position#1{\if@ACM@affiliation@obeypunctuation#1\else#1\par\fi}% \def\institution#1{\if@ACM@affiliation@obeypunctuation#1\else#1\par\fi}% \newcommand\department[2][0]{\if@ACM@affiliation@obeypunctuation #2\else#2\par\fi}% \let\city\@ACM@addtoaddress \let\state\@ACM@addtoaddress \let\country\@ACM@addtoaddress \fi \def\@mkauthors{\begingroup \hsize=\textwidth \ifcase\ACM@format@nr \relax % manuscript \@mkauthors@i \or % acmsmall \@mkauthors@i \or % acmlarge \@mkauthors@i \or % acmtog \@mkauthors@i \or % sigconf \@mkauthors@iii \or % siggraph \@mkauthors@iii \or % sigplan \@mkauthors@iii \or % sigchi \@mkauthors@iii \or % sigchi-a \@mkauthors@iv \fi \endgroup } \def\@authorfont{\Large\sffamily} \def\@affiliationfont{\normalsize\normalfont} \ifcase\ACM@format@nr \relax % manuscript \or % acmsmall \def\@authorfont{\large\sffamily} \def\@affiliationfont{\small\normalfont} \or % acmlarge \or % acmtog \def\@authorfont{\LARGE\sffamily} \def\@affiliationfont{\large} \or % sigconf \def\@authorfont{\LARGE} \def\@affiliationfont{\large} \or % siggraph \def\@authorfont{\normalsize\normalfont} \def\@affiliationfont{\normalsize\normalfont} \or % sigplan \def\@authorfont{\Large\normalfont} \def\@affiliationfont{\normalsize\normalfont} \or % sigchi \def\@authorfont{\bfseries} \def\@affiliationfont{\mdseries} \or % sigchi-a \def\@authorfont{\bfseries} \def\@affiliationfont{\mdseries} \fi \def\@typeset@author@line{% \andify\@currentauthors\par\noindent \@currentauthors\def\@currentauthors{}% \ifx\@currentaffiliations\@empty\else \andify\@currentaffiliations \unskip, {\@currentaffiliations}\par \fi \def\@currentaffiliations{}} \def\@mkauthors@i{% \def\@currentauthors{}% \def\@currentaffiliations{}% \global\let\and\@typeset@author@line \def\@author##1{% \ifx\@currentauthors\@empty \gdef\@currentauthors{\@authorfont\MakeTextUppercase{##1}}% \else \g@addto@macro{\@currentauthors}{\and\MakeTextUppercase{##1}}% \fi \gdef\and{}}% \def\email##1##2{}% \def\affiliation##1##2{% \def\@tempa{##2}\ifx\@tempa\@empty\else \ifx\@currentaffiliations\@empty \gdef\@currentaffiliations{% \setkeys{@ACM@affiliation@}{obeypunctuation=false}% \setkeys{@ACM@affiliation@}{##1}% \@affiliationfont##2}% \else \g@addto@macro{\@currentaffiliations}{\and \setkeys{@ACM@affiliation@}{obeypunctuation=false}% \setkeys{@ACM@affiliation@}{##1}##2}% \fi \fi \global\let\and\@typeset@author@line}% \global\setbox\mktitle@bx=\vbox{\noindent\box\mktitle@bx\par\medskip \noindent\addresses\@typeset@author@line \par\medskip}% } \newbox\author@bx \newdimen\author@bx@wd \newskip\author@bx@sep \author@bx@sep=1pc\relax \def\@typeset@author@bx{\bgroup\hsize=\author@bx@wd\def\and{\par}% \global\setbox\author@bx=\vtop{\if@ACM@sigchiamode\else\centering\fi \@authorfont\@currentauthors\par\@affiliationfont \@currentaffiliation}\egroup \box\author@bx\hspace{\author@bx@sep}% \gdef\@currentauthors{}% \gdef\@currentaffiliation{}} \def\@mkauthors@iii{% \author@bx@wd=\textwidth\relax \advance\author@bx@wd by -\author@bx@sep\relax \ifnum\@ACM@authorsperrow>0\relax \divide\author@bx@wd by \@ACM@authorsperrow\relax \else \ifcase\num@authorgroups \relax % 0? \or % 1=one author per row \or % 2=two authors per row \divide\author@bx@wd by \num@authorgroups\relax \or % 3=three authors per row \divide\author@bx@wd by \num@authorgroups\relax \or % 4=two authors per row (!) \divide\author@bx@wd by 2\relax \else % three authors per row \divide\author@bx@wd by 3\relax \fi \fi \advance\author@bx@wd by -\author@bx@sep\relax \gdef\@currentauthors{}% \gdef\@currentaffiliation{}% \def\@author##1{\ifx\@currentauthors\@empty \gdef\@currentauthors{\par##1}% \else \g@addto@macro\@currentauthors{\par##1}% \fi \gdef\and{}}% \def\email##1##2{\ifx\@currentaffiliation\@empty \gdef\@currentaffiliation{\nolinkurl{##2}}% \else \g@addto@macro\@currentaffiliation{\par\nolinkurl{##2}}% \fi}% \def\affiliation##1##2{\ifx\@currentaffiliation\@empty \gdef\@currentaffiliation{% \setkeys{@ACM@affiliation@}{obeypunctuation=false}% \setkeys{@ACM@affiliation@}{##1}##2}% \else \g@addto@macro\@currentaffiliation{\par \setkeys{@ACM@affiliation@}{obeypunctuation=false}% \setkeys{@ACM@affiliation@}{##1}##2}% \fi \global\let\and\@typeset@author@bx }% \hsize=\textwidth \global\setbox\mktitle@bx=\vbox{\noindent \box\mktitle@bx\par\medskip\leavevmode \lineskip=1pc\relax\centering\hspace{-1em}% \addresses\let\and\@typeset@author@bx\and\par\bigskip}} \def\@mkauthors@iv{% \author@bx@wd=\columnwidth\relax \advance\author@bx@wd by -\author@bx@sep\relax \ifnum\@ACM@authorsperrow>0\relax \divide\author@bx@wd by \@ACM@authorsperrow\relax \else \ifcase\num@authorgroups \relax % 0? \or % 1=one author per row \else % 2=two authors per row \divide\author@bx@wd by 2\relax \fi \fi \advance\author@bx@wd by -\author@bx@sep\relax \gdef\@currentauthors{}% \gdef\@currentaffiliation{}% \def\@author##1{\ifx\@currentauthors\@empty \gdef\@currentauthors{\par##1}% \else \g@addto@macro\@currentauthors{\par##1}% \fi \gdef\and{}}% \def\email##1##2{\ifx\@currentaffiliation\@empty \gdef\@currentaffiliation{\nolinkurl{##2}}% \else \g@addto@macro\@currentaffiliation{\par\nolinkurl{##2}}% \fi}% \def\affiliation##1##2{\ifx\@currentaffiliation\@empty \gdef\@currentaffiliation{% \setkeys{@ACM@affiliation@}{obeypunctuation=false}% \setkeys{@ACM@affiliation@}{##1}##2}% \else \g@addto@macro\@currentaffiliation{\par \setkeys{@ACM@affiliation@}{obeypunctuation=false}% \setkeys{@ACM@affiliation@}{##1}##2}% \fi \global\let\and\@typeset@author@bx}% \bgroup\hsize=\columnwidth \par\raggedright\leftskip=\z@ \lineskip=1pc\noindent \addresses\let\and\@typeset@author@bx\and\par\bigskip\egroup} \def\@authornotemark{\g@addto@macro\@currentauthors{\footnotemark\relax}} \def\@@authornotemark#1{\g@addto@macro\@currentauthors{\footnotemark[#1]}} \def\@mkteasers{% \ifx\@teaserfigures\@empty\else \def\@teaser##1{\par\bigskip\bgroup \captionsetup{type=figure}##1\egroup\par} \global\setbox\mktitle@bx=\vbox{\noindent\box\mktitle@bx\par \noindent\@teaserfigures\par\medskip}% \fi} \def\@setaddresses{} \def\@mkabstract{\bgroup \ifx\@abstract\@lempty\else {\phantomsection\addcontentsline{toc}{section}{Abstract}% \if@ACM@journal \small\noindent \else \section{Abstract}% \fi \ignorespaces\@abstract\par}% \fi\egroup} \def\@mkbibcitation{\bgroup \def\footnotemark{}% \def\\{\unskip{} \ignorespaces}% \def\footnote{\ClassError{\@classname}{Please do note use footnotes inside \string\title{} or \string\author{} command! Use \string\titlenote{} or \string\authornote{} instead!}}% \par\medskip\small\noindent{\bfseries ACM Reference Format:}\par\nobreak \noindent\authors. \@acmYear. \@title. \if@ACM@journal \textit{\@journalNameShort} \@acmVolume, \@acmNumber, Article~\@acmArticle\ (\@acmPubDate), \ref{TotPages}~pages. \else In \textit{Proceedings of \acmConference@name, \acmConference@venue, \acmConference@date \ifx\acmConference@name\acmConference@shortname\else \ (\acmConference@shortname)\fi ,} \ref{TotPages}~pages. \fi\par \noindent\@formatdoi{\@acmDOI} \par\egroup} \def\@printendtopmatter{\par\medskip \ifcase\ACM@format@nr \relax % manuscript \noindent\hrulefill\par\medskip \or % acmsmall \noindent\hrulefill\par\medskip \or % acmlarge \noindent\hrulefill\par\medskip \or % acmtog \par\bigskip \or % sigconf \par\bigskip \or % siggraph \par\bigskip \or % sigplan \par\bigskip \or % sigchi \par\bigskip \or % sigchi-a \fi } \def\@setthanks{\long\def\thanks##1{\par##1\@addpunct.}\thankses} \RequirePackage{fancyhdr} \if@ACM@review \newsavebox{\ACM@linecount@bx} \newlength\ACM@linecount@bxht \newcount\ACM@linecount \ACM@linecount\@ne\relax \def\ACM@mk@linecount{% \savebox{\ACM@linecount@bx}[4em][t]{\parbox[t]{4em}{% \setlength{\ACM@linecount@bxht}{-\baselineskip}% \loop{\color{red}\scriptsize\the\ACM@linecount}\\ \global\advance\ACM@linecount by \@ne \addtolength{\ACM@linecount@bxht}{\baselineskip}% \ifdim\ACM@linecount@bxht<\textheight\repeat}}} \fi \def\ACM@linecountL{% \if@ACM@review \ACM@mk@linecount \begin{picture}(0,0)% \put(-26,-22){\usebox{\ACM@linecount@bx}}% \end{picture}% \fi} \def\ACM@linecountR{% \if@ACM@review \ACM@mk@linecount \begin{picture}(0,0)% \put(20,-22){\usebox{\ACM@linecount@bx}}% \end{picture}% \fi} \if@ACM@timestamp % Subtracting 30 from \time gives us the effect of rounding-down despite % \numexpr rounding to nearest \newcounter{ACM@time@hours} \setcounter{ACM@time@hours}{\numexpr (\time - 30) / 60 \relax} \newcounter{ACM@time@minutes} \setcounter{ACM@time@minutes}{\numexpr \time - \theACM@time@hours * 60 \relax} \newcommand\ACM@timestamp{% \footnotesize% \the\year-\two@digits{\the\month}-\two@digits{\the\day}{ }% \two@digits{\theACM@time@hours}:\two@digits{\theACM@time@minutes}{ }% page~\thepage\ (pp. \@startPage-\pageref{TotPages})% \ifx\@acmSubmissionID\@empty\relax\else ~Submission~ID: \@acmSubmissionID \fi } \fi \def\@shortauthors{\if@ACM@anonymous Anon.\else\shortauthors\fi} \def\@headfootfont{% \ifcase\ACM@format@nr \relax % manuscript \sffamily \or % acmsmall \sffamily \or % acmlarge \sffamily \or % acmtog \sffamily \or % sigconf \sffamily \or % siggraph \sffamily \or % sigplan \sffamily \or % sigchi \sffamily \or % sigchi-a \sffamily \fi} \fancypagestyle{standardpagestyle}{% \fancyhf{}% \renewcommand{\headrulewidth}{\z@}% \renewcommand{\footrulewidth}{\z@}% \ifcase\ACM@format@nr \relax % manuscript \fancyhead[LE]{\ACM@linecountL\if@ACM@printfolios\thepage\fi}% \fancyhead[RO]{\if@ACM@printfolios\thepage\fi}% \fancyhead[RE]{\@shortauthors}% \fancyhead[LO]{\ACM@linecountL\shorttitle}% \fancyfoot[RO,LE]{\footnotesize Manuscript submitted to ACM}% \or % acmsmall \fancyhead[LE]{\ACM@linecountL\@headfootfont\@acmArticle\if@ACM@printfolios:\thepage\fi}% \fancyhead[RO]{\@headfootfont\@acmArticle\if@ACM@printfolios:\thepage\fi}% \fancyhead[RE]{\@headfootfont\@shortauthors}% \fancyhead[LO]{\ACM@linecountL\@headfootfont\shorttitle}% \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No. \@acmNumber, Article \@acmArticle. Publication date: \@acmPubDate.}% \or % acmlarge \fancyhead[LE]{\ACM@linecountL\@headfootfont \@acmArticle\if@ACM@printfolios:\thepage\fi\quad\textbullet\quad\@shortauthors}% \fancyhead[LO]{\ACM@linecountL}% \fancyhead[RO]{\@headfootfont \shorttitle\quad\textbullet\quad\@acmArticle\if@ACM@printfolios:\thepage\fi}% \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No. \@acmNumber, Article \@acmArticle. Publication date: \@acmPubDate.}% \or % acmtog \fancyhead[LE]{\ACM@linecountL\@headfootfont \@acmArticle\if@ACM@printfolios:\thepage\fi\quad\textbullet\quad\@shortauthors}% \fancyhead[LO]{\ACM@linecountL}% \fancyhead[RE]{\ACM@linecountR}% \fancyhead[RO]{\@headfootfont \shorttitle\quad\textbullet\quad\@acmArticle\if@ACM@printfolios:\thepage\fi\ACM@linecountR}% \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No. \@acmNumber, Article \@acmArticle. Publication date: \@acmPubDate.}% \else % Proceedings \fancyfoot[C]{\if@ACM@printfolios\footnotesize\thepage\fi}% \fancyhead[LO]{\ACM@linecountL\@headfootfont\shorttitle}% \fancyhead[RE]{\@headfootfont\@shortauthors\ACM@linecountR}% \fancyhead[LE]{\ACM@linecountL\@headfootfont\acmConference@shortname, \acmConference@date, \acmConference@venue}% \fancyhead[RO]{\@headfootfont\acmConference@shortname, \acmConference@date, \acmConference@venue\ACM@linecountR}% \fi \if@ACM@sigchiamode \fancyheadoffset[L]{\dimexpr(\marginparsep+\marginparwidth)}% \fi \if@ACM@timestamp \fancyfoot[LO,RE]{\ACM@timestamp} \fi } \pagestyle{standardpagestyle} \newdimen\@folio@wd \@folio@wd=\z@ \newdimen\@folio@ht \@folio@ht=\z@ \newdimen\@folio@voffset \@folio@voffset=\z@ \def\@folio@max{1} \ifcase\ACM@format@nr \relax % manuscript \or % acmsmall \@folio@wd=45.75pt\relax \@folio@ht=1.25in\relax \@folio@voffset=.2in\relax \def\@folio@max{8} \or % acmlarge \@folio@wd=43.25pt\relax \@folio@ht=79pt\relax \@folio@voffset=.55in\relax \def\@folio@max{10} \fi \def\@folioblob{\@tempcnta=\@acmArticleSeq\relax \loop \ifnum\@tempcnta>\@folio@max\relax \advance\@tempcnta by - \@folio@max \repeat \advance\@tempcnta by -1\relax \@tempdima=\@folio@ht\relax \multiply\@tempdima by \the\@tempcnta\relax \advance\@tempdima by -\@folio@voffset\relax \begin{picture}(0,0) \makebox[\z@]{\raisebox{-\@tempdima}{% \rlap{% \raisebox{-0.45\@folio@ht}[\z@][\z@]{% \rule{\@folio@wd}{\@folio@ht}}}% \parbox{\@folio@wd}{% \centering \textcolor{white}{\LARGE\bfseries\sffamily\@acmArticle}}}} \end{picture}} \fancypagestyle{firstpagestyle}{% \fancyhf{}% \renewcommand{\headrulewidth}{\z@}% \renewcommand{\footrulewidth}{\z@}% \ifcase\ACM@format@nr \relax % manuscript \fancyhead[L]{\ACM@linecountL}% \fancyfoot[RO,LE]{\if@ACM@printfolios\small\thepage\fi}% \fancyfoot[RE,LO]{\footnotesize Manuscript submitted to ACM}% \or % acmsmall \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No. \@acmNumber, Article \@acmArticle. Publication date: \@acmPubDate.}% \fancyhead[LE]{\ACM@linecountL\@folioblob}% \fancyhead[LO]{\ACM@linecountL}% \fancyhead[RO]{\@folioblob}% \fancyheadoffset[RO,LE]{0.6\@folio@wd}% \or % acmlarge \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No. \@acmNumber, Article \@acmArticle. Publication date: \@acmPubDate.}% \fancyhead[RO]{\@folioblob}% \fancyhead[LE]{\ACM@linecountL\@folioblob}% \fancyhead[LO]{\ACM@linecountL}% \fancyheadoffset[RO,LE]{1.4\@folio@wd}% \or % acmtog \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No. \@acmNumber, Article \@acmArticle. Publication date: \@acmPubDate.}% \fancyhead[L]{\ACM@linecountL}% \fancyhead[R]{\ACM@linecountR}% \else % Conference proceedings \fancyhead[L]{\ACM@linecountL}% \fancyhead[R]{\ACM@linecountR}% \fancyfoot[C]{\if@ACM@printfolios\footnotesize\thepage\fi}% \fi \if@ACM@timestamp \ifnum\ACM@format@nr=0\relax % Manuscript \fancyfoot[LO,RE]{\ACM@timestamp\quad \footnotesize Manuscript submitted to ACM} \else \fancyfoot[LO,RE]{\ACM@timestamp} \fi \fi } \renewcommand\section{\@startsection{section}{1}{\z@}% {-.75\baselineskip \@plus -2\p@ \@minus -.2\p@}% {.25\baselineskip}% {\@secfont}} \renewcommand\subsection{\@startsection{subsection}{2}{\z@}% {-.75\baselineskip \@plus -2\p@ \@minus -.2\p@}% {.25\baselineskip}% {\@subsecfont}} \renewcommand\subsubsection{\@startsection{subsubsection}{3}{10pt}% {-.5\baselineskip \@plus -2\p@ \@minus -.2\p@}% {-3.5\p@}% {\@subsubsecfont\@adddotafter}} \renewcommand\paragraph{\@startsection{paragraph}{4}{\parindent}% {-.5\baselineskip \@plus -2\p@ \@minus -.2\p@}% {-3.5\p@}% {\@parfont\@adddotafter}} \renewcommand\part{\@startsection{part}{9}{\z@}% {-10\p@ \@plus -4\p@ \@minus -2\p@}% {4\p@}% {\@parfont}} \def\section@raggedright{\@rightskip\@flushglue \rightskip\@rightskip \leftskip\z@skip \parindent\z@} \def\@secfont{\sffamily\bfseries\section@raggedright\MakeTextUppercase} \def\@subsecfont{\sffamily\bfseries\section@raggedright} \def\@subsubsecfont{\sffamily\itshape} \def\@parfont{\itshape} \setcounter{secnumdepth}{3} \ifcase\ACM@format@nr \relax % manuscript \or % acmsmall \or % acmlarge \def\@secfont{\sffamily\large\section@raggedright\MakeTextUppercase} \def\@subsecfont{\sffamily\large\section@raggedright} \or % acmtog \def\@secfont{\sffamily\large\section@raggedright\MakeTextUppercase} \def\@subsecfont{\sffamily\large\section@raggedright} \or % sigconf \def\@secfont{\bfseries\Large\section@raggedright\MakeTextUppercase} \def\@subsecfont{\bfseries\Large\section@raggedright} \or % siggraph \def\@secfont{\bfseries\sffamily\Large\section@raggedright\MakeTextUppercase} \def\@subsecfont{\bfseries\sffamily\Large\section@raggedright} \or % sigplan \def\@secfont{\bfseries\Large\section@raggedright} \def\@subsecfont{\bfseries\section@raggedright} \renewcommand\subsubsection{\@startsection{subsubsection}{3}{\z@}% {-.75\baselineskip \@plus -2\p@ \@minus -.2\p@}% {.25\baselineskip}% {\@subsubsecfont}} \def\@subsubsecfont{\bfseries\section@raggedright} \renewcommand\paragraph{\@startsection{paragraph}{4}{\z@}% {-.5\baselineskip \@plus -2\p@ \@minus -.2\p@}% {-3.5\p@}% {\@parfont\@addspaceafter}} \def\@parfont{\bfseries\itshape} \renewcommand\subparagraph{\@startsection{subparagraph}{5}{\z@}% {-.5\baselineskip \@plus -2\p@ \@minus -.2\p@}% {-3.5\p@}% {\@subparfont\@addspaceafter}} \def\@subparfont{\itshape} \or % sigchi \setcounter{secnumdepth}{1} \def\@secfont{\bfseries\sffamily\section@raggedright\MakeTextUppercase} \def\@subsecfont{\bfseries\sffamily\section@raggedright} \or % sigchi-a \setcounter{secnumdepth}{0} \def\@secfont{\bfseries\sffamily\section@raggedright\MakeTextUppercase} \def\@subsecfont{\bfseries\sffamily\section@raggedright} \fi \def\@adddotafter#1{#1\@addpunct{.}} \def\@addspaceafter#1{#1\@addpunct{\enspace}} \providecommand\@dotsep{4.5} \def\@acmplainbodyfont{\itshape} \def\@acmplainindent{\parindent} \def\@acmplainheadfont{\scshape} \def\@acmplainnotefont{\@empty} \ifcase\ACM@format@nr \relax % manuscript \or % acmsmall \or % acmlarge \or % acmtog \or % sigconf \or % siggraph \or % sigplan \def\@acmplainbodyfont{\itshape} \def\@acmplainindent{\z@} \def\@acmplainheadfont{\bfseries} \def\@acmplainnotefont{\normalfont} \or % sigchi \or % sigchi-a \fi \newtheoremstyle{acmplain}% {.5\baselineskip\@plus.2\baselineskip \@minus.2\baselineskip}% space above {.5\baselineskip\@plus.2\baselineskip \@minus.2\baselineskip}% space below {\@acmplainbodyfont}% body font {\@acmplainindent}% indent amount {\@acmplainheadfont}% head font {.}% punctuation after head {.5em}% spacing after head {\thmname{#1}\thmnumber{ #2}\thmnote{ {\@acmplainnotefont(#3)}}}% head spec \def\@acmdefinitionbodyfont{\normalfont} \def\@acmdefinitionindent{\parindent} \def\@acmdefinitionheadfont{\itshape} \def\@acmdefinitionnotefont{\@empty} \ifcase\ACM@format@nr \relax % manuscript \or % acmsmall \or % acmlarge \or % acmtog \or % sigconf \or % siggraph \or % sigplan \def\@acmdefinitionbodyfont{\normalfont} \def\@acmdefinitionindent{\z@} \def\@acmdefinitionheadfont{\bfseries} \def\@acmdefinitionnotefont{\normalfont} \or % sigchi \or % sigchi-a \fi \newtheoremstyle{acmdefinition}% {.5\baselineskip\@plus.2\baselineskip \@minus.2\baselineskip}% space above {.5\baselineskip\@plus.2\baselineskip \@minus.2\baselineskip}% space below {\@acmdefinitionbodyfont}% body font {\@acmdefinitionindent}% indent amount {\@acmdefinitionheadfont}% head font {.}% punctuation after head {.5em}% spacing after head {\thmname{#1}\thmnumber{ #2}\thmnote{ {\@acmdefinitionnotefont(#3)}}}% head spec \theoremstyle{acmplain} \newtheorem{theorem}{Theorem}[section] \newtheorem{conjecture}[theorem]{Conjecture} \newtheorem{proposition}[theorem]{Proposition} \newtheorem{lemma}[theorem]{Lemma} \newtheorem{corollary}[theorem]{Corollary} \theoremstyle{acmdefinition} \newtheorem{example}[theorem]{Example} \newtheorem{definition}[theorem]{Definition} \theoremstyle{acmplain} \def\@proofnamefont{\scshape} \def\@proofindent{\indent} \ifcase\ACM@format@nr \relax % manuscript \or % acmsmall \or % acmlarge \or % acmtog \or % sigconf \or % siggraph \or % sigplan \def\@proofnamefont{\itshape} \def\@proofindent{\noindent} \or % sigchi \or % sigchi-a \fi \renewenvironment{proof}[1][\proofname]{\par \pushQED{\qed}% \normalfont \topsep6\p@\@plus6\p@\relax \trivlist \item[\@proofindent\hskip\labelsep {\@proofnamefont #1\@addpunct{.}}]\ignorespaces }{% \popQED\endtrivlist\@endpefalse } \specialcomment{acks}{% \begingroup \section*{Acknowledgments} \phantomsection\addcontentsline{toc}{section}{Acknowledgments} }{% \endgroup } \def\grantsponsor#1#2#3{#2} \newcommand\grantnum[3][]{#3% \def\@tempa{#1}\ifx\@tempa\@empty\else\space(\url{#1})\fi} \if@ACM@screen \includecomment{screenonly} \excludecomment{printonly} \else \excludecomment{screenonly} \includecomment{printonly} \fi \if@ACM@anonymous \excludecomment{anonsuppress} \excludecomment{acks} \else \includecomment{anonsuppress} \fi \newcommand\showeprint[2][arxiv]{% \def\@tempa{#1}% \ifx\@tempa\@empty\def\@tempa{arxiv}\fi \def\@tempb{arxiv}% \ifx\@tempa\@tempb arXiv:\href{http://arxiv.org/abs/#2}{#2}\else arXiv:#2% \fi} \normalsize\normalfont\frenchspacing \endinput %% %% End of file `acmart.cls'.

Ignore Space Show notes View

Koli_2017/acmart.cls 0 → 100644

%%
%% This is file `acmart.cls',
%% generated with the docstrip utility.
%%
%% The original source files were:
%%
%% acmart.dtx  (with options: `class')
%% 
%% IMPORTANT NOTICE:
%% 
%% For the copyright see the source file.
%% 
%% Any modified versions of this file must be renamed
%% with new filenames distinct from acmart.cls.
%% 
%% For distribution of the original source see the terms
%% for copying and modification in the file acmart.dtx.
%% 
%% This generated file may be distributed as long as the
%% original source files, as listed above, are part of the
%% same distribution. (The sources need not necessarily be
%% in the same archive or directory.)
%% \CharacterTable
%%  {Upper-case    \A\B\C\D\E\F\G\H\I\J\K\L\M\N\O\P\Q\R\S\T\U\V\W\X\Y\Z
%%   Lower-case    \a\b\c\d\e\f\g\h\i\j\k\l\m\n\o\p\q\r\s\t\u\v\w\x\y\z
%%   Digits        \0\1\2\3\4\5\6\7\8\9
%%   Exclamation   \!     Double quote  \"     Hash (number) \#
%%   Dollar        \$     Percent       \%     Ampersand     \&
%%   Acute accent  \'     Left paren    $     Right paren   $
%%   Asterisk      \*     Plus          \+     Comma         \,
%%   Minus         \-     Point         \.     Solidus       \/
%%   Colon         \:     Semicolon     \;     Less than     \<
%%   Equals        \=     Greater than  \>     Question mark \?
%%   Commercial at \@     Left bracket  \[     Backslash     \\
%%   Right bracket \]     Circumflex    \^     Underscore    \_
%%   Grave accent  \`     Left brace    \{     Vertical bar  \|
%%   Right brace   \}     Tilde         \~}
\NeedsTeXFormat{LaTeX2e}
\ProvidesClass{acmart}
[2017/07/09 v1.43 Typesetting articles for Association of
Computing Machinery]
\def\@classname{acmart}
\InputIfFileExists{acmart-preload-hook.tex}{%
  \ClassWarning{\@classname}{%
    I am loading acmart-preload-hook.tex. You are fully responsible
    for any problems from now on.}}{}
\RequirePackage{xkeyval}
\RequirePackage{xstring}
\define@choicekey*+{acmart.cls}{format}[\ACM@format\ACM@format@nr]{%
  manuscript, acmsmall, acmlarge, acmtog, sigconf, siggraph,
  sigplan, sigchi, sigchi-a}[manuscript]{}{%
  \ClassError{\@classname}{The option format must be manuscript,
    acmsmall, acmlarge, acmtog, sigconf, siggraph,
    sigplan, sigchi or sigchi-a}}
\def\@DeclareACMFormat#1{\DeclareOptionX{#1}{\setkeys{acmart.cls}{format=#1}}}
\@DeclareACMFormat{manuscript}
\@DeclareACMFormat{acmsmall}
\@DeclareACMFormat{acmlarge}
\@DeclareACMFormat{acmtog}
\@DeclareACMFormat{sigconf}
\@DeclareACMFormat{siggraph}
\@DeclareACMFormat{sigplan}
\@DeclareACMFormat{sigchi}
\@DeclareACMFormat{sigchi-a}
\ExecuteOptionsX{format}
\define@boolkey+{acmart.cls}[@ACM@]{screen}[true]{%
  \if@ACM@screen
    \PackageInfo{\@classname}{Using screen mode}%
  \else
    \PackageInfo{\@classname}{Not using screen mode}%
  \fi}{\PackageError{\@classname}{Option screen can be either true or
    false}}
\ExecuteOptionsX{screen=false}
\define@boolkey+{acmart.cls}[@ACM@]{review}[true]{%
  \if@ACM@review
    \PackageInfo{\@classname}{Using review mode}%
  \else
    \PackageInfo{\@classname}{Not using review mode}%
  \fi}{\PackageError{\@classname}{Option review can be either true or
    false}}
\ExecuteOptionsX{review=false}
\define@boolkey+{acmart.cls}[@ACM@]{authorversion}[true]{%
  \if@ACM@authorversion
    \PackageInfo{\@classname}{Using authorversion mode}%
  \else
    \PackageInfo{\@classname}{Not using authorversion mode}%
  \fi}{\PackageError{\@classname}{Option authorversion can be either true or
    false}}
\ExecuteOptionsX{authorversion=false}
\define@boolkey+{acmart.cls}[@ACM@]{natbib}[true]{%
  \if@ACM@natbib
    \PackageInfo{\@classname}{Explicitly selecting natbib mode}%
  \else
    \PackageInfo{\@classname}{Explicitly deselecting natbib mode}%
  \fi}{\PackageError{\@classname}{Option natbib can be either true or
    false}}
\ExecuteOptionsX{natbib=true}
\define@boolkey+{acmart.cls}[@ACM@]{anonymous}[true]{%
  \if@ACM@anonymous
    \PackageInfo{\@classname}{Using anonymous mode}%
  \else
    \PackageInfo{\@classname}{Not using anonymous mode}%
  \fi}{\PackageError{\@classname}{Option anonymous can be either true or
    false}}
\ExecuteOptionsX{anonymous=false}
\define@boolkey+{acmart.cls}[@ACM@]{timestamp}[true]{%
  \if@ACM@timestamp
    \PackageInfo{\@classname}{Using timestamp mode}%
  \else
    \PackageInfo{\@classname}{Not using timestamp mode}%
  \fi}{\PackageError{\@classname}{Option timestamp can be either true or
    false}}
\ExecuteOptionsX{timestamp=false}
\define@boolkey+{acmart.cls}[@ACM@]{authordraft}[true]{%
  \if@ACM@authordraft
    \PackageInfo{\@classname}{Using authordraft mode}%
    \@ACM@timestamptrue
    \@ACM@reviewtrue
  \else
    \PackageInfo{\@classname}{Not using authordraft mode}%
  \fi}{\PackageError{\@classname}{Option authordraft can be either true or
    false}}
\ExecuteOptionsX{authordraft=false}
\def\ACM@fontsize{}
\DeclareOptionX{9pt}{\edef\ACM@fontsize{\CurrentOption}}
\DeclareOptionX{10pt}{\edef\ACM@fontsize{\CurrentOption}}
\DeclareOptionX{11pt}{\edef\ACM@fontsize{\CurrentOption}}
\DeclareOptionX{12pt}{\edef\ACM@fontsize{\CurrentOption}}
\DeclareOptionX{draft}{\PassOptionsToClass{\CurrentOption}{amsart}}
\DeclareOptionX{*}{\PassOptionsToClass{\CurrentOption}{amsart}}
\ProcessOptionsX
\ClassInfo{\@classname}{Using format \ACM@format, number \ACM@format@nr}
\newif\if@ACM@manuscript
\newif\if@ACM@journal
\newif\if@ACM@sigchiamode
\ifnum\ACM@format@nr=5\relax % siggraph
  \ClassWarning{\@classname}{The format `siggraph' is now obsolete.
    I am switching to sigconf.}
  \setkeys{acmart.cls}{format=sigconf}
\fi
\ifnum\ACM@format@nr=0\relax
  \@ACM@manuscripttrue
\else
  \@ACM@manuscriptfalse
\fi
\@ACM@sigchiamodefalse
\ifcase\ACM@format@nr
\relax % manuscript
  \@ACM@journaltrue
\or % acmsmall
  \@ACM@journaltrue
\or % acmlarge
  \@ACM@journaltrue
\or % acmtog
  \@ACM@journaltrue
\or % sigconf
  \@ACM@journalfalse
\or % siggraph
  \@ACM@journalfalse
 \or % sigplan
  \@ACM@journalfalse
 \or % sigchi
  \@ACM@journalfalse
\or % sigchi-a
  \@ACM@journalfalse
  \@ACM@sigchiamodetrue
\fi
\ifx\ACM@fontsize\@empty
  \ifcase\ACM@format@nr
  \relax % manuscript
    \def\ACM@fontsize{9pt}%
  \or % acmsmall
    \def\ACM@fontsize{10pt}%
  \or % acmlarge
    \def\ACM@fontsize{10pt}%
  \or % acmtog
    \def\ACM@fontsize{9pt}%
  \or % sigconf
    \def\ACM@fontsize{9pt}%
  \or % siggraph
    \def\ACM@fontsize{9pt}%
   \or % sigplan
    \def\ACM@fontsize{10pt}%
   \or % sigchi
    \def\ACM@fontsize{10pt}%
  \or % sigchi-a
    \def\ACM@fontsize{10pt}%
  \fi
\fi
\ClassInfo{\@classname}{Using fontsize \ACM@fontsize}
\LoadClass[\ACM@fontsize, reqno]{amsart}
\RequirePackage{microtype}
\RequirePackage{totpages}
\RequirePackage{environ}
\if@ACM@manuscript
\RequirePackage{setspace}
\onehalfspacing
\fi
\RequirePackage{textcase}
\if@ACM@natbib
  \RequirePackage{natbib}
  \renewcommand{\bibsection}{%
     \section*{\refname}%
     \phantomsection\addcontentsline{toc}{section}{\refname}%
  }
  \renewcommand{\bibfont}{\bibliofont}
  \renewcommand\setcitestyle[1]{
  \@for\@tempa:=#1\do
  {\def\@tempb{round}\ifx\@tempa\@tempb
     \renewcommand\NAT@open{(}\renewcommand\NAT@close{)}\fi
   \def\@tempb{square}\ifx\@tempa\@tempb
     \renewcommand\NAT@open{[}\renewcommand\NAT@close{]}\fi
   \def\@tempb{angle}\ifx\@tempa\@tempb
     \renewcommand\NAT@open{$<$}\renewcommand\NAT@close{$>$}\fi
   \def\@tempb{curly}\ifx\@tempa\@tempb
     \renewcommand\NAT@open{\{}\renewcommand\NAT@close{\}}\fi
   \def\@tempb{semicolon}\ifx\@tempa\@tempb
     \renewcommand\NAT@sep{;}\fi
   \def\@tempb{colon}\ifx\@tempa\@tempb
     \renewcommand\NAT@sep{;}\fi
   \def\@tempb{comma}\ifx\@tempa\@tempb
     \renewcommand\NAT@sep{,}\fi
   \def\@tempb{authoryear}\ifx\@tempa\@tempb
     \NAT@numbersfalse\fi
   \def\@tempb{numbers}\ifx\@tempa\@tempb
     \NAT@numberstrue\NAT@superfalse\fi
   \def\@tempb{super}\ifx\@tempa\@tempb
     \NAT@numberstrue\NAT@supertrue\fi
   \def\@tempb{nobibstyle}\ifx\@tempa\@tempb
     \let\bibstyle=\@gobble\fi
   \def\@tempb{bibstyle}\ifx\@tempa\@tempb
     \let\bibstyle=\@citestyle\fi
   \def\@tempb{sort}\ifx\@tempa\@tempb
     \def\NAT@sort{\@ne}\fi
   \def\@tempb{nosort}\ifx\@tempa\@tempb
     \def\NAT@sort{\z@}\fi
   \def\@tempb{compress}\ifx\@tempa\@tempb
     \def\NAT@cmprs{\@ne}\fi
   \def\@tempb{nocompress}\ifx\@tempa\@tempb
     \def\NAT@cmprs{\@z}\fi
   \def\@tempb{sort&compress}\ifx\@tempa\@tempb
     \def\NAT@sort{\@ne}\def\NAT@cmprs{\@ne}\fi
   \def\@tempb{mcite}\ifx\@tempa\@tempb
     \let\NAT@merge\@ne\fi
   \def\@tempb{merge}\ifx\@tempa\@tempb
     \@ifnum{\NAT@merge<\tw@}{\let\NAT@merge\tw@}{}\fi
   \def\@tempb{elide}\ifx\@tempa\@tempb
     \@ifnum{\NAT@merge<\thr@@}{\let\NAT@merge\thr@@}{}\fi
   \def\@tempb{longnamesfirst}\ifx\@tempa\@tempb
     \NAT@longnamestrue\fi
   \def\@tempb{nonamebreak}\ifx\@tempa\@tempb
     \def\NAT@nmfmt#1{\mbox{\NAT@up#1}}\fi
   \expandafter\NAT@find@eq\@tempa=\relax\@nil
   \if\@tempc\relax\else
     \expandafter\NAT@rem@eq\@tempc
     \def\@tempb{open}\ifx\@tempa\@tempb
      \xdef\NAT@open{\@tempc}\fi
     \def\@tempb{close}\ifx\@tempa\@tempb
      \xdef\NAT@close{\@tempc}\fi
     \def\@tempb{aysep}\ifx\@tempa\@tempb
      \xdef\NAT@aysep{\@tempc}\fi
     \def\@tempb{yysep}\ifx\@tempa\@tempb
      \xdef\NAT@yrsep{\@tempc}\fi
     \def\@tempb{notesep}\ifx\@tempa\@tempb
      \xdef\NAT@cmt{\@tempc}\fi
     \def\@tempb{citesep}\ifx\@tempa\@tempb
      \xdef\NAT@sep{\@tempc}\fi
   \fi
  }%
  \NAT@@setcites
  }
  \renewcommand\citestyle[1]{%
    \ifcsname bibstyle@#1\endcsname%
    \csname bibstyle@#1\endcsname\let\bibstyle\@gobble%
    \else%
    \@latex@error{Undefined `#1' citestyle}%
    \fi
  }%
\fi
\newcommand{\bibstyle@acmauthoryear}{%
  \setcitestyle{%
    authoryear,%
    open={[},close={]},citesep={;},%
    aysep={},yysep={,},%
    notesep={, }}}
\newcommand{\bibstyle@acmnumeric}{%
  \setcitestyle{%
    numbers,sort&compress,%
    open={[},close={]},citesep={,},%
    notesep={, }}}
\if@ACM@natbib
\citestyle{acmnumeric}
\fi
\def\@startsection#1#2#3#4#5#6{%
  \if@noskipsec \leavevmode \fi
  \par
  \@tempskipa #4\relax
  \@afterindenttrue
  \ifdim \@tempskipa <\z@
    \@tempskipa -\@tempskipa \@afterindentfalse
  \fi
  \if@nobreak
    \everypar{}%
  \else
    \addpenalty\@secpenalty\addvspace\@tempskipa
  \fi
  \@ifstar
    {\@ssect{#3}{#4}{#5}{#6}}%
    {\@dblarg{\@sect{#1}{#2}{#3}{#4}{#5}{#6}}}}
\def\@sect#1#2#3#4#5#6[#7]#8{%
  \edef\@toclevel{\ifnum#2=\@m 0\else\number#2\fi}%
  \ifnum #2>\c@secnumdepth
    \let\@svsec\@empty
  \else
    \refstepcounter{#1}%
    \protected@edef\@svsec{\@seccntformat{#1}\relax}%
  \fi
  \@tempskipa #5\relax
  \ifdim \@tempskipa>\z@
    \begingroup
      #6{%
        \@hangfrom{\hskip #3\relax\@svsec}%
          \interlinepenalty \@M #8\@@par}%
    \endgroup
    \csname #1mark\endcsname{#7}%
    \ifnum #2>\c@secnumdepth \else
        \@tochangmeasure{\csname the#1\endcsname}%
    \fi
    \addcontentsline{toc}{#1}{%
      \ifnum #2>\c@secnumdepth \else
        \protect\numberline{\csname the#1\endcsname}%
      \fi
      #7}%
  \else
    \def\@svsechd{%
      #6{\hskip #3\relax
      \@svsec #8}%
      \csname #1mark\endcsname{#7}%
      \ifnum #2>\c@secnumdepth \else
        \@tochangmeasure{\csname the#1\endcsname\space}%
      \fi
      \addcontentsline{toc}{#1}{%
        \ifnum #2>\c@secnumdepth \else
          \protect\numberline{\csname the#1\endcsname}%
        \fi
        #7}}%
  \fi
  \@xsect{#5}}
\def\@xsect#1{%
  \@tempskipa #1\relax
  \ifdim \@tempskipa>\z@
    \par \nobreak
    \vskip \@tempskipa
    \@afterheading
  \else
    \@nobreakfalse
    \global\@noskipsectrue
    \everypar{%
      \if@noskipsec
        \global\@noskipsecfalse
       {\setbox\z@\lastbox}%
        \clubpenalty\@M
        \begingroup \@svsechd \endgroup
        \unskip
        \@tempskipa #1\relax
        \hskip -\@tempskipa
      \else
        \clubpenalty \@clubpenalty
        \everypar{}%
      \fi}%
  \fi
  \ignorespaces}
\def\@seccntformat#1{\csname the#1\endcsname\quad}
\def\@ssect#1#2#3#4#5{%
  \@tempskipa #3\relax
  \ifdim \@tempskipa>\z@
    \begingroup
      #4{%
        \@hangfrom{\hskip #1}%
          \interlinepenalty \@M #5\@@par}%
    \endgroup
  \else
    \def\@svsechd{#4{\hskip #1\relax #5}}%
  \fi
  \@xsect{#3}}
\def\@starttoc#1#2{\begingroup\makeatletter
  \setTrue{#1}%
  \par\removelastskip\vskip\z@skip
  \@startsection{section}\@M\z@{\linespacing\@plus\linespacing}%
    {.5\linespacing}{\centering\contentsnamefont}{#2}%
  \@input{\jobname.#1}%
  \if@filesw
    \@xp\newwrite\csname tf@#1\endcsname
    \immediate\@xp\openout\csname tf@#1\endcsname \jobname.#1\relax
  \fi
  \global\@nobreakfalse \endgroup
  \addvspace{32\p@\@plus14\p@}%
}
\def\l@section{\@tocline{1}{0pt}{1pc}{2pc}{}}
\def\l@subsection{\@tocline{2}{0pt}{1pc}{3pc}{}}
\def\l@subsubsection{\@tocline{2}{0pt}{1pc}{5pc}{}}
\let\@footnotemark@nolink\@footnotemark
\let\@footnotetext@nolink\@footnotetext
\RequirePackage[bookmarksnumbered,unicode]{hyperref}
\pdfstringdefDisableCommands{%
  \def\unskip{}%
  \def\textbullet{- }%
  \def\textrightarrow{ -> }%
  \def\footnotemark{}%
}
\urlstyle{rm}
\ifcase\ACM@format@nr
\relax % manuscript
\or % acmsmall
\or % acmlarge
\or % acmtog
\or % sigconf
\or % siggraph
\or % sigplan
  \urlstyle{sf}
\or % sigchi
\or % sigchi-a
  \urlstyle{sf}
\fi
\if@ACM@screen
  \hypersetup{colorlinks,
    linkcolor=ACMRed,
    citecolor=ACMPurple,
    urlcolor=ACMDarkBlue,
    filecolor=ACMDarkBlue}
\else
  \hypersetup{hidelinks}
\fi
\RequirePackage{cleveref}
\if@ACM@natbib
  \let\citeN\cite
  \let\cite\citep
  \let\citeANP\citeauthor
  \let\citeNN\citeyearpar
  \let\citeyearNP\citeyear
  \let\citeyear\citeyearpar
  \let\citeNP\citealt
  \DeclareRobustCommand\citeA
     {\begingroup\NAT@swafalse
       \let\NAT@ctype\@ne\NAT@partrue\NAT@fullfalse\NAT@open\NAT@citetp}%
  \providecommand\newblock{}%
\else
  \AtBeginDocument{%
    \let\shortcite\cite%
    \providecommand\citename[1]{#1}}
\fi
\newcommand\shortcite[2][]{%
  \ifNAT@numbers\cite[#1]{#2}\else\citeyear[#1]{#2}\fi}
\def\bibliographystyle#1{%
  \ifx\@begindocumenthook\@undefined\else
    \expandafter\AtBeginDocument
  \fi
    {\if@filesw
       \immediate\write\@auxout{\string\bibstyle{#1}}%
     \fi}}
\RequirePackage{graphicx, xcolor}
\definecolor[named]{ACMBlue}{cmyk}{1,0.1,0,0.1}
\definecolor[named]{ACMYellow}{cmyk}{0,0.16,1,0}
\definecolor[named]{ACMOrange}{cmyk}{0,0.42,1,0.01}
\definecolor[named]{ACMRed}{cmyk}{0,0.90,0.86,0}
\definecolor[named]{ACMLightBlue}{cmyk}{0.49,0.01,0,0}
\definecolor[named]{ACMGreen}{cmyk}{0.20,0,1,0.19}
\definecolor[named]{ACMPurple}{cmyk}{0.55,1,0,0.15}
\definecolor[named]{ACMDarkBlue}{cmyk}{1,0.58,0,0.21}
\if@ACM@authordraft
  \RequirePackage{draftwatermark}
  \SetWatermarkFontSize{0.5in}
  \SetWatermarkColor[gray]{.9}
  \SetWatermarkText{\parbox{12em}{\centering
      Unpublished working draft\\
      Not for distribution}}
\fi
\RequirePackage{geometry}
\ifcase\ACM@format@nr
\relax % manuscript
   \geometry{letterpaper,head=13pt,
   marginparwidth=6pc,heightrounded}%
\or % acmsmall
   \geometry{twoside=true,
     includeheadfoot, head=13pt, foot=2pc,
     paperwidth=6.75in, paperheight=10in,
     top=58pt, bottom=44pt, inner=46pt, outer=46pt,
     marginparwidth=2pc,heightrounded
   }%
\or % acmlarge
   \geometry{twoside=true, head=13pt, foot=2pc,
     paperwidth=8.5in, paperheight=11in,
     includeheadfoot,
     top=78pt, bottom=114pt, inner=81pt, outer=81pt,
     marginparwidth=4pc,heightrounded
     }%
\or % acmtog
   \geometry{twoside=true, head=13pt, foot=2pc,
     paperwidth=8.5in, paperheight=11in,
     includeheadfoot, columnsep=24pt,
     top=52pt, bottom=75pt, inner=52pt, outer=52pt,
     marginparwidth=2pc,heightrounded
     }%
\or % sigconf
   \geometry{twoside=true, head=13pt,
     paperwidth=8.5in, paperheight=11in,
     includeheadfoot, columnsep=2pc,
     top=57pt, bottom=73pt, inner=54pt, outer=54pt,
     marginparwidth=2pc,heightrounded
     }%
\or % siggraph
   \geometry{twoside=true, head=13pt,
     paperwidth=8.5in, paperheight=11in,
     includeheadfoot, columnsep=2pc,
     top=57pt, bottom=73pt, inner=54pt, outer=54pt,
     marginparwidth=2pc,heightrounded
     }%
\or % sigplan
   \geometry{twoside=true, head=13pt,
     paperwidth=8.5in, paperheight=11in,
     includeheadfoot=false, columnsep=2pc,
     top=1in, bottom=1in, inner=0.75in, outer=0.75in,
     marginparwidth=2pc,heightrounded
     }%
\or % sigchi
   \geometry{twoside=true, head=13pt,
     paperwidth=8.5in, paperheight=11in,
     includeheadfoot, columnsep=2pc,
     top=66pt, bottom=73pt, inner=54pt, outer=54pt,
     marginparwidth=2pc,heightrounded
     }%
\or % sigchi-a
   \geometry{twoside=false, head=13pt,
     paperwidth=11in, paperheight=8.5in,
     includeheadfoot, marginparsep=72pt,
     marginparwidth=170pt, columnsep=20pt,
     top=72pt, bottom=72pt, left=314pt, right=72pt
     }%
     \@mparswitchfalse
     \reversemarginpar
\fi
\setlength\parindent{10\p@}
\setlength\parskip{\z@}
\ifcase\ACM@format@nr
\relax % manuscript
\or % acmsmall
\or % acmlarge
\or % acmtog
  \setlength\parindent{9\p@}%
\or % sigconf
\or % siggraph
\or % sigplan
\or % sigchi
\or % sigchi-a
\fi
\setlength\normalparindent{\parindent}
\def\copyrightpermissionfootnoterule{\kern-3\p@
  \hrule \@width \columnwidth \kern 2.6\p@}
\RequirePackage{manyfoot}
\SelectFootnoteRule[2]{copyrightpermission}
\DeclareNewFootnote{copyrightpermission}
\def\footnoterule{\kern-3\p@
  \hrule \@width 4pc \kern 2.6\p@}
\def\endminipage{%
    \par
    \unskip
    \ifvoid\@mpfootins\else
      \vskip\skip\@mpfootins
      \normalcolor
      \unvbox\@mpfootins
    \fi
    \@minipagefalse   %% added 24 May 89
  \color@endgroup
  \egroup
  \expandafter\@iiiparbox\@mpargs{\unvbox\@tempboxa}}
\def\@makefntext{\noindent\@makefnmark}
\if@ACM@sigchiamode
\long\def\@footnotetext#1{\marginpar{%
    \reset@font\small
    \interlinepenalty\interfootnotelinepenalty
    \protected@edef\@currentlabel{%
       \csname p@footnote\endcsname\@thefnmark
    }%
    \color@begingroup
      \@makefntext{%
        \rule\z@\footnotesep\ignorespaces#1\@finalstrut\strutbox}%
    \color@endgroup}}%
\fi
\long\def\@mpfootnotetext#1{%
  \global\setbox\@mpfootins\vbox{%
    \unvbox\@mpfootins
    \reset@font\footnotesize
    \hsize\columnwidth
    \@parboxrestore
    \protected@edef\@currentlabel
         {\csname p@mpfootnote\endcsname\@thefnmark}%
    \color@begingroup\centering
      \@makefntext{%
        \rule\z@\footnotesep\ignorespaces#1\@finalstrut\strutbox}%
    \color@endgroup}}
\def\@makefnmark{\hbox{\@textsuperscript{\normalfont\@thefnmark}}}
\def\@textbottom{\vskip \z@ \@plus 1pt}
\let\@texttop\relax
\RequirePackage{iftex}
\ifPDFTeX
\input{glyphtounicode}
\pdfglyphtounicode{f_f}{FB00}
\pdfglyphtounicode{f_f_i}{FB03}
\pdfglyphtounicode{f_f_l}{FB04}
\pdfglyphtounicode{f_i}{FB01}
\pdfglyphtounicode{t_t}{00740074}
\pdfglyphtounicode{f_t}{00660074}
\pdfglyphtounicode{T_h}{00540068}
\pdfgentounicode=1
\fi
\RequirePackage{cmap}
\newif\if@ACM@newfonts
\@ACM@newfontstrue
\IfFileExists{libertine.sty}{}{\ClassWarning{\@classname}{You do not
    have libertine package installed.  Please upgrade your
    TeX}\@ACM@newfontsfalse}
\IfFileExists{zi4.sty}{}{\ClassWarning{\@classname}{You do not
    have zi4 package installed.  Please upgrade your TeX}\@ACM@newfontsfalse}
\IfFileExists{newtxmath.sty}{}{\ClassWarning{\@classname}{You do not
    have newtxmath package installed.  Please upgrade your
    TeX}\@ACM@newfontsfalse}
\if@ACM@newfonts
\RequirePackage[tt=false, type1=true]{libertine}
\RequirePackage[varqu]{zi4}
\RequirePackage[libertine]{newtxmath}
\RequirePackage[T1]{fontenc}
\fi
\if@ACM@sigchiamode
  \renewcommand{\familydefault}{\sfdefault}
\fi
\RequirePackage{caption, float}
\captionsetup[table]{position=top}
\if@ACM@journal
  \captionsetup{labelfont={sf, small},
    textfont={sf, small},  margin=\z@}
  \captionsetup[figure]{name={Fig.}}
\else
  \captionsetup{labelfont={bf},
    textfont={bf}, labelsep=colon, margin=\z@}
  \ifcase\ACM@format@nr
  \relax % manuscript
  \or % acmsmall
  \or % acmlarge
  \or % acmtog
  \or % sigconf
  \or % siggraph
    \captionsetup{textfont={it}}
  \or % sigplan
    \captionsetup{labelfont={bf},
      textfont={normalfont}, labelsep=period, margin=\z@}
  \or % sigchi
    \captionsetup[figure]{labelfont={bf, small},
      textfont={bf, small}}
  \or % sigchi-a
    \captionsetup[figure]{labelfont={bf, small},
      textfont={bf, small}}
  \fi
\fi
\newfloat{sidebar}{}{sbar}
\floatname{sidebar}{Sidebar}
\renewenvironment{sidebar}{\Collect@Body\@sidebar}{}
\long\def\@sidebar#1{\bgroup\captionsetup{type=sidebar}%
  \marginpar{\small#1}\egroup}
\newenvironment{marginfigure}{\Collect@Body\@marginfigure}{}
\long\def\@marginfigure#1{\bgroup\captionsetup{type=figure}%
  \marginpar{\centering\small#1}\egroup}
\newenvironment{margintable}{\Collect@Body\@margintable}{}
\long\def\@margintable#1{\bgroup\captionsetup{type=table}%
  \marginpar{\centering\small#1}\egroup}
\newdimen\fulltextwidth
\fulltextwidth=\dimexpr(\textwidth+\marginparwidth+\marginparsep)
\if@ACM@sigchiamode
\def\@dblfloat{\bgroup\columnwidth=\fulltextwidth
  \let\@endfloatbox\@endwidefloatbox
  \def\@fpsadddefault{\def\@fps{tp}}%
  \@float}
\fi
\if@ACM@sigchiamode
\def\end@dblfloat{%
    \end@float\egroup}
\fi
\def\@endwidefloatbox{%
  \par\vskip\z@skip
  \@minipagefalse
  \outer@nobreak
  \egroup
  \color@endbox
  \global\setbox\@currbox=\vbox{\moveleft
    \dimexpr(\fulltextwidth-\textwidth)\box\@currbox}%
  \wd\@currbox=\textwidth
}
\ifcase\ACM@format@nr
\relax % manuscript
\or % acmsmall
\or % acmlarge
\or % acmtog
\or % sigconf
\or % siggraph
\or % sigplan
\def\labelenumi{\theenumi.}
\def\labelenumii{\theenumii.}
\def\labelenumiii{\theenumiii.}
\def\labelenumiv{\theenumiv.}
\or % sigchi
\or % sigchi-a
\fi
\newdimen\@ACM@labelwidth
\AtBeginDocument{%
  \setlength\labelsep{4pt}
  \setlength{\@ACM@labelwidth}{6.5pt}

%% First-level list: when beginning after the first line of an
  %% indented paragraph or ending before an indented paragraph, labels
  %% should not hang to the left of the preceding/following text.
  \setlength\leftmargini{\z@}
  \addtolength\leftmargini{\parindent}
  \addtolength\leftmargini{2\labelsep}
  \addtolength\leftmargini{\@ACM@labelwidth}

%% Second-level and higher lists.
  \setlength\leftmarginii{\z@}
  \addtolength\leftmarginii{0.5\labelsep}
  \addtolength\leftmarginii{\@ACM@labelwidth}
  \setlength\leftmarginiii{\leftmarginii}
  \setlength\leftmarginiv{\leftmarginiii}
  \setlength\leftmarginv{\leftmarginiv}
  \setlength\leftmarginvi{\leftmarginv}
  \@listi}
\newskip\listisep
\listisep\smallskipamount
\def\@listI{\leftmargin\leftmargini
  \labelwidth\leftmargini \advance\labelwidth-\labelsep
  \listparindent\z@
  \topsep\listisep}
\let\@listi\@listI
\def\@listii{\leftmargin\leftmarginii
  \labelwidth\leftmarginii \advance\labelwidth-\labelsep
  \topsep\z@skip}
\def\@listiii{\leftmargin\leftmarginiii
  \labelwidth\leftmarginiii \advance\labelwidth-\labelsep}
\def\@listiv{\leftmargin\leftmarginiv
  \labelwidth\leftmarginiv \advance\labelwidth-\labelsep}
\def\@listv{\leftmargin\leftmarginv
  \labelwidth\leftmarginv \advance\labelwidth-\labelsep}
\def\@listvi{\leftmargin\leftmarginvi
  \labelwidth\leftmarginvi \advance\labelwidth-\labelsep}
\renewcommand{\descriptionlabel}[1]{\upshape\bfseries #1}
\renewenvironment{description}{\list{}{%
    \labelwidth\@ACM@labelwidth
    \let\makelabel\descriptionlabel}%
}{
  \endlist
}
\let\enddescription=\endlist % for efficiency
\define@choicekey*+{ACM}{acmJournal}[\@journalCode\@journalCode@nr]{%
  CIE,%
  CSUR,%
  IMWUT,%
  JACM,%
  JDIQ,%
  JEA,%
  JERIC,%
  JETC,%
  JOCCH,%
  PACMHCI,%
  PACMPL,%
  POMACS,%
  TAAS,%
  TACCESS,%
  TACO,%
  TALG,%
  TALLIP,%
  TAP,%
  TCPS,%
  TEAC,%
  TECS,%
  TIIS,%
  TISSEC,%
  TIST,%
  TKDD,%
  TMIS,%
  TOCE,%
  TOCHI,%
  TOCL,%
  TOCS,%
  TOCT,%
  TODAES,%
  TODS,%
  TOG,%
  TOIS,%
  TOIT,%
  TOMACS,%
  TOMM,%
  TOMPECS,%
  TOMS,%
  TOPC,%
  TOPS,%
  TOPLAS,%
  TOS,%
  TOSEM,%
  TOSN,%
  TRETS,%
  TSAS,%
  TSC,%
  TSLP,%
  TWEB%
}{%
\ifcase\@journalCode@nr
\relax % CIE
  \def\@journalName{ACM Computers in Entertainment}%
  \def\@journalNameShort{ACM Comput. Entertain.}%
  \def\@permissionCodeOne{1544-3574}%
\or % CSUR
  \def\@journalName{ACM Computing Surveys}%
  \def\@journalNameShort{ACM Comput. Surv.}%
  \def\@permissionCodeOne{0360-0300}%
\or % IMWUT
  \def\@journalName{Proceedings of the ACM on Interactive, Mobile,
    Wearable and Ubiquitous Technologies}%
  \def\@journalNameShort{Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.}%
  \def\@permissionCodeOne{2474-9567}%
\or % JACM
  \def\@journalName{Journal of the ACM}%
  \def\@journalNameShort{J. ACM}%
  \def\@permissionCodeOne{0004-5411}%
\or % JDIQ
  \def\@journalName{ACM Journal of Data and Information Quality}%
  \def\@journalNameShort{ACM J. Data Inform. Quality}%
  \def\@permissionCodeOne{1936-1955}%
\or % JEA
  \def\@journalName{ACM Journal of Experimental Algorithmics}%
  \def\@journalNameShort{ACM J. Exp. Algor.}%
  \def\@permissionCodeOne{1084-6654}%
\or % JERIC
  \def\@journalName{ACM Journal of Educational Resources in Computing}%
  \def\@journalNameShort{ACM J. Edu. Resources in Comput.}%
  \def\@permissionCodeOne{1073-0516}%
\or % JETC
  \def\@journalName{ACM Journal on Emerging Technologies in Computing Systems}%
  \def\@journalNameShort{ACM J. Emerg. Technol. Comput. Syst.}%
  \def\@permissionCodeOne{1550-4832}%
\or % JOCCH
  \def\@journalName{ACM Journal on Computing and Cultural Heritage}%
  \def\@journalName{ACM J. Comput. Cult. Herit.}%
\or % PACMHCI
  \def\@journalName{Proceedings of the ACM on Human-Computer Interaction}%
  \def\@journalName{Proc. ACM Hum.-Comput. Interact.}%
  \def\@permissionCodeOne{2573-0142}%
\or % PACMPL
  \def\@journalName{Proceedings of the ACM on Programming Languages}%
  \def\@journalName{Proc. ACM Program. Lang.}%
  \def\@permissionCodeOne{2475-1421}%
\or % POMACS
  \def\@journalName{Proceedings of the ACM on Measurement and Analysis of Computing Systems}%
  \def\@journalName{Proc. ACM Meas. Anal. Comput. Syst.}%
  \def\@permissionCodeOne{2476-1249}%
\or % TAAS
  \def\@journalName{ACM Transactions on Autonomous and Adaptive Systems}%
  \def\@journalNameShort{ACM Trans. Autonom. Adapt. Syst.}%
  \def\@permissionCodeOne{1556-4665}%
\or % TACCESS
  \def\@journalName{ACM Transactions on Accessible Computing}%
  \def\@journalNameShort{ACM Trans. Access. Comput.}%
  \def\@permissionCodeOne{1936-7228}%
\or % TACO
  \def\@journalName{ACM Transactions on Architecture and Code Optimization}%
\or % TALG
  \def\@journalName{ACM Transactions on Algorithms}%
  \def\@journalNameShort{ACM Trans. Algor.}%
  \def\@permissionCodeOne{1549-6325}%
\or % TALLIP
  \def\@journalName{ACM Transactions on Asian and Low-Resource Language Information Processing}%
  \def\@journalNameShort{ACM Trans. Asian Low-Resour. Lang. Inf. Process.}%
  \def\@permissionCodeOne{2375-4699}%
\or % TAP
  \def\@journalName{ACM Transactions on Applied Perception}%
\or % TCPS
  \def\@journalName{ACM Transactions on Cyber-Physical Systems}%
\or % TEAC
  \def\@journalName{ACM Transactions on Economics and Computation}%
\or % TECS
  \def\@journalName{ACM Transactions on Embedded Computing Systems}%
  \def\@journalNameShort{ACM Trans. Embedd. Comput. Syst.}%
  \def\@permissionCodeOne{1539-9087}%
\or % TIIS
  \def\@journalName{ACM Transactions on Interactive Intelligent Systems}%
  \def\@journalNameShort{ACM Trans. Interact. Intell. Syst.}%
  \def\@permissionCodeOne{2160-6455}%
\or % TISSEC
  \def\@journalName{ACM Transactions on Information and System Security}%
  \def\@journalNameShort{ACM Trans. Info. Syst. Sec.}%
  \def\@permissionCodeOne{1094-9224}%
\or % TIST
  \def\@journalName{ACM Transactions on Intelligent Systems and Technology}%
  \def\@journalNameShort{ACM Trans. Intell. Syst. Technol.}%
  \def\@permissionCodeOne{2157-6904}%
\or % TKDD
  \def\@journalName{ACM Transactions on Knowledge Discovery from Data}%
  \def\@journalNameShort{ACM Trans. Knowl. Discov. Data.}%
  \def\@permissionCodeOne{1556-4681}%
\or % TMIS
  \def\@journalName{ACM Transactions on Management Information Systems}%
  \def\@journalNameShort{ACM Trans. Manag. Inform. Syst.}%
  \def\@permissionCodeOne{2158-656X}%
\or % TOCE
  \def\@journalName{ACM Transactions on Computing Education}%
  \def\@journalNameShort{ACM Trans. Comput. Educ.}%
  \def\@permissionCodeOne{1946-6226}%
\or % TOCHI
  \def\@journalName{ACM Transactions on Computer-Human Interaction}%
  \def\@journalNameShort{ACM Trans. Comput.-Hum. Interact.}%
  \def\@permissionCodeOne{1073-0516}%
\or % TOCL
  \def\@journalName{ACM Transactions on Computational Logic}%
  \def\@journalNameShort{ACM Trans. Comput. Logic}%
  \def\@permissionCodeOne{1529-3785}%
\or % TOCS
  \def\@journalName{ACM Transactions on Computer Systems}%
  \def\@journalNameShort{ACM Trans. Comput. Syst.}%
  \def\@permissionCodeOne{0734-2071}%
\or % TOCT
  \def\@journalName{ACM Transactions on Computation Theory}%
  \def\@journalNameShort{ACM Trans. Comput. Theory}%
  \def\@permissionCodeOne{1942-3454}%
\or % TODAES
  \def\@journalName{ACM Transactions on Design Automation of Electronic Systems}%
  \def\@journalNameShort{ACM Trans. Des. Autom. Electron. Syst.}%
  \def\@permissionCodeOne{1084-4309}%
\or % TODS
  \def\@journalName{ACM Transactions on Database Systems}%
  \def\@journalNameShort{ACM Trans. Datab. Syst.}%
  \def\@permissionCodeOne{0362-5915}%
\or % TOG
  \def\@journalName{ACM Transactions on Graphics}%
  \def\@journalNameShort{ACM Trans. Graph.}%
  \def\@permissionCodeOne{0730-0301}
\or % TOIS
  \def\@journalName{ACM Transactions on Information Systems}%
  \def\@journalName{ACM Transactions on Information Systems}%
  \def\@permissionCodeOne{1046-8188}%
\or % TOIT
  \def\@journalName{ACM Transactions on Internet Technology}%
  \def\@journalNameShort{ACM Trans. Internet Technol.}%
  \def\@permissionCodeOne{1533-5399}%
\or % TOMACS
  \def\@journalName{ACM Transactions on Modeling and Computer Simulation}%
  \def\@journalName{ACM Transactions on Modeling and Computer Simulation}%
  \def\@journalNameShort{ACM Trans. Model. Comput. Simul.}%
\or % TOMM
  \def\@journalName{ACM Transactions on Multimedia Computing, Communications and Applications}%
  \def\@journalNameShort{ACM Trans. Multimedia Comput. Commun. Appl.}%
  \def\@permissionCodeOne{1551-6857}%
  \def\@permissionCodeTwo{0100}%
\or % TOMPECS
  \def\@journalName{ACM Transactions on Modeling and Performance Evaluation of Computing Systems}%
  \def\@journalNameShort{ACM Trans. Model. Perform. Eval. Comput. Syst.}%
  \def\@permissionCodeOne{2376-3639}%
\or % TOMS
  \def\@journalName{ACM Transactions on Mathematical Software}%
  \def\@journalNameShort{ACM Trans. Math. Softw.}%
  \def\@permissionCodeOne{0098-3500}%
\or % TOPC
  \def\@journalName{ACM Transactions on Parallel Computing}%
  \def\@journalNameShort{ACM Trans. Parallel Comput.}%
  \def\@permissionCodeOne{1539-9087}%
\or % TOPS
  \def\@journalName{ACM Transactions on Privacy and Security}%
  \def\@journalNameShort{ACM Trans. Priv. Sec.}%
  \def\@permissionCodeOne{2471-2566}%
\or % TOPLAS
  \def\@journalName{ACM Transactions on Programming Languages and Systems}%
  \def\@journalNameShort{ACM Trans. Program. Lang. Syst.}%
  \def\@permissionCodeOne{0164-0925}%
\or % TOS
  \def\@journalName{ACM Transactions on Storage}%
  \def\@journalNameShort{ACM Trans. Storage}%
  \def\@permissionCodeOne{1553-3077}%
\or % TOSEM
  \def\@journalName{ACM Transactions on Software Engineering and Methodology}%
  \def\@journalNameShort{ACM Trans. Softw. Eng. Methodol.}%
  \def\@permissionCodeOne{1049-331X}%
\or % TOSN
  \def\@journalName{ACM Transactions on Sensor Networks}%
  \def\@journalNameShort{ACM Trans. Sensor Netw.}%
  \def\@permissionCodeOne{1550-4859}%
\or % TRETS
  \def\@journalName{ACM Transactions on Reconfigurable Technology and Systems}%
  \def\@journalNameShort{ACM Trans. Reconfig. Technol. Syst.}%
  \def\@permissionCodeOne{1936-7406}%
\or % TSAS
  \def\@journalName{ACM Transactions on Spatial Algorithms and Systems}%
  \def\@journalNameShort{ACM Trans. Spatial Algorithms Syst.}%
  \def\@permissionCodeOne{2374-0353}%
\or % TSC
  \def\@journalName{ACM Transactions on Social Computing}%
  \def\@journalNameShort{ACM Trans. Soc. Comput.}%
  \def\@permissionCodeOne{2469-7818}%
\or % TSLP
  \def\@journalName{ACM Transactions on Speech and Language Processing}%
  \def\@journalNameShort{ACM Trans. Speech Lang. Process.}%
  \def\@permissionCodeOne{1550-4875}%
\or % TWEB
  \def\@journalName{ACM Transactions on the Web}%
  \def\@journalNameShort{ACM Trans. Web}%
  \def\@permissionCodeOne{1559-1131}%
\fi
\ClassInfo{\@classname}{Using journal code \@journalCode}%
}{%
  \ClassError{\@classname}{Incorrect journal #1}%
}%
\def\acmJournal#1{\setkeys{ACM}{acmJournal=#1}}
\def\@journalCode@nr{0}
\def\@journalName{}%
\def\@journalNameShort{\@journalName}%
\def\@permissionCodeOne{XXXX-XXXX}%
\def\@permissionCodeTwo{}%
\newcommand\acmConference[4][]{%
  \gdef\acmConference@shortname{#1}%
  \gdef\acmConference@name{#2}%
  \gdef\acmConference@date{#3}%
  \gdef\acmConference@venue{#4}%
  \ifx\acmConference@shortname\@empty
    \gdef\acmConference@shortname{#2}%
  \fi}
\acmConference[Conference'17]{ACM Conference}{July 2017}{Washington,
  DC, USA}
\def\subtitle#1{\def\@subtitle{#1}}
\subtitle{}
\newcount\num@authorgroups
\num@authorgroups=0\relax
\newif\if@insideauthorgroup
\@insideauthorgroupfalse
\renewcommand\author[2][]{%
  \if@insideauthorgroup\else
    \global\advance\num@authorgroups by 1\relax
    \global\@insideauthorgrouptrue
  \fi
  \ifx\addresses\@empty
    \if@ACM@anonymous
      \gdef\addresses{\@author{Anonymous Author(s)}}%
      \gdef\authors{Anonymous Author(s)}%
    \else
      \gdef\addresses{\@author{#2}}%
      \gdef\authors{#2}%
    \fi
  \else
    \if@ACM@anonymous\else
      \g@addto@macro\addresses{\and\@author{#2}}%
      \g@addto@macro\authors{\and#2}%
    \fi
  \fi
  \if@ACM@anonymous
    \ifx\shortauthors\@empty
      \gdef\shortauthors{Anon.}%
    \fi
  \else
    \def\@tempa{#1}%
    \ifx\@tempa\@empty
      \ifx\shortauthors\@empty
        \gdef\shortauthors{#2}%
      \else
        \g@addto@macro\shortauthors{\and#2}%
      \fi
    \else
      \ifx\shortauthors\@empty
        \gdef\shortauthors{#1}%
      \else
        \g@addto@macro\shortauthors{\and#1}%
      \fi
    \fi
  \fi}
\newcommand{\affiliation}[2][]{%
  \global\@insideauthorgroupfalse
  \if@ACM@anonymous\else
    \g@addto@macro\addresses{\affiliation{#1}{#2}}%
  \fi}
\define@boolkey+{@ACM@affiliation@}[@ACM@affiliation@]{obeypunctuation}%
[true]{}{\ClassError{\@classname}{obeypunctuation must be true or false}}
\def\additionalaffiliation#1{\authornote{\@additionalaffiliation{#1}}}
\def\@additionalaffiliation#1{\bgroup
  \def\position##1{\ignorespaces}%
  \def\institution##1{##1\ignorespaces}%
  \def\department{\@ifnextchar[{\@department}{\@department[]}}%
  \def\@department[##1]##2{\unskip, ##2\ignorespaces}%
  \let\streetaddress\position
  \let\city\position
  \let\state\position
  \let\postcode\position
  \let\country\position
  Also with #1\unskip.\egroup}
\renewcommand{\email}[2][]{%
  \if@ACM@anonymous\else
    \g@addto@macro\addresses{\email{#1}{#2}}%
  \fi}
\def\orcid#1{\unskip\ignorespaces}
\def\@titlenotes{}
\def\titlenote#1{%
  \g@addto@macro\@title{\footnotemark}%
  \if@ACM@anonymous
    \g@addto@macro\@titlenotes{%
      \stepcounter{footnote}\footnotetext{Title note}}%
  \else
    \g@addto@macro\@titlenotes{\stepcounter{footnote}\footnotetext{#1}}%
  \fi}
\def\@subtitlenotes{}
\def\subtitlenote#1{%
  \g@addto@macro\@subtitle{\footnotemark}%
  \if@ACM@anonymous
    \g@addto@macro\@subtitlenotes{%
      \stepcounter{footnote}\footnotetext{Subtitle note}}%
  \else
    \g@addto@macro\@subtitlenotes{%
      \stepcounter{footnote}\footnotetext{#1}}%
  \fi}
\def\@authornotes{}
\def\authornote#1{%
  \if@ACM@anonymous\else
    \g@addto@macro\addresses{\@authornotemark}%
    \g@addto@macro\@authornotes{%
      \stepcounter{footnote}\footnotetext{#1}}%
  \fi}
\newcommand\authornotemark[1][\relax]{%
  \ifx#1\relax\relax\relax
  \g@addto@macro\addresses{\@authornotemark}%
  \else
  \g@addto@macro\addresses{\@@authornotemark{#1}}%
  \fi}
\def\acmVolume#1{\def\@acmVolume{#1}}
\acmVolume{1}
\def\acmNumber#1{\def\@acmNumber{#1}}
\acmNumber{1}
\def\acmArticle#1{\def\@acmArticle{#1}}
\acmArticle{1}
\def\acmArticleSeq#1{\def\@acmArticleSeq{#1}}
\acmArticleSeq{\@acmArticle}
\def\acmYear#1{\def\@acmYear{#1}}
\acmYear{\the\year}
\def\acmMonth#1{\def\@acmMonth{#1}}
\acmMonth{\the\month}
\def\@acmPubDate{\ifcase\@acmMonth\or
  January\or February\or March\or April\or May\or June\or
  July\or August\or September\or October\or November\or
  December\fi~\@acmYear}
\def\acmPrice#1{\def\@acmPrice{#1}}
\acmPrice{15.00}
\def\acmSubmissionID#1{\def\@acmSubmissionID{#1}}
\acmSubmissionID{}
\def\acmISBN#1{\def\@acmISBN{#1}}
\acmISBN{978-x-xxxx-xxxx-x/YY/MM}
\def\acmDOI#1{\def\@acmDOI{#1}}
\acmDOI{10.1145/nnnnnnn.nnnnnnn}
\newif\if@ACM@badge
\@ACM@badgefalse
\newlength\@ACM@badge@width
\setlength\@ACM@badge@width{5pc}
\newlength\@ACM@title@width
\newlength\@ACM@badge@skip
\setlength\@ACM@badge@skip{1pc}
\newcommand\acmBadgeR[2][]{\@ACM@badgetrue
  \def\@acmBadgeR@url{#1}%
  \def\@acmBadgeR@image{#2}}
\def\@acmBadgeR@url{}
\def\@acmBadgeR@image{}
\newcommand\acmBadgeL[2][]{\@ACM@badgetrue
  \def\@acmBadgeL@url{#1}%
  \def\@acmBadgeL@image{#2}}
\def\@acmBadgeL@url{}
\def\@acmBadgeL@image{}
\def\startPage#1{\def\@startPage{#1}}
\startPage{}
\def\terms#1{\ClassWarning{\@classname}{The command \string\terms{} is
    obsolete.  I am going to ignore it}}
\def\keywords#1{\def\@keywords{#1}}
\keywords{}
\renewenvironment{abstract}{\Collect@Body\@saveabstract}{}
\long\def\@saveabstract#1{\long\gdef\@abstract{#1}}
\@saveabstract{}
\long\def\@lempty{}
\define@boolkey+{@ACM@topmatter@}[@ACM@]{printccs}[true]{%
  \if@ACM@printccs
    \ClassInfo{\@classname}{Printing CCS}%
  \else
    \ClassInfo{\@classname}{Suppressing CCS}%
  \fi}{\ClassError{\@classname}{printccs must be true or false}}
\define@boolkey+{@ACM@topmatter@}[@ACM@]{printacmref}[true]{%
  \if@ACM@printacmref
    \ClassInfo{\@classname}{Printing bibformat}%
  \else
    \ClassInfo{\@classname}{Suppressing bibformat}%
  \fi}{\ClassError{\@classname}{printacmref must be true or false}}
\define@boolkey+{@ACM@topmatter@}[@ACM@]{printfolios}[true]{%
  \if@ACM@printfolios
    \ClassInfo{\@classname}{Printing folios}%
  \else
    \ClassInfo{\@classname}{Suppressing folios}%
  \fi}{\ClassError{\@classname}{printfolios must be true or false}}
\define@cmdkey{@ACM@topmatter@}[@ACM@]{authorsperrow}[0]{%
  \IfInteger{#1}{\ClassInfo{\@classname}{Setting authorsperrow to
      #1}}{\ClassWarning{\@classname}{Parameter authorsperrow must be
      numerical. Ignoring the input #1}\gdef\@ACM@authorsperrow{0}}}
\def\settopmatter#1{\setkeys{@ACM@topmatter@}{#1}}
\settopmatter{printccs=true, printacmref=true}
\if@ACM@manuscript
  \settopmatter{printfolios=true}
\else
  \if@ACM@journal
    \settopmatter{printfolios=true}
  \else
    \settopmatter{printfolios=false}
  \fi
\fi
\settopmatter{authorsperrow=0}
\def\@received{}
\newcommand\received[2][]{\def\@tempa{#1}%
  \ifx\@tempa\@empty
    \ifx\@received\@empty
      \gdef\@received{Received #2}%
    \else
      \g@addto@macro{\@received}{; revised #2}%
    \fi
  \else
    \ifx\@received\@empty
      \gdef\@received{#1 #2}%
    \else
      \g@addto@macro{\@received}{; #1 #2}%
    \fi
  \fi}
\AtEndDocument{%
  \ifx\@received\@empty\else
    \par\bigskip\noindent\small\normalfont\@received\par
  \fi}
\RequirePackage{comment}
\excludecomment{CCSXML}
\let\@concepts\@empty
\newcommand\ccsdesc[2][100]{%
  \ccsdesc@parse#1~#2~~\ccsdesc@parse@end}
\RequirePackage{textcomp}
\def\ccsdesc@parse#1~#2~#3~{%
  \expandafter\ifx\csname CCS@General@#2\endcsname\relax
    \expandafter\gdef\csname CCS@General@#2\endcsname{\textbullet\
      \textbf{#2}}%
    \expandafter\gdef\csname CCS@Punctuation@#2\endcsname{; }%
    \expandafter\gdef\csname CCS@Specific@#2\endcsname{}%
  \g@addto@macro{\@concepts}{\csname CCS@General@#2\endcsname
    \csname CCS@Punctuation@#2\endcsname
    \csname CCS@Specific@#2\endcsname}%
  \fi
  \ifx#3\relax\relax\else
    \expandafter\gdef\csname CCS@Punctuation@#2\endcsname{
      \textrightarrow\ }%
    \expandafter\g@addto@macro\expandafter{\csname CCS@Specific@#2\endcsname}{%
     \ifnum#1>499\textbf{#3}; \else
     \ifnum#1>299\textit{#3}; \else
     #3; \fi\fi}%
  \fi
\ccsdesc@parse@finish}
\def\ccsdesc@parse@finish#1\ccsdesc@parse@end{}
\newif\if@printcopyright
\@printcopyrighttrue
\newif\if@printpermission
\@printpermissiontrue
\newif\if@acmowned
\@acmownedtrue
\define@choicekey*{ACM@}{acmcopyrightmode}[%
  \acm@copyrightinput\acm@copyrightmode]{none,%
    acmcopyright,acmlicensed,rightsretained,%
    usgov,usgovmixed,cagov,cagovmixed,%
    licensedusgovmixed,%
    licensedcagov,licensedcagovmixed,othergov,licensedothergov}{%
  \@printpermissiontrue
  \@printcopyrighttrue
  \@acmownedtrue
  \ifnum\acm@copyrightmode=0\relax % none
   \@printpermissionfalse
   \@printcopyrightfalse
   \@acmownedfalse
  \fi
  \ifnum\acm@copyrightmode=2\relax % acmlicensed
   \@acmownedfalse
  \fi
  \ifnum\acm@copyrightmode=3\relax % rightsretained
   \@acmownedfalse
   \acmPrice{}%
  \fi
  \ifnum\acm@copyrightmode=4\relax % usgov
   \@printpermissiontrue
   \@printcopyrightfalse
   \@acmownedfalse
   \acmPrice{}%
  \fi
  \ifnum\acm@copyrightmode=6\relax % cagov
   \@acmownedfalse
  \fi
  \ifnum\acm@copyrightmode=8\relax % licensedusgovmixed
   \@acmownedfalse
  \fi
  \ifnum\acm@copyrightmode=9\relax % licensedcagov
   \@acmownedfalse
  \fi
  \ifnum\acm@copyrightmode=10\relax % licensedcagovmixed
   \@acmownedfalse
  \fi
  \ifnum\acm@copyrightmode=11\relax % othergov
   \@acmownedtrue
  \fi
  \ifnum\acm@copyrightmode=12\relax % licensedothergov
   \@acmownedfalse
  \fi}
\def\setcopyright#1{\setkeys{ACM@}{acmcopyrightmode=#1}}
\setcopyright{acmcopyright}
\def\@copyrightowner{%
  \ifcase\acm@copyrightmode\relax % none
  \or % acmcopyright
  Association for Computing Machinery.
  \or % acmlicensed
  Copyright held by the owner/author(s). Publication rights licensed to
  Association for Computing Machinery.
  \or % rightsretained
  Copyright held by the owner/author(s).
  \or % usgov
  \or % usgovmixed
  Association for Computing Machinery.
  \or % cagov
  Crown in Right of Canada.
  \or %cagovmixed
  Association for Computing Machinery.
  \or %licensedusgovmixed
  Copyright held by the owner/author(s). Publication rights licensed to
  Association for Computing Machinery.
  \or % licensedcagov
  Crown in Right of Canada. Publication rights licensed to
  Association for Computing Machinery.
  \or %licensedcagovmixed
  Copyright held by the owner/author(s). Publication rights licensed to
  Association for Computing Machinery.
  \or % othergov
  Association for Computing Machinery.
  \or % licensedothergov
  Copyright held by the owner/author(s). Publication rights licensed to
  Association for Computing Machinery.
  \fi}
\def\@formatdoi#1{\url{https://doi.org/#1}}
\def\@copyrightpermission{%
  \ifcase\acm@copyrightmode\relax % none
  \or % acmcopyright
   Permission to make digital or hard copies of all or part of this
   work for personal or classroom use is granted without fee provided
   that copies are not made or distributed for profit or commercial
   advantage and that copies bear this notice and the full citation on
   the first page. Copyrights for components of this work owned by
   others than ACM must be honored. Abstracting with credit is
   permitted. To copy otherwise, or republish, to post on servers or to
   redistribute to lists, requires prior specific permission
   and\hspace*{.5pt}/or  a fee. Request permissions from
   permissions@acm.org.
  \or % acmlicensed
   Permission to make digital or hard copies of all or part of this
   work for personal or classroom use is granted without fee provided
   that copies are not made or distributed for profit or commercial
   advantage and that copies bear this notice and the full citation on
   the first page. Copyrights for components of this work owned by
   others than the author(s) must be honored. Abstracting with credit
   is permitted.  To copy otherwise, or republish, to post on servers
   or to  redistribute to lists, requires prior specific permission
   and\hspace*{.5pt}/or  a fee. Request permissions from
   permissions@acm.org.
  \or % rightsretained
   Permission to make digital or hard copies of part or all of this work
   for personal or classroom use is granted without fee provided that
   copies are not made or distributed for profit or commercial advantage
   and that copies bear this notice and the full citation on the first
   page. Copyrights for third-party components of this work must be
   honored. For all other uses, contact the
   owner\hspace*{.5pt}/author(s).
  \or % usgov
   This paper is authored by an employee(s) of the United States
   Government and is in the public domain. Non-exclusive copying or
   redistribution is allowed, provided that the article citation is
   given and the authors and agency are clearly identified as its
   source.
  \or % usgovmixed
   ACM acknowledges that this contribution was authored or co-authored
   by an employee, contractor, or affiliate of the United States government. As such,
   the United States government retains a nonexclusive, royalty-free right to
   publish or reproduce this article, or to allow others to do so, for
   government purposes only.
  \or % cagov
   This article was authored by employees of the Government of Canada.
   As such, the Canadian government retains all interest in the
   copyright to this work and grants to ACM a nonexclusive,
   royalty-free right to publish or reproduce this article, or to allow
   others to do so, provided that clear attribution is given both to
   the authors and the Canadian government agency employing them.
   Permission to make digital or hard copies for personal or classroom
   use is granted. Copies must bear this notice and the full citation
   on the first page.  Copyrights for components of this work owned by
   others than the Canadian Government must be honored. To copy
   otherwise, distribute, republish, or post, requires prior specific
   permission and\hspace*{.5pt}/or a fee. Request permissions from
   permissions@acm.org.
  \or % cagovmixed
   ACM acknowledges that this contribution was co-authored by an
   affiliate of the national government of Canada. As such, the Crown
   in Right of Canada retains an equal interest in the copyright.
   Reprints must include clear attribution to ACM and the author's
   government agency affiliation.  Permission to make digital or hard
   copies for personal or classroom use is granted.  Copies must bear
   this notice and the full citation on the first page. Copyrights for
   components of this work owned by others than ACM must be honored.
   To copy otherwise, distribute, republish, or post, requires prior
   specific permission and\hspace*{.5pt}/or a fee. Request permissions
   from permissions@acm.org.
  \or % licensedusgovmixed
   Publication rights licensed to ACM\@. ACM acknowledges that this
   contribution was authored or co-authored by an employee, contractor
   or affiliate of the United States government. As such, the
   Government retains a nonexclusive, royalty-free right to publish or
   reproduce this article, or to allow others to do so, for Government
   purposes only.
  \or % licensedcagov
   This article was authored by employees of the Government of Canada.
   As such, the Canadian government retains all interest in the
   copyright to this work and grants to ACM a nonexclusive,
   royalty-free right to publish or reproduce this article, or to allow
   others to do so, provided that clear attribution is given both to
   the authors and the Canadian government agency employing them.
   Permission to make digital or hard copies for personal or classroom
   use is granted. Copies must bear this notice and the full citation
   on the first page.  Copyrights for components of this work owned by
   others than the Canadian Government must be honored. To copy
   otherwise, distribute, republish, or post, requires prior specific
   permission and\hspace*{.5pt}/or a fee. Request permissions from
   permissions@acm.org.
  \or % licensedcagovmixed
   Publication rights licensed to ACM\@. ACM acknowledges that this
   contribution was authored or co-authored by an employee, contractor
   or affiliate of the national government of Canada. As such, the
   Government retains a nonexclusive, royalty-free right to publish or
   reproduce this article, or to allow others to do so, for Government
   purposes only.
  \or % othergov
   ACM acknowledges that this contribution was authored or co-authored
   by an employee, contractor or affiliate of a national government. As
   such, the Government retains a nonexclusive, royalty-free right to
   publish or reproduce this article, or to allow others to do so, for
   Government purposes only.
  \or % licensedothergov
   Publication rights licensed to ACM\@. ACM acknowledges that this
   contribution was authored or co-authored by an employee, contractor
   or affiliate of a national government. As such, the Government
   retains a nonexclusive, royalty-free right to publish or reproduce
   this article, or to allow others to do so, for Government purposes
   only.
  \fi}
\def\copyrightyear#1{\def\@copyrightyear{#1}}
\copyrightyear{\@acmYear}
\def\@teaserfigures{}
\newenvironment{teaserfigure}{\Collect@Body\@saveteaser}{}
\long\def\@saveteaser#1{\g@addto@macro\@teaserfigures{\@teaser{#1}}}
\renewcommand{\thanks}[1]{%
  \@ifnotempty{#1}{%
    \if@ACM@anonymous
      \g@addto@macro\thankses{\thanks{A note}}%
   \else
    \g@addto@macro\thankses{\thanks{#1}}%
   \fi}}
\newbox\mktitle@bx
\def\maketitle{%
  \if@ACM@anonymous
    % Anonymize omission of \author-s
    \ifnum\num@authorgroups=0\author{}\fi
  \fi
  \begingroup
  \let\@footnotemark\@footnotemark@nolink
  \let\@footnotetext\@footnotetext@nolink
  \renewcommand\thefootnote{\@fnsymbol\c@footnote}%
  \global\@topnum\z@ % this prevents floats from falling
                     % at the top of page 1
  \global\@botnum\z@ % we do not want them to be on bottom either
  \hsize=\textwidth
  \def\@makefnmark{\hbox{\@textsuperscript{\@thefnmark}}}%
  \@mktitle\if@ACM@sigchiamode\else\@mkauthors\fi\@mkteasers
  \@printtopmatter
  \if@ACM@sigchiamode\@mkauthors\fi
  \setcounter{footnote}{0}%
  \def\@makefnmark{\hbox{\@textsuperscript{\normalfont\@thefnmark}}}%
  \@titlenotes
  \@subtitlenotes
  \@authornotes
  \let\@makefnmark\relax  \let\@thefnmark\relax
  \let\@makefntext\noindent
  \ifx\@empty\thankses\else
    \footnotetextcopyrightpermission{%
      \def\par{\let\par\@par}\parindent\z@\@setthanks}%
  \fi
  \footnotetextcopyrightpermission{%
    \if@ACM@authordraft
        \raisebox{-2ex}[\z@][\z@]{\makebox[0pt][l]{\large\bfseries
            Unpublished
            working draft. Not for distribution}}%
       \color[gray]{0.9}%
    \fi
    \parindent\z@\parskip0.1\baselineskip
    \if@ACM@authorversion\else
      \if@printpermission\@copyrightpermission\par\fi
    \fi
    \if@ACM@manuscript\else
       \if@ACM@journal\else % Print the conference information
         {\itshape \acmConference@shortname, \acmConference@date, \acmConference@venue}\par
       \fi
    \fi
    \if@printcopyright
      \copyright\ \@copyrightyear\ \@copyrightowner\\
    \else
     \@copyrightyear.\
    \fi
    \if@ACM@manuscript
      Manuscript submitted to ACM\\
    \else
      \if@ACM@authorversion
          This is the author's version of the work. It is posted here for
          your personal use. Not for redistribution. The definitive Version
          of Record was published in
          \if@ACM@journal
            \emph{\@journalName}%
          \else
            \emph{Proceedings of \acmConference@name, \acmConference@date}%
          \fi
          \ifx\@acmDOI\@empty
          .
          \else
            , \@formatdoi{\@acmDOI}.
          \fi\\
        \else
          \if@ACM@journal
            \@permissionCodeOne/\@acmYear/\@acmMonth-ART\@acmArticle
            \ifx\@acmPrice\@empty\else\ \$\@acmPrice\fi\\
            \@formatdoi{\@acmDOI}%
          \else % Conference
             ACM~ISBN~\@acmISBN
             \ifx\@acmPrice\@empty.\else\dots\$\@acmPrice\fi\\
             \@formatdoi{\@acmDOI}%
          \fi
        \fi
      \fi}
  \endgroup
  \setcounter{footnote}{0}%
  \@mkabstract
  \if@ACM@printccs
    \ifx\@concepts\@empty\else\bgroup
      {\@specialsection{CCS Concepts}%
         \@concepts\par}\egroup
     \fi
   \fi
   \ifx\@keywords\@empty\else\bgroup
      {\if@ACM@journal
         \@specialsection{Additional Key Words and Phrases}%
       \else
         \@specialsection{Keywords}%
       \fi
         \@keywords}\par\egroup
   \fi
  \andify\authors
  \andify\shortauthors
  \global\let\authors=\authors
  \global\let\shortauthors=\shortauthors
  \if@ACM@printacmref
     \@mkbibcitation
  \fi
  \hypersetup{pdfauthor={\authors},
    pdftitle={\@title},
    pdfsubject={\@concepts},
    pdfkeywords={\@keywords}}%
  \@printendtopmatter
  \@afterindentfalse
  \@afterheading
}
\def\@specialsection#1{%
  \ifcase\ACM@format@nr
  \relax % manuscript
    \par\medskip\small\noindent#1: %
  \or % acmsmall
    \par\medskip\small\noindent#1: %
  \or % acmlarge
    \par\medskip\small\noindent#1: %
  \or % acmtog
    \par\medskip\small\noindent#1: %
  \or % sigconf
    \section*{#1}%
  \or % siggraph
    \section*{#1}%
  \or % sigplan
    \paragraph*{#1}%
  \or % sigchi
    \section*{#1}%
  \or % sigchi-a
    \section*{#1}%
  \fi}
\def\@printtopmatter{%
  \ifx\@startPage\@empty
     \gdef\@startPage{1}%
  \else
     \setcounter{page}{\@startPage}%
  \fi
  \thispagestyle{firstpagestyle}%
  \noindent
  \ifcase\ACM@format@nr
  \relax % manuscript
    \box\mktitle@bx\par
    \noindent\hrulefill\par
  \or % acmsmall
    \box\mktitle@bx\par
    \noindent\hrulefill\par
  \or % acmlarge
    \box\mktitle@bx\par
    \noindent\hrulefill\par
  \or % acmtog
    \twocolumn[\box\mktitle@bx]%
  \or % sigconf
    \twocolumn[\box\mktitle@bx]%
  \or % siggraph
    \twocolumn[\box\mktitle@bx]%
  \or % sigplan
    \twocolumn[\box\mktitle@bx]%
  \or % sigchi
    \twocolumn[\box\mktitle@bx]%
  \or % sigchi-a
    \par\box\mktitle@bx\par\bigskip
    \if@ACM@badge
       \marginpar{\noindent
         \ifx\@acmBadgeL@image\@empty\else
           \href{\@acmBadgeL@url}{%
             \includegraphics[width=\@ACM@badge@width]{\@acmBadgeL@image}}%
            \hskip\@ACM@badge@skip
          \fi
         \ifx\@acmBadgeR@image\@empty\else
           \href{\@acmBadgeR@url}{%
             \includegraphics[width=\@ACM@badge@width]{\@acmBadgeR@image}}%
          \fi}%
    \fi
  \fi
}
\def\@mktitle{%
  \ifcase\ACM@format@nr
  \relax % manuscript
    \@mktitle@i
  \or % acmsmall
    \@mktitle@i
  \or % acmlarge
    \@mktitle@i
  \or % acmtog
    \@mktitle@i
  \or % sigconf
    \@mktitle@iii
  \or % siggraph
    \@mktitle@iii
  \or % sigplan
    \@mktitle@iii
  \or % sigchi
    \@mktitle@iii
  \or % sigchi-a
    \@mktitle@iv
  \fi
}
\def\@titlefont{%
  \ifcase\ACM@format@nr
  \relax % manuscript
    \LARGE\bfseries\sffamily
  \or % acmsmall
    \LARGE\bfseries\sffamily
  \or % acmlarge
    \LARGE\bfseries\sffamily
  \or % acmtog
    \Huge\sffamily
  \or % sigconf
    \Huge\sffamily\bfseries
  \or % siggraph
    \Huge\sffamily\bfseries
  \or % sigplan
    \Huge\bfseries
  \or % sigchi
    \Huge\sffamily\bfseries
  \or % sigchi-a
     \Huge\bfseries
  \fi}
\def\@subtitlefont{\normalsize
  \ifcase\ACM@format@nr
  \relax % manuscript
    \mdseries
  \or % acmsmall
    \mdseries
  \or % acmlarge
    \mdseries
  \or % acmtog
     \LARGE
  \or % sigconf
     \LARGE\mdseries
  \or % siggraph
     \LARGE\mdseries
  \or % sigplan
     \LARGE\mdseries
  \or % sigchi
     \LARGE\mdseries
  \or % sigchi-a
     \mdseries
  \fi}
\def\@mktitle@i{\hsize=\textwidth
  \@ACM@title@width=\hsize
  \ifx\@acmBadgeL@image\@empty\else
    \advance\@ACM@title@width by -\@ACM@badge@width
    \advance\@ACM@title@width by -\@ACM@badge@skip
  \fi
  \ifx\@acmBadgeR@image\@empty\else
    \advance\@ACM@title@width by -\@ACM@badge@width
    \advance\@ACM@title@width by -\@ACM@badge@skip
  \fi
  \setbox\mktitle@bx=\vbox{\noindent\@titlefont
    \ifx\@acmBadgeL@image\@empty\else
      \raisebox{-.5\baselineskip}[\z@][\z@]{\href{\@acmBadgeL@url}{%
          \includegraphics[width=\@ACM@badge@width]{\@acmBadgeL@image}}}%
      \hskip\@ACM@badge@skip
    \fi
    \parbox[t]{\@ACM@title@width}{\raggedright
      \@titlefont\noindent
      \@title
  \ifx\@subtitle\@empty\else
    \par\noindent{\@subtitlefont\@subtitle}
  \fi}%
  \ifx\@acmBadgeR@image\@empty\else
    \hskip\@ACM@badge@skip
    \raisebox{-.5\baselineskip}[\z@][\z@]{\href{\@acmBadgeR@url}{%
        \includegraphics[width=\@ACM@badge@width]{\@acmBadgeR@image}}}%
  \fi
  \par\bigskip}}%
\def\@mktitle@iii{\hsize=\textwidth
    \setbox\mktitle@bx=\vbox{\@titlefont\centering
      \@ACM@title@width=\hsize
      \if@ACM@badge
        \advance\@ACM@title@width by -2\@ACM@badge@width
        \advance\@ACM@title@width by -2\@ACM@badge@skip
        \parbox[b]{\@ACM@badge@width}{\strut
          \ifx\@acmBadgeL@image\@empty\else
            \raisebox{-.5\baselineskip}[\z@][\z@]{\href{\@acmBadgeL@url}{%
                \includegraphics[width=\@ACM@badge@width]{\@acmBadgeL@image}}}%
          \fi}%
        \hskip\@ACM@badge@skip
      \fi
      \parbox[t]{\@ACM@title@width}{\centering\@titlefont
        \@title
        \ifx\@subtitle\@empty\else
          \par\noindent{\@subtitlefont\@subtitle}
        \fi
      }%
      \if@ACM@badge
        \hskip\@ACM@badge@skip
        \parbox[b]{\@ACM@badge@width}{\strut
          \ifx\@acmBadgeR@image\@empty\else
            \raisebox{-.5\baselineskip}[\z@][\z@]{\href{\@acmBadgeR@url}{%
                \includegraphics[width=\@ACM@badge@width]{\@acmBadgeR@image}}}%
          \fi}%
      \fi
      \par\bigskip}}%
\def\@mktitle@iv{\hsize=\textwidth
    \setbox\mktitle@bx=\vbox{\raggedright\leftskip5pc\@titlefont
      \noindent\leavevmode\leaders\hrule height 2pt\hfill\kern0pt\par
      \noindent\@title
     \ifx\@subtitle\@empty\else
       \par\noindent\@subtitlefont\@subtitle
     \fi
     \par\bigskip}}%
\newbox\@ACM@commabox
\def\@ACM@addtoaddress#1{%
  \ifvmode\else
    \if@ACM@affiliation@obeypunctuation\else
    \setbox\@ACM@commabox=\hbox{, }%
    \unskip\cleaders\copy\@ACM@commabox\hskip\wd\@ACM@commabox
  \fi\fi
  #1}
\def\streetaddress#1{\unskip\ignorespaces}
\def\postcode#1{\unskip\ignorespaces}
\if@ACM@journal
  \def\position#1{\unskip\ignorespaces}
  \def\institution#1{#1\ignorespaces}
  \def\city#1{\unskip\ignorespaces}
  \def\state#1{\unskip\ignorespaces}
  \newcommand\department[2][0]{}
  \def\country#1{\if@ACM@affiliation@obeypunctuation\else, \fi#1\ignorespaces}
\else
  \def\position#1{\if@ACM@affiliation@obeypunctuation#1\else#1\par\fi}%
  \def\institution#1{\if@ACM@affiliation@obeypunctuation#1\else#1\par\fi}%
  \newcommand\department[2][0]{\if@ACM@affiliation@obeypunctuation
    #2\else#2\par\fi}%
  \let\city\@ACM@addtoaddress
  \let\state\@ACM@addtoaddress
  \let\country\@ACM@addtoaddress
\fi
\def\@mkauthors{\begingroup
  \hsize=\textwidth
  \ifcase\ACM@format@nr
  \relax % manuscript
    \@mkauthors@i
  \or % acmsmall
    \@mkauthors@i
  \or % acmlarge
    \@mkauthors@i
  \or % acmtog
    \@mkauthors@i
  \or % sigconf
    \@mkauthors@iii
  \or % siggraph
    \@mkauthors@iii
  \or % sigplan
    \@mkauthors@iii
  \or % sigchi
    \@mkauthors@iii
  \or % sigchi-a
    \@mkauthors@iv
  \fi
  \endgroup
}
\def\@authorfont{\Large\sffamily}
\def\@affiliationfont{\normalsize\normalfont}
\ifcase\ACM@format@nr
\relax % manuscript
\or % acmsmall
  \def\@authorfont{\large\sffamily}
  \def\@affiliationfont{\small\normalfont}
\or % acmlarge
\or % acmtog
  \def\@authorfont{\LARGE\sffamily}
  \def\@affiliationfont{\large}
\or % sigconf
  \def\@authorfont{\LARGE}
  \def\@affiliationfont{\large}
\or % siggraph
  \def\@authorfont{\normalsize\normalfont}
  \def\@affiliationfont{\normalsize\normalfont}
\or % sigplan
  \def\@authorfont{\Large\normalfont}
  \def\@affiliationfont{\normalsize\normalfont}
\or % sigchi
  \def\@authorfont{\bfseries}
  \def\@affiliationfont{\mdseries}
\or % sigchi-a
  \def\@authorfont{\bfseries}
  \def\@affiliationfont{\mdseries}
\fi
\def\@typeset@author@line{%
  \andify\@currentauthors\par\noindent
  \@currentauthors\def\@currentauthors{}%
  \ifx\@currentaffiliations\@empty\else
    \andify\@currentaffiliations
      \unskip, {\@currentaffiliations}\par
  \fi
  \def\@currentaffiliations{}}
\def\@mkauthors@i{%
  \def\@currentauthors{}%
  \def\@currentaffiliations{}%
  \global\let\and\@typeset@author@line
  \def\@author##1{%
    \ifx\@currentauthors\@empty
      \gdef\@currentauthors{\@authorfont\MakeTextUppercase{##1}}%
    \else
       \g@addto@macro{\@currentauthors}{\and\MakeTextUppercase{##1}}%
    \fi
    \gdef\and{}}%
  \def\email##1##2{}%
  \def\affiliation##1##2{%
    \def\@tempa{##2}\ifx\@tempa\@empty\else
       \ifx\@currentaffiliations\@empty
          \gdef\@currentaffiliations{%
            \setkeys{@ACM@affiliation@}{obeypunctuation=false}%
            \setkeys{@ACM@affiliation@}{##1}%
            \@affiliationfont##2}%
       \else
         \g@addto@macro{\@currentaffiliations}{\and
           \setkeys{@ACM@affiliation@}{obeypunctuation=false}%
           \setkeys{@ACM@affiliation@}{##1}##2}%
      \fi
    \fi
     \global\let\and\@typeset@author@line}%
  \global\setbox\mktitle@bx=\vbox{\noindent\box\mktitle@bx\par\medskip
    \noindent\addresses\@typeset@author@line
   \par\medskip}%
}
\newbox\author@bx
\newdimen\author@bx@wd
\newskip\author@bx@sep
\author@bx@sep=1pc\relax
\def\@typeset@author@bx{\bgroup\hsize=\author@bx@wd\def\and{\par}%
  \global\setbox\author@bx=\vtop{\if@ACM@sigchiamode\else\centering\fi
    \@authorfont\@currentauthors\par\@affiliationfont
    \@currentaffiliation}\egroup
  \box\author@bx\hspace{\author@bx@sep}%
  \gdef\@currentauthors{}%
  \gdef\@currentaffiliation{}}
\def\@mkauthors@iii{%
  \author@bx@wd=\textwidth\relax
  \advance\author@bx@wd by -\author@bx@sep\relax
  \ifnum\@ACM@authorsperrow>0\relax
    \divide\author@bx@wd by \@ACM@authorsperrow\relax
  \else
    \ifcase\num@authorgroups
    \relax % 0?
    \or  % 1=one author per row
    \or  % 2=two authors per row
       \divide\author@bx@wd by \num@authorgroups\relax
    \or  % 3=three authors per row
       \divide\author@bx@wd by \num@authorgroups\relax
    \or  % 4=two authors per row (!)
       \divide\author@bx@wd by 2\relax
    \else % three authors per row
       \divide\author@bx@wd by 3\relax
    \fi
  \fi
  \advance\author@bx@wd by -\author@bx@sep\relax
  \gdef\@currentauthors{}%
  \gdef\@currentaffiliation{}%
  \def\@author##1{\ifx\@currentauthors\@empty
    \gdef\@currentauthors{\par##1}%
  \else
    \g@addto@macro\@currentauthors{\par##1}%
  \fi
  \gdef\and{}}%
  \def\email##1##2{\ifx\@currentaffiliation\@empty
    \gdef\@currentaffiliation{\nolinkurl{##2}}%
  \else
    \g@addto@macro\@currentaffiliation{\par\nolinkurl{##2}}%
  \fi}%
  \def\affiliation##1##2{\ifx\@currentaffiliation\@empty
    \gdef\@currentaffiliation{%
      \setkeys{@ACM@affiliation@}{obeypunctuation=false}%
      \setkeys{@ACM@affiliation@}{##1}##2}%
  \else
    \g@addto@macro\@currentaffiliation{\par
      \setkeys{@ACM@affiliation@}{obeypunctuation=false}%
      \setkeys{@ACM@affiliation@}{##1}##2}%
  \fi
  \global\let\and\@typeset@author@bx
}%
  \hsize=\textwidth
  \global\setbox\mktitle@bx=\vbox{\noindent
    \box\mktitle@bx\par\medskip\leavevmode
    \lineskip=1pc\relax\centering\hspace*{-1em}%
    \addresses\let\and\@typeset@author@bx\and\par\bigskip}}
\def\@mkauthors@iv{%
  \author@bx@wd=\columnwidth\relax
  \advance\author@bx@wd by -\author@bx@sep\relax
  \ifnum\@ACM@authorsperrow>0\relax
    \divide\author@bx@wd by \@ACM@authorsperrow\relax
  \else
    \ifcase\num@authorgroups
    \relax % 0?
    \or  % 1=one author per row
    \else  % 2=two authors per row
       \divide\author@bx@wd by 2\relax
    \fi
  \fi
  \advance\author@bx@wd by -\author@bx@sep\relax
  \gdef\@currentauthors{}%
  \gdef\@currentaffiliation{}%
  \def\@author##1{\ifx\@currentauthors\@empty
    \gdef\@currentauthors{\par##1}%
  \else
    \g@addto@macro\@currentauthors{\par##1}%
  \fi
  \gdef\and{}}%
  \def\email##1##2{\ifx\@currentaffiliation\@empty
    \gdef\@currentaffiliation{\nolinkurl{##2}}%
  \else
    \g@addto@macro\@currentaffiliation{\par\nolinkurl{##2}}%
  \fi}%
  \def\affiliation##1##2{\ifx\@currentaffiliation\@empty
    \gdef\@currentaffiliation{%
      \setkeys{@ACM@affiliation@}{obeypunctuation=false}%
           \setkeys{@ACM@affiliation@}{##1}##2}%
  \else
    \g@addto@macro\@currentaffiliation{\par
      \setkeys{@ACM@affiliation@}{obeypunctuation=false}%
      \setkeys{@ACM@affiliation@}{##1}##2}%
  \fi
  \global\let\and\@typeset@author@bx}%
    \bgroup\hsize=\columnwidth
    \par\raggedright\leftskip=\z@
    \lineskip=1pc\noindent
    \addresses\let\and\@typeset@author@bx\and\par\bigskip\egroup}
\def\@authornotemark{\g@addto@macro\@currentauthors{\footnotemark\relax}}
\def\@@authornotemark#1{\g@addto@macro\@currentauthors{\footnotemark[#1]}}
\def\@mkteasers{%
  \ifx\@teaserfigures\@empty\else
    \def\@teaser##1{\par\bigskip\bgroup
      \captionsetup{type=figure}##1\egroup\par}
    \global\setbox\mktitle@bx=\vbox{\noindent\box\mktitle@bx\par
    \noindent\@teaserfigures\par\medskip}%
  \fi}
\def\@setaddresses{}
\def\@mkabstract{\bgroup
  \ifx\@abstract\@lempty\else
  {\phantomsection\addcontentsline{toc}{section}{Abstract}%
    \if@ACM@journal
       \small\noindent
    \else
      \section*{Abstract}%
    \fi
   \ignorespaces\@abstract\par}%
  \fi\egroup}
\def\@mkbibcitation{\bgroup
  \def\footnotemark{}%
  \def\\{\unskip{} \ignorespaces}%
  \def\footnote{\ClassError{\@classname}{Please do note use footnotes
      inside \string\title{} or \string\author{} command! Use
      \string\titlenote{} or \string\authornote{} instead!}}%
  \par\medskip\small\noindent{\bfseries ACM Reference Format:}\par\nobreak
  \noindent\authors. \@acmYear. \@title.
  \if@ACM@journal
     \textit{\@journalNameShort}
     \@acmVolume, \@acmNumber, Article~\@acmArticle\ (\@acmPubDate),
     \ref{TotPages}~pages.
  \else
     In \textit{Proceedings of \acmConference@name, \acmConference@venue,
       \acmConference@date
       \ifx\acmConference@name\acmConference@shortname\else
         \ (\acmConference@shortname)\fi
         ,} \ref{TotPages}~pages.
  \fi\par
  \noindent\@formatdoi{\@acmDOI}
\par\egroup}
\def\@printendtopmatter{\par\medskip
  \ifcase\ACM@format@nr
  \relax % manuscript
    \noindent\hrulefill\par\medskip
  \or % acmsmall
    \noindent\hrulefill\par\medskip
  \or % acmlarge
    \noindent\hrulefill\par\medskip
  \or % acmtog
    \par\bigskip
  \or % sigconf
    \par\bigskip
  \or % siggraph
    \par\bigskip
  \or % sigplan
    \par\bigskip
  \or % sigchi
    \par\bigskip
  \or % sigchi-a
  \fi
}
\def\@setthanks{\long\def\thanks##1{\par##1\@addpunct.}\thankses}
\RequirePackage{fancyhdr}
\if@ACM@review
  \newsavebox{\ACM@linecount@bx}
  \newlength\ACM@linecount@bxht
  \newcount\ACM@linecount
  \ACM@linecount\@ne\relax
  \def\ACM@mk@linecount{%
    \savebox{\ACM@linecount@bx}[4em][t]{\parbox[t]{4em}{%
        \setlength{\ACM@linecount@bxht}{-\baselineskip}%
        \loop{\color{red}\scriptsize\the\ACM@linecount}\\
        \global\advance\ACM@linecount by \@ne
        \addtolength{\ACM@linecount@bxht}{\baselineskip}%
        \ifdim\ACM@linecount@bxht<\textheight\repeat}}}
\fi
\def\ACM@linecountL{%
  \if@ACM@review
  \ACM@mk@linecount
  \begin{picture}(0,0)%
    \put(-26,-22){\usebox{\ACM@linecount@bx}}%
  \end{picture}%
  \fi}
\def\ACM@linecountR{%
  \if@ACM@review
  \ACM@mk@linecount
  \begin{picture}(0,0)%
    \put(20,-22){\usebox{\ACM@linecount@bx}}%
  \end{picture}%
  \fi}
\if@ACM@timestamp
  % Subtracting 30 from \time gives us the effect of rounding-down despite
  % \numexpr rounding to nearest
  \newcounter{ACM@time@hours}
  \setcounter{ACM@time@hours}{\numexpr (\time - 30) / 60 \relax}
  \newcounter{ACM@time@minutes}
  \setcounter{ACM@time@minutes}{\numexpr \time - \theACM@time@hours * 60 \relax}
  \newcommand\ACM@timestamp{%
    \footnotesize%
    \the\year-\two@digits{\the\month}-\two@digits{\the\day}{ }%
    \two@digits{\theACM@time@hours}:\two@digits{\theACM@time@minutes}{ }%
    page~\thepage\ (pp. \@startPage-\pageref*{TotPages})%
    \ifx\@acmSubmissionID\@empty\relax\else
    ~Submission~ID: \@acmSubmissionID
    \fi
  }
\fi
\def\@shortauthors{\if@ACM@anonymous Anon.\else\shortauthors\fi}
\def\@headfootfont{%
  \ifcase\ACM@format@nr
  \relax % manuscript
    \sffamily
  \or % acmsmall
    \sffamily
  \or % acmlarge
    \sffamily
  \or % acmtog
    \sffamily
  \or % sigconf
    \sffamily
  \or % siggraph
    \sffamily
  \or % sigplan
    \sffamily
  \or % sigchi
    \sffamily
  \or % sigchi-a
    \sffamily
  \fi}
\fancypagestyle{standardpagestyle}{%
  \fancyhf{}%
  \renewcommand{\headrulewidth}{\z@}%
  \renewcommand{\footrulewidth}{\z@}%
  \ifcase\ACM@format@nr
  \relax % manuscript
    \fancyhead[LE]{\ACM@linecountL\if@ACM@printfolios\thepage\fi}%
    \fancyhead[RO]{\if@ACM@printfolios\thepage\fi}%
    \fancyhead[RE]{\@shortauthors}%
    \fancyhead[LO]{\ACM@linecountL\shorttitle}%
    \fancyfoot[RO,LE]{\footnotesize Manuscript submitted to ACM}%
  \or % acmsmall
    \fancyhead[LE]{\ACM@linecountL\@headfootfont\@acmArticle\if@ACM@printfolios:\thepage\fi}%
    \fancyhead[RO]{\@headfootfont\@acmArticle\if@ACM@printfolios:\thepage\fi}%
    \fancyhead[RE]{\@headfootfont\@shortauthors}%
    \fancyhead[LO]{\ACM@linecountL\@headfootfont\shorttitle}%
    \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No.
    \@acmNumber, Article \@acmArticle.  Publication date: \@acmPubDate.}%
  \or % acmlarge
    \fancyhead[LE]{\ACM@linecountL\@headfootfont
      \@acmArticle\if@ACM@printfolios:\thepage\fi\quad\textbullet\quad\@shortauthors}%
    \fancyhead[LO]{\ACM@linecountL}%
    \fancyhead[RO]{\@headfootfont
      \shorttitle\quad\textbullet\quad\@acmArticle\if@ACM@printfolios:\thepage\fi}%
    \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No.
    \@acmNumber, Article \@acmArticle.  Publication date: \@acmPubDate.}%
  \or % acmtog
    \fancyhead[LE]{\ACM@linecountL\@headfootfont
      \@acmArticle\if@ACM@printfolios:\thepage\fi\quad\textbullet\quad\@shortauthors}%
    \fancyhead[LO]{\ACM@linecountL}%
    \fancyhead[RE]{\ACM@linecountR}%
    \fancyhead[RO]{\@headfootfont
      \shorttitle\quad\textbullet\quad\@acmArticle\if@ACM@printfolios:\thepage\fi\ACM@linecountR}%
    \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No.
    \@acmNumber, Article \@acmArticle.  Publication date: \@acmPubDate.}%
  \else % Proceedings
    \fancyfoot[C]{\if@ACM@printfolios\footnotesize\thepage\fi}%
    \fancyhead[LO]{\ACM@linecountL\@headfootfont\shorttitle}%
    \fancyhead[RE]{\@headfootfont\@shortauthors\ACM@linecountR}%
    \fancyhead[LE]{\ACM@linecountL\@headfootfont\acmConference@shortname,
      \acmConference@date, \acmConference@venue}%
    \fancyhead[RO]{\@headfootfont\acmConference@shortname,
      \acmConference@date, \acmConference@venue\ACM@linecountR}%
  \fi
  \if@ACM@sigchiamode
     \fancyheadoffset[L]{\dimexpr(\marginparsep+\marginparwidth)}%
  \fi
  \if@ACM@timestamp
     \fancyfoot[LO,RE]{\ACM@timestamp}
  \fi
}
\pagestyle{standardpagestyle}
\newdimen\@folio@wd
\@folio@wd=\z@
\newdimen\@folio@ht
\@folio@ht=\z@
\newdimen\@folio@voffset
\@folio@voffset=\z@
\def\@folio@max{1}
\ifcase\ACM@format@nr
\relax % manuscript
\or % acmsmall
  \@folio@wd=45.75pt\relax
  \@folio@ht=1.25in\relax
  \@folio@voffset=.2in\relax
  \def\@folio@max{8}
\or % acmlarge
  \@folio@wd=43.25pt\relax
  \@folio@ht=79pt\relax
  \@folio@voffset=.55in\relax
  \def\@folio@max{10}
\fi
\def\@folioblob{\@tempcnta=\@acmArticleSeq\relax
  \loop
     \ifnum\@tempcnta>\@folio@max\relax
      \advance\@tempcnta by - \@folio@max
   \repeat
    \advance\@tempcnta by -1\relax
    \@tempdima=\@folio@ht\relax
    \multiply\@tempdima by \the\@tempcnta\relax
    \advance\@tempdima by -\@folio@voffset\relax
    \begin{picture}(0,0)
    \makebox[\z@]{\raisebox{-\@tempdima}{%
        \rlap{%
          \raisebox{-0.45\@folio@ht}[\z@][\z@]{%
            \rule{\@folio@wd}{\@folio@ht}}}%
        \parbox{\@folio@wd}{%
          \centering
          \textcolor{white}{\LARGE\bfseries\sffamily\@acmArticle}}}}
  \end{picture}}

\fancypagestyle{firstpagestyle}{%
  \fancyhf{}%
  \renewcommand{\headrulewidth}{\z@}%
  \renewcommand{\footrulewidth}{\z@}%
  \ifcase\ACM@format@nr
  \relax % manuscript
    \fancyhead[L]{\ACM@linecountL}%
    \fancyfoot[RO,LE]{\if@ACM@printfolios\small\thepage\fi}%
    \fancyfoot[RE,LO]{\footnotesize Manuscript submitted to ACM}%
  \or % acmsmall
    \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No.
    \@acmNumber, Article \@acmArticle.  Publication date:
    \@acmPubDate.}%
    \fancyhead[LE]{\ACM@linecountL\@folioblob}%
    \fancyhead[LO]{\ACM@linecountL}%
    \fancyhead[RO]{\@folioblob}%
    \fancyheadoffset[RO,LE]{0.6\@folio@wd}%
  \or % acmlarge
    \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No.
    \@acmNumber, Article \@acmArticle.  Publication date:
    \@acmPubDate.}%
    \fancyhead[RO]{\@folioblob}%
    \fancyhead[LE]{\ACM@linecountL\@folioblob}%
    \fancyhead[LO]{\ACM@linecountL}%
    \fancyheadoffset[RO,LE]{1.4\@folio@wd}%
  \or % acmtog
    \fancyfoot[RO,LE]{\footnotesize \@journalName, Vol. \@acmVolume, No.
    \@acmNumber, Article \@acmArticle.  Publication date:
    \@acmPubDate.}%
    \fancyhead[L]{\ACM@linecountL}%
    \fancyhead[R]{\ACM@linecountR}%
  \else % Conference proceedings
    \fancyhead[L]{\ACM@linecountL}%
    \fancyhead[R]{\ACM@linecountR}%
    \fancyfoot[C]{\if@ACM@printfolios\footnotesize\thepage\fi}%
  \fi
  \if@ACM@timestamp
    \ifnum\ACM@format@nr=0\relax % Manuscript
    \fancyfoot[LO,RE]{\ACM@timestamp\quad
      \footnotesize Manuscript submitted to ACM}
    \else
    \fancyfoot[LO,RE]{\ACM@timestamp}
    \fi
  \fi
}
\renewcommand\section{\@startsection{section}{1}{\z@}%
  {-.75\baselineskip \@plus -2\p@ \@minus -.2\p@}%
  {.25\baselineskip}%
  {\@secfont}}
\renewcommand\subsection{\@startsection{subsection}{2}{\z@}%
  {-.75\baselineskip \@plus -2\p@ \@minus -.2\p@}%
  {.25\baselineskip}%
  {\@subsecfont}}
\renewcommand\subsubsection{\@startsection{subsubsection}{3}{10pt}%
  {-.5\baselineskip \@plus -2\p@ \@minus -.2\p@}%
  {-3.5\p@}%
  {\@subsubsecfont\@adddotafter}}
\renewcommand\paragraph{\@startsection{paragraph}{4}{\parindent}%
  {-.5\baselineskip \@plus -2\p@ \@minus -.2\p@}%
  {-3.5\p@}%
  {\@parfont\@adddotafter}}
\renewcommand\part{\@startsection{part}{9}{\z@}%
  {-10\p@ \@plus -4\p@ \@minus -2\p@}%
  {4\p@}%
  {\@parfont}}
\def\section@raggedright{\@rightskip\@flushglue
  \rightskip\@rightskip
  \leftskip\z@skip
  \parindent\z@}
\def\@secfont{\sffamily\bfseries\section@raggedright\MakeTextUppercase}
\def\@subsecfont{\sffamily\bfseries\section@raggedright}
\def\@subsubsecfont{\sffamily\itshape}
\def\@parfont{\itshape}
\setcounter{secnumdepth}{3}
\ifcase\ACM@format@nr
\relax % manuscript
\or % acmsmall
\or % acmlarge
 \def\@secfont{\sffamily\large\section@raggedright\MakeTextUppercase}
 \def\@subsecfont{\sffamily\large\section@raggedright}
\or % acmtog
 \def\@secfont{\sffamily\large\section@raggedright\MakeTextUppercase}
 \def\@subsecfont{\sffamily\large\section@raggedright}
\or % sigconf
 \def\@secfont{\bfseries\Large\section@raggedright\MakeTextUppercase}
 \def\@subsecfont{\bfseries\Large\section@raggedright}
\or % siggraph
 \def\@secfont{\bfseries\sffamily\Large\section@raggedright\MakeTextUppercase}
 \def\@subsecfont{\bfseries\sffamily\Large\section@raggedright}
\or % sigplan
 \def\@secfont{\bfseries\Large\section@raggedright}
 \def\@subsecfont{\bfseries\section@raggedright}
 \renewcommand\subsubsection{\@startsection{subsubsection}{3}{\z@}%
   {-.75\baselineskip \@plus -2\p@ \@minus -.2\p@}%
   {.25\baselineskip}%
   {\@subsubsecfont}}
 \def\@subsubsecfont{\bfseries\section@raggedright}
 \renewcommand\paragraph{\@startsection{paragraph}{4}{\z@}%
   {-.5\baselineskip \@plus -2\p@ \@minus -.2\p@}%
   {-3.5\p@}%
   {\@parfont\@addspaceafter}}
 \def\@parfont{\bfseries\itshape}
 \renewcommand\subparagraph{\@startsection{subparagraph}{5}{\z@}%
   {-.5\baselineskip \@plus -2\p@ \@minus -.2\p@}%
   {-3.5\p@}%
   {\@subparfont\@addspaceafter}}
 \def\@subparfont{\itshape}
\or % sigchi
 \setcounter{secnumdepth}{1}
 \def\@secfont{\bfseries\sffamily\section@raggedright\MakeTextUppercase}
 \def\@subsecfont{\bfseries\sffamily\section@raggedright}
\or % sigchi-a
 \setcounter{secnumdepth}{0}
 \def\@secfont{\bfseries\sffamily\section@raggedright\MakeTextUppercase}
 \def\@subsecfont{\bfseries\sffamily\section@raggedright}
\fi
\def\@adddotafter#1{#1\@addpunct{.}}
\def\@addspaceafter#1{#1\@addpunct{\enspace}}
\providecommand*\@dotsep{4.5}
\def\@acmplainbodyfont{\itshape}
\def\@acmplainindent{\parindent}
\def\@acmplainheadfont{\scshape}
\def\@acmplainnotefont{\@empty}
\ifcase\ACM@format@nr
\relax % manuscript
\or % acmsmall
\or % acmlarge
\or % acmtog
\or % sigconf
\or % siggraph
\or % sigplan
  \def\@acmplainbodyfont{\itshape}
  \def\@acmplainindent{\z@}
  \def\@acmplainheadfont{\bfseries}
  \def\@acmplainnotefont{\normalfont}
\or % sigchi
\or % sigchi-a
\fi
\newtheoremstyle{acmplain}%
  {.5\baselineskip\@plus.2\baselineskip
    \@minus.2\baselineskip}% space above
  {.5\baselineskip\@plus.2\baselineskip
    \@minus.2\baselineskip}% space below
  {\@acmplainbodyfont}% body font
  {\@acmplainindent}% indent amount
  {\@acmplainheadfont}% head font
  {.}% punctuation after head
  {.5em}% spacing after head
  {\thmname{#1}\thmnumber{ #2}\thmnote{ {\@acmplainnotefont(#3)}}}% head spec
\def\@acmdefinitionbodyfont{\normalfont}
\def\@acmdefinitionindent{\parindent}
\def\@acmdefinitionheadfont{\itshape}
\def\@acmdefinitionnotefont{\@empty}
\ifcase\ACM@format@nr
\relax % manuscript
\or % acmsmall
\or % acmlarge
\or % acmtog
\or % sigconf
\or % siggraph
\or % sigplan
  \def\@acmdefinitionbodyfont{\normalfont}
  \def\@acmdefinitionindent{\z@}
  \def\@acmdefinitionheadfont{\bfseries}
  \def\@acmdefinitionnotefont{\normalfont}
\or % sigchi
\or % sigchi-a
\fi
\newtheoremstyle{acmdefinition}%
  {.5\baselineskip\@plus.2\baselineskip
    \@minus.2\baselineskip}% space above
  {.5\baselineskip\@plus.2\baselineskip
    \@minus.2\baselineskip}% space below
  {\@acmdefinitionbodyfont}% body font
  {\@acmdefinitionindent}% indent amount
  {\@acmdefinitionheadfont}% head font
  {.}% punctuation after head
  {.5em}% spacing after head
  {\thmname{#1}\thmnumber{ #2}\thmnote{ {\@acmdefinitionnotefont(#3)}}}% head spec
\theoremstyle{acmplain}
\newtheorem{theorem}{Theorem}[section]
\newtheorem{conjecture}[theorem]{Conjecture}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{corollary}[theorem]{Corollary}
\theoremstyle{acmdefinition}
\newtheorem{example}[theorem]{Example}
\newtheorem{definition}[theorem]{Definition}
\theoremstyle{acmplain}
\def\@proofnamefont{\scshape}
\def\@proofindent{\indent}
\ifcase\ACM@format@nr
\relax % manuscript
\or % acmsmall
\or % acmlarge
\or % acmtog
\or % sigconf
\or % siggraph
\or % sigplan
  \def\@proofnamefont{\itshape}
  \def\@proofindent{\noindent}
\or % sigchi
\or % sigchi-a
\fi
\renewenvironment{proof}[1][\proofname]{\par
  \pushQED{\qed}%
  \normalfont \topsep6\p@\@plus6\p@\relax
  \trivlist
  \item[\@proofindent\hskip\labelsep
        {\@proofnamefont #1\@addpunct{.}}]\ignorespaces
}{%
  \popQED\endtrivlist\@endpefalse
}
\specialcomment{acks}{%
  \begingroup
  \section*{Acknowledgments}
  \phantomsection\addcontentsline{toc}{section}{Acknowledgments}
}{%
  \endgroup
}
\def\grantsponsor#1#2#3{#2}
\newcommand\grantnum[3][]{#3%
  \def\@tempa{#1}\ifx\@tempa\@empty\else\space(\url{#1})\fi}
\if@ACM@screen
  \includecomment{screenonly}
  \excludecomment{printonly}
\else
  \excludecomment{screenonly}
  \includecomment{printonly}
\fi
\if@ACM@anonymous
  \excludecomment{anonsuppress}
  \excludecomment{acks}
\else
  \includecomment{anonsuppress}
\fi
\newcommand\showeprint[2][arxiv]{%
  \def\@tempa{#1}%
  \ifx\@tempa\@empty\def\@tempa{arxiv}\fi
  \def\@tempb{arxiv}%
  \ifx\@tempa\@tempb
     arXiv:\href{http://arxiv.org/abs/#2}{#2}\else arXiv:#2%
  \fi}
\normalsize\normalfont\frenchspacing
\endinput
%%
%% End of file `acmart.cls'.

Ignore Space Show notes View Koli_2017/db_metrics.xlsx 0 → 100644
Not supported

Ignore Space Show notes View Koli_2017/distribution_figures.R 0 → 100644
# http://www.statmethods.net/advgraphs/layout.html par(fig = c(0,1,0,0.875), new = FALSE) hist(marksData[marksData$Year %in% c(2013:2014),]$Mark, breaks = seq(from = 0, to = 100, by = 10), xlab = "Grade (%)", main = "") par(fig=c(0,1,0.55,1), new = TRUE) boxplot(marksData[marksData$Year %in% c(2013:2014),]$Mark, horizontal = TRUE, axes = FALSE, ylim = c(0,100)) par(fig = c(0,1,0,0.875), new = FALSE) hist(marksData[marksData$Year %in% c(2009:2012,2015,2016),]$Mark, breaks = seq(from = 0, to = 100, by = 10), xlab = "Grade (%)", main = "") par(fig = c(0,1,0.55,1), new = TRUE) boxplot(marksData[marksData$Year %in% c(2009:2012,2015,2016),]$Mark, horizontal = TRUE, axes = FALSE, ylim = c(0,100)) par(mfrow = c(3,3), mar = c(4, 2, 1, 1), oma = c(2.5, 2.5, 0, 0)) for(i in c(2009:2016)) { hist(marksData[marksData$Year == i,]$Mark, breaks = seq(from = 0, to = 100, by = 10), xlab = "", ylab = "", main = "", xaxt = "n") axis(1, at = seq(from = 0, to = 100, by = 20), labels = c(0, "", 40, "", 80, "")) title(xlab = i, line = 2.3) } mtext(side = 1, "Grade (%)", outer = TRUE, line = 0.6) mtext(side = 2, "Frequency", outer = TRUE, line = 0.8)

Ignore Space Show notes View

Koli_2017/distribution_figures.R 0 → 100644

# http://www.statmethods.net/advgraphs/layout.html
par(fig = c(0,1,0,0.875), new = FALSE)
hist(marksData[marksData$Year %in% c(2013:2014),]$Mark, breaks = seq(from = 0, to = 100, by = 10), xlab = "Grade (%)", main = "")
par(fig=c(0,1,0.55,1), new = TRUE)
boxplot(marksData[marksData$Year %in% c(2013:2014),]$Mark, horizontal = TRUE, axes = FALSE, ylim = c(0,100))

par(fig = c(0,1,0,0.875), new = FALSE)
hist(marksData[marksData$Year %in% c(2009:2012,2015,2016),]$Mark, breaks = seq(from = 0, to = 100, by = 10), xlab = "Grade (%)", main = "")
par(fig = c(0,1,0.55,1), new = TRUE)
boxplot(marksData[marksData$Year %in% c(2009:2012,2015,2016),]$Mark, horizontal = TRUE, axes = FALSE, ylim = c(0,100))

par(mfrow = c(3,3), mar = c(4, 2, 1, 1), oma = c(2.5, 2.5, 0, 0))
for(i in c(2009:2016))
{
    hist(marksData[marksData$Year == i,]$Mark, breaks = seq(from = 0, to = 100, by = 10), xlab = "", ylab = "", main = "", xaxt = "n")
    axis(1, at = seq(from = 0, to = 100, by = 20), labels = c(0, "", 40, "", 80, ""))
    title(xlab = i, line = 2.3)
}
mtext(side = 1, "Grade (%)", outer = TRUE, line = 0.6)   
mtext(side = 2, "Frequency", outer = TRUE, line = 0.8)

Ignore Space Show notes View Koli_2017/gpa_summary.xlsx 0 → 100644
Not supported

Ignore Space Show notes View Koli_2017/grades_all_years.pdf 0 → 100644
Not supported

Ignore Space Show notes View Koli_2017/grades_with_student_mode.pdf 0 → 100644
Not supported

Ignore Space Show notes View Koli_2017/grades_without_student_mode.pdf 0 → 100644
Not supported

Ignore Space Show notes View Koli_2017/images/BDL_ERD.pu 0 → 100644
@startuml hide empty members skinparam { DefaultFontName "Linux Biolinum O" DefaultFontSize 9 Style strictuml Monochrome true LineType ortho RankSep 40 ClassBorderThickness 0.7 } entity Customer entity Staff entity Supplier entity Sale_Head entity Sale_Line entity Order_Head entity Order_Line entity Product entity Assembly entity Component Customer \|\|..o{ Sale_Head Supplier -[hidden]- Order_Head Supplier \|\|..o{ Order_Head Sale_Head }o.\|\| Staff Staff \|\|.o{ Order_Head Sale_Head \|\|--\|{ Sale_Line Order_Head \|\|--\|{ Order_Line Sale_Line }o--\|\| Product Supplier \|\|..o{ Component Order_Line }o--\|\| Component Product \|\|-\|{ Assembly Assembly }\|-\|\| Component @enduml

Ignore Space Show notes View

Koli_2017/images/BDL_ERD.pu 0 → 100644

Ignore Space Show notes View Koli_2017/images/web_output.png 0 → 100644

Ignore Space Show notes View Koli_2017/latexmkrc 0 → 100644
$bibtex = 'bibtex -min-crossrefs=2 %O %B';

Ignore Space Show notes View Koli_2017/marks_data.xlsx 0 → 100644
Not supported

Ignore Space Show notes View Koli_2017/notes.txt 0 → 100644
Numerous prior systems for automatic grading of student queries, but these are all focused on the DML side of things. • RDBI (Dietrich 1993) supports relational algebra, domain and tuple relational calculus, and SQL (SELECT only). Uses its own internal DBMS. \citeauthor{Dietrich.S-1993a-An-educational}'s \emph{RDBI} \cite{Dietrich.S-1993a-An-educational} was a Prolog-based interpreter for relational algebra, tuple and domain relational calculus, and SQL. It focused primarily on queries, and used its own non-SQL data definition language. RDBI did not provide feedback on students' attempts beyond basic syntax checking and displaying query results. • esql (Kearns et al., 1997) supports CREATE TABLE only as a pass-through to set up or modify a schema. It uses its own internal DBMS. \citeauthor{Kearns.R-1997a-A-teaching}'s \emph{esql} \cite{Kearns.R-1997a-A-teaching} supported students in learning the fundamental concepts underlying SQL. It could parse and execute \texttt{CREATE}, \texttt{DROP}, \texttt{ALTER}, \texttt{DELETE}, \texttt{INSERT}, and \texttt{SELECT} statements, but all of these except \texttt{SELECT} were simply passed through to the DBMS. The system enabled students to better understand the steps in the execution of a query by visualizing the intermediate tables generated by each step of the query. It did not provide feedback on students' attempts beyond basic syntax checking and displaying query results. • SQL-Tutor (Mitrovic, 1998) is an intelligent tutoring system that only only supports SELECT. \citeauthor{Mitrovic.A-1998a-Learning}'s \emph{SQL-Tutor} \cite{Mitrovic.A-1998a-Learning} was an intelligent teaching system that provided students with a guided discovery learning environment for SQL queries. It supported only the \texttt{SELECT} statement, and used constraint-based modeling \cite{Ohlsson.S-1992a-Constraint-based,Ohlsson.S-2016a-Constraint-based} to provide feedback to students on both syntactic and semantic SQL errors. • SQLator (Sadiq et al., 2004) only supports SELECT. \citeauthor{Sadiq.S-2004a-SQLator} \emph{SQLator} \cite{Sadiq.S-2004a-SQLator} was a web-based interactive tool for learning SQL. Students were presented with a series of questions in English, and had to write SQL \texttt{SELECT} statements to answer these questions. SQLator used an ``equivalence engine'' to determine whether an SQL query fulfilled the requirements of the original English question. SQLator supported only the \texttt{SELECT} statement, and provided only basic feedback (correct or incorrect) to students. SQLator was able to automatically mark about a third of submitted queries as correct, thus improving the speed of grading. • AsseSQL (Prior & Lister, 2004) only supports SELECT. \citeauthor{Prior.J-2004a-Backwash}'s \emph{AsseSQL} \cite{Prior.J-2004a-Backwash} was an online examination environment for evaluating students' ability to formulate SQL queries. Students would write and execute their queries, and the data set produced by their query would be compared against the correct data set. The answer would then be flagged as correct or incorrect as appropriate. AsseSQL supported only the \texttt{SELECT} statement. • ActiveSQL (Cumming & Russell, 2005; Russell & Cumming, 2005) only supports SELECT. \citeauthor{Russell.G-2004a-Improving}'s \emph{ActiveSQL}\footnote{\url{https://db.grussell.org/}} \cite{Russell.G-2004a-Improving,Russell.G-2005a-Online} was an online interactive learning environment for SQL that provided immediate feedback to students. ActiveSQL measured the accuracy of a query in a similar way to \citeauthor{Prior.J-2004a-Backwash}'s AsseSQL, but instead of a simple correct/incorrect answer, it computed an accuracy score based on the differences between the query output and the correct answer. It was also able to detect ``hard-coded'' queries that produced the desired result, but would fail if the data set changed \cite{Russell.G-2005a-Online}. ActiveSQL supported only the \texttt{SELECT} statement. • SQLify (Dekeyser et al., 2007) only supports SELECT. \citeauthor{Dekeyser.S-2007a-Computer}'s \emph{SQLify} \cite{Dekeyser.S-2007a-Computer} was another online SQL learning system that incorporated semantic feedback and automatic assessment. SQLify evaluated each query on an eight-level scale that covered query syntax, output schema, and query semantics. Instructors could use this information to award an overall grade. Again, SQLify supported only the \texttt{SELECT} statement. • SQL Exploratorium (Brusilovsky et al., 2010) \citeauthor{Brusilovsky.P-2010a-Learning}'s \emph{SQL Exploratorium} \cite{Brusilovsky.P-2010a-Learning} took an interesting approach to generating problems, using parameterized query templates to generate the questions given to students. Again, the SQL Exploratorium supported only the \texttt{SELECT} statement. • aSQLg (Kleiner et al., 2013) only supports SELECT. Works with any back end DBMS. Uses the DBMS as a syntax checker. \citeauthor{Kleiner.C-2013a-Automated}'s \emph{aSQLg} \cite{Kleiner.C-2013a-Automated} was an automated assessment tool that provided feedback to students. This enabled students to improve their learning by making further submissions after incorporating this feedback. The aSQLg system checked queries for syntax, efficiency (cost), result correctness, and statement style. Again, aSQLg supported only the \texttt{SELECT} statement. • (Kenny & Pahl, 2005) queries only. \citeauthor{Kenny.C-2005a-Automated} \cite{Kenny.C-2005a-Automated} described an SQL learning system similar to those already described, which also incorporated an assessment of a student's previous progress. This enabled a more personalized and adaptive approach to student learning, where feedback was tailored according to student progress. Again, this system supported only the \texttt{SELECT} statement. • XData (Bhangdiya et al., 2015, Chandra et al., 2015; Chandra et al., 2016) only supports SELECT. \citeauthor{Bhangdiya.A-2015a-XDa-TA}'s \emph{XDa-TA}\footnote{\url{http://www.cse.iitb.ac.in/infolab/xdata/}} extended the idea of automated grading of SQL by adding the ability to generate data sets designed to catch common errors. These data sets were automatically derived from a set of correct SQL queries \cite{Bhangdiya.A-2015a-XDa-TA,Chandra.B-2015a-Data}. Later work \cite{Chandra.B-2016a-Partial} added support for awarding partial marks. • “CS 121 Automation Tool” (anjoola, cs12x-automate, GitHub, last updated 2015) Automated marking system for SQL that appears be customisable for all kinds of statements? However the code to test CREATE TABLE “Simply executes the CREATE TABLE statements to make sure they have the correct syntax”. It doesn't check against any form of specification. \citeauthor{Gong.A-2015a-CS-121-Automation}'s ``CS 121 Automation Tool'' \cite{Gong.A-2015a-CS-121-Automation} was a tool designed to semi-automate the grading of SQL assessments, again focusing on SQL DML statements. Interestingly, the system appears to be extensible and could thus potentially be modified to support grading of SQL DDL statements. On the pure testing side of things: • SVTe (Farré et al., 2008) tests the “correctness” of a schema, but focuses mainly on consistency of constraints. • Ambler (n.d., http://www.agiledata.org/essays/databaseTesting.html) talks purely about testing the database functionality. There appears to be very little work on automated grading of students’ schema definitions. No-one has used unit testing as a framework for doing so. What use of unit testing there is, is more focused on testing database /applications/ and effective generation of test cases: • (Binnig et al., 2013) Generating test databases (fixtures) for OLTP applications. • (Chays & Shahid, 2008) Generating test inputs for database applications based on analysing the queries used. • (Marcozzi et al, 2012) Generating test inputs for database applications using a formal language (ImperDB). No-one appears to have tried to automatically test whether an extant schema is consistent with its specification. This is almost more of a formal methods thing? These are more focused on /generating/ a conforming schema from the specification (e.g., Vatanawood & Rivepiboon, 2004; Luković et al., 2003), rather than checking that an already existing schema conforms. Also more focused on schema /transformations/ and evolution (e.g., Bench-Capon et al., 1998). ------------------------------------------------------------ NOTE: Checking structure of table Product. TEST: [[ Product ]] + OK +++ PASSED: Table Product exists. TEST: [[ Product.Product_code ]] + OK ... +++ PASSED: Table Product contains all the expected columns. TEST: [[ Product.Product_code: data type is NUMBER \| INTEGER ]] + OK ... +++ PASSED: All columns of table Product have data types compatible with the specification. TEST: [[ Product.Product_code precision and scale = 8 (with scale 0) ]] + OK ... +++ PASSED: All columns of table Product have lengths compatible with the specification. TEST: [[ Product.Product_code nullability should be N ]] + OK ... +++ PASSED: All columns of table Product have the expected nullability. TEST: [[ Product PK ]] + OK +++ PASSED: Primary key of table Product exists. TEST: [[ Product PK: Product_code ]] + OK +++ PASSED: Primary key of table Product includes (only) the expected columns. TEST: [[ Product check constraint PRODUCT_STOCK_INVALID ]] + OK ... +++ PASSED: All constraints of table Product that should be are explicitly named. NOTE: Testing constraints of table Product. TEST: [[ Product.Stock_count accepts “0” ]] + OK TEST: [[ Product.Stock_count accepts “99999” ]] + OK TEST: [[ Product.Restock_level accepts “0” ]] - FAILED! Column Product.Restock_level won’t accept legal value 0 [-0.5]. Failed asserting that false is true. TEST: [[ Product.Restock_level accepts “99999” ]] + OK TEST: [[ Product.Minimum_level accepts “0” ]] - FAILED! Column Product.Minimum_level won’t accept legal value 0 [-0.5]. Failed asserting that false is true. TEST: [[ Product.Minimum_level accepts “653” ]] + OK TEST: [[ Product.List_price accepts “0” ]] + OK TEST: [[ Product.List_price accepts “99999.99” ]] + OK --- FAILED: 2 of 8 legal values tested were rejected by a CHECK constraint.

Ignore Space Show notes View

Koli_2017/notes.txt 0 → 100644

Numerous prior systems for automatic grading of student queries, but these are all focused on the DML side of things.

• RDBI (Dietrich 1993) supports relational algebra, domain and tuple relational calculus, and SQL (SELECT only). Uses its own internal DBMS.
\citeauthor{Dietrich.S-1993a-An-educational}'s \emph{RDBI} \cite{Dietrich.S-1993a-An-educational} was a Prolog-based interpreter for relational algebra, tuple and domain relational calculus, and SQL. It focused primarily on queries, and used its own non-SQL data definition language. RDBI did not provide feedback on students' attempts beyond basic syntax checking and displaying query results.

• esql (Kearns et al., 1997) supports CREATE TABLE only as a pass-through to set up or modify a schema. It uses its own internal DBMS.
\citeauthor{Kearns.R-1997a-A-teaching}'s \emph{esql} \cite{Kearns.R-1997a-A-teaching} supported students in learning the fundamental concepts underlying SQL. It could parse and execute \texttt{CREATE}, \texttt{DROP}, \texttt{ALTER}, \texttt{DELETE}, \texttt{INSERT}, and \texttt{SELECT} statements, but all of these except \texttt{SELECT} were simply passed through to the DBMS. The system enabled students to better understand the steps in the execution of a query by visualizing the intermediate tables generated by each step of the query. It did not provide feedback on students' attempts beyond basic syntax checking and displaying query results.

• SQL-Tutor (Mitrovic, 1998) is an intelligent tutoring system that only only supports SELECT.
\citeauthor{Mitrovic.A-1998a-Learning}'s \emph{SQL-Tutor} \cite{Mitrovic.A-1998a-Learning} was an intelligent teaching system  that provided students with a guided discovery learning environment for SQL queries. It supported only the \texttt{SELECT} statement, and used constraint-based modeling \cite{Ohlsson.S-1992a-Constraint-based,Ohlsson.S-2016a-Constraint-based} to provide feedback to students on both syntactic and semantic SQL errors.

• SQLator (Sadiq et al., 2004) only supports SELECT.
\citeauthor{Sadiq.S-2004a-SQLator} \emph{SQLator} \cite{Sadiq.S-2004a-SQLator} was a web-based interactive tool for learning SQL. Students were presented with a series of questions in English, and had to write SQL \texttt{SELECT} statements to answer these questions. SQLator used an ``equivalence engine'' to determine whether an SQL query fulfilled the requirements of the original English question. SQLator supported only the \texttt{SELECT} statement, and provided only basic feedback (correct or incorrect) to students. SQLator was able to automatically mark about a third of submitted queries as correct, thus improving the speed of grading.

• AsseSQL (Prior & Lister, 2004) only supports SELECT.
\citeauthor{Prior.J-2004a-Backwash}'s \emph{AsseSQL} \cite{Prior.J-2004a-Backwash} was an online examination environment for evaluating students' ability to formulate SQL queries. Students would write and execute their queries, and the data set produced by their query would be compared against the correct data set. The answer would then be flagged as correct or incorrect as appropriate. AsseSQL supported only the \texttt{SELECT} statement.

• ActiveSQL (Cumming & Russell, 2005; Russell & Cumming, 2005) only supports SELECT.
\citeauthor{Russell.G-2004a-Improving}'s \emph{ActiveSQL}\footnote{\url{https://db.grussell.org/}} \cite{Russell.G-2004a-Improving,Russell.G-2005a-Online} was an online interactive learning environment for SQL that provided immediate feedback to students. ActiveSQL measured the accuracy of a query in a similar way to \citeauthor{Prior.J-2004a-Backwash}'s AsseSQL, but instead of a simple correct/incorrect answer, it computed an accuracy score based on the differences between the query output and the correct answer. It was also able to detect ``hard-coded'' queries that produced the desired result, but would fail if the data set changed \cite{Russell.G-2005a-Online}. ActiveSQL supported only the \texttt{SELECT} statement.

• SQLify (Dekeyser et al., 2007) only supports SELECT.
\citeauthor{Dekeyser.S-2007a-Computer}'s \emph{SQLify} \cite{Dekeyser.S-2007a-Computer} was another online SQL learning system that incorporated semantic feedback and automatic assessment. SQLify evaluated each query on an eight-level scale that covered query syntax, output schema, and query semantics. Instructors could use this information to award an overall grade. Again, SQLify supported only the \texttt{SELECT} statement.

• SQL Exploratorium (Brusilovsky et al., 2010)
\citeauthor{Brusilovsky.P-2010a-Learning}'s \emph{SQL Exploratorium} \cite{Brusilovsky.P-2010a-Learning} took an interesting approach to generating problems, using parameterized query templates to generate the questions given to students. Again, the SQL Exploratorium supported only the \texttt{SELECT} statement.

• aSQLg (Kleiner et al., 2013) only supports SELECT. Works with any back end DBMS. Uses the DBMS as a syntax checker.
\citeauthor{Kleiner.C-2013a-Automated}'s \emph{aSQLg} \cite{Kleiner.C-2013a-Automated} was an automated assessment tool that provided feedback to students. This enabled students to improve their learning by making further submissions after incorporating this feedback. The aSQLg system checked queries for syntax, efficiency (cost), result correctness, and statement style. Again, aSQLg supported only the \texttt{SELECT} statement.

• (Kenny & Pahl, 2005) queries only.
\citeauthor{Kenny.C-2005a-Automated} \cite{Kenny.C-2005a-Automated} described an SQL learning system similar to those already described, which also incorporated an assessment of a student's previous progress. This enabled a more personalized and adaptive approach to student learning, where feedback was tailored according to student progress. Again, this system supported only the \texttt{SELECT} statement.

• XData (Bhangdiya et al., 2015, Chandra et al., 2015; Chandra et al., 2016) only supports SELECT.
\citeauthor{Bhangdiya.A-2015a-XDa-TA}'s \emph{XDa-TA}\footnote{\url{http://www.cse.iitb.ac.in/infolab/xdata/}} extended the idea of automated grading of SQL by adding the ability to generate data sets designed to catch common errors. These data sets were automatically derived from a set of correct SQL queries \cite{Bhangdiya.A-2015a-XDa-TA,Chandra.B-2015a-Data}. Later work \cite{Chandra.B-2016a-Partial} added support for awarding partial marks.

• “CS 121 Automation Tool” (anjoola, cs12x-automate, GitHub, last updated 2015) Automated marking system for SQL that appears be customisable for all kinds of statements? However the code to test CREATE TABLE “Simply executes the CREATE TABLE statements to make sure they have the correct syntax”. It doesn't check against any form of specification.
\citeauthor{Gong.A-2015a-CS-121-Automation}'s ``CS 121 Automation Tool'' \cite{Gong.A-2015a-CS-121-Automation} was a tool designed to semi-automate the grading of SQL assessments, again focusing on SQL DML statements. Interestingly, the system appears to be extensible and could thus potentially be modified to support grading of SQL DDL statements.

On the pure testing side of things:

• SVTe (Farré et al., 2008) tests the “correctness” of a schema, but focuses mainly on consistency of constraints.
• Ambler (n.d., http://www.agiledata.org/essays/databaseTesting.html) talks purely about testing the database functionality.

There appears to be very little work on automated grading of students’ schema definitions. No-one has used unit testing as a framework for doing so. What use of unit testing there is, is more focused on testing database /applications/ and effective generation of test cases:

• (Binnig et al., 2013) Generating test databases (fixtures) for OLTP applications.
• (Chays & Shahid, 2008) Generating test inputs for database applications based on analysing the queries used.
• (Marcozzi et al, 2012) Generating test inputs for database applications using a formal language (ImperDB).

No-one appears to have tried to automatically test whether an extant schema is consistent with its specification. This is almost more of a formal methods thing? These are more focused on /generating/ a conforming schema from the specification (e.g., Vatanawood & Rivepiboon, 2004; Luković et al., 2003), rather than checking that an already existing schema conforms. Also more focused on schema /transformations/ and evolution (e.g., Bench-Capon et al., 1998).

------------------------------------------------------------
NOTE: Checking structure of table Product.
TEST: [[ Product ]] 
    + OK
+++ PASSED: Table Product exists.
TEST: [[ Product.Product_code ]] 
    + OK
...
+++ PASSED: Table Product contains all the expected columns.
TEST: [[ Product.Product_code: data type is NUMBER | INTEGER ]] 
    + OK
...
+++ PASSED: All columns of table Product have data types compatible with the specification.
TEST: [[ Product.Product_code precision and scale = 8 (with scale 0) ]] 
    + OK
...
+++ PASSED: All columns of table Product have lengths compatible with the specification.
TEST: [[ Product.Product_code nullability should be N ]] 
    + OK
...
+++ PASSED: All columns of table Product have the expected nullability.
TEST: [[ Product PK ]] 
    + OK
+++ PASSED: Primary key of table Product exists.
TEST: [[ Product PK: Product_code ]] 
    + OK
+++ PASSED: Primary key of table Product includes (only) the expected columns.
TEST: [[ Product check constraint PRODUCT_STOCK_INVALID ]] 
    + OK
...
+++ PASSED: All constraints of table Product that should be are explicitly named.
NOTE: Testing constraints of table Product.
TEST: [[ Product.Stock_count accepts “0” ]] 
    + OK
TEST: [[ Product.Stock_count accepts “99999” ]] 
    + OK
TEST: [[ Product.Restock_level accepts “0” ]] 
    - FAILED! Column Product.Restock_level won’t accept legal value 0 [-0.5].
Failed asserting that false is true.
TEST: [[ Product.Restock_level accepts “99999” ]] 
    + OK
TEST: [[ Product.Minimum_level accepts “0” ]] 
    - FAILED! Column Product.Minimum_level won’t accept legal value 0 [-0.5].
Failed asserting that false is true.
TEST: [[ Product.Minimum_level accepts “653” ]] 
    + OK
TEST: [[ Product.List_price accepts “0” ]] 
    + OK
TEST: [[ Product.List_price accepts “99999.99” ]] 
    + OK
--- FAILED: 2 of 8 legal values tested were rejected by a CHECK constraint.

Showing you all comments on commits in this comparison.