next

1. Relevance: Is it paper highly relevant to COLING?
2. Readability/clarity: may be the paper well-structured and well-written?
3. Data/code supply: may be the data/code (as appropriate) open to the investigation community or perhaps is here a reason that is compelling why this isn’t feasible?
4. Analysis: In the event that paper surely could reproduce the outcomes of the earlier in the day work, does it plainly set down exactly exactly what would have to be filled in to do therefore? It clearly identify what information was missing/the likely causes if it wasn’t able to replicate the results of earlier work, does?
5. Generalizability: Does the paper rise above replicating the outcomes regarding the initial to explore whether or not they could be reproduced an additional environment? Instead, in instances of non-replicability, does the paper talk about the wider implications of the outcome?
6. Informativeness: To exactly exactly exactly what degree does the analysis reported into the paper deepen our comprehension of the methodology utilized or the issue approached? Will the information into the paper assistance professionals with regards to selection of technique/resource?
7. Significant comparison: as well as distinguishing the experimental outcomes being replicated, does the paper motivate why these specific email address details are a important target for reproduction and exactly just what the long term implications are of the having been reproduced or been discovered to be non-reproducible?
8. General suggestion: there are lots of submissions that are good for slots at COLING 2018; essential could it be to feature this 1? Will people learn great deal by reading this paper or seeing it presented? Please be decisive—it is way better to change from other reviewers rather than grade every thing in the centre.

Site paper

Documents in this track provide a language resource that is new. This may be a corpus, but in addition might be an annotation standard, device, an such like.

1. Relevance: Is it paper relevant to COLING? Will the resource presented likely be of good use to your community?
2. Readability/clarity: Through the real method the paper is created, are you able to inform the way the resource had been produced, the way the quality of annotations (if any) had been examined, and just why the resource ought to be of great interest?
3. Originality: Does the resource fill a need when you look at the current assortment of available resources? Remember that originality could possibly be within the range of language/language variety or genre, within the design associated with annotation scheme, within the scale of this resource, or nevertheless other parameters.
4. Site quality: what type of quality control had been completed? If appropriate, ended up being inter-annotator contract measured, and in case so, with appropriate metrics? Otherwise, the other assessment ended up being carried out, and how acceptable were the outcomes?
5. Site accessibility: could it be simple for researchers to down load or otherwise access the resource so that you can make use of it in their own personal work? As to the level can perhaps work according to this resource be provided? answers to add: Yes, i’ve confirmed
6. Metadata: Do the authors explain whoever language use is captured into the resource also to just just exactly what populations experimental outcomes based in the resource might be generalized to? In the event of annotated resources, will be the demographics of this annotators also characterized?
7. Significant contrast: may be the brand new resource situated with regards to current work with the industry, including comparable resources it took motivation from or improves on? Could it be clear what exactly is unique about the resource?
8. General suggestion: there are lots of submissions that are good for slots at COLING 2018; essential could it be to feature this 1? Will people learn great deal by looking over this paper or seeing it presented? Please be decisive—it is way better to change from other reviewers rather than grade every thing in the centre.

Position paper

A situation paper presents a challenge to mainstream reasoning or perhaps a futuristic vision that is new. It may open a brand new area or unique technology, propose changes in existing research, or provide a brand new pair of ground guidelines.

1. Relevance: Is it paper highly relevant to COLING?
2. Readability/clarity: Is it clear exactly exactly just what the career is the fact that the paper is arguing for? Would be the arguments because of it laid call at a way that is understandable?
3. Soundness: Are the arguments presented when you look at the paper appropriate and coherent? May be the eyesight well-defined, with success criteria? (Note: it ought to be feasible to provide a top score right here even although you don’t buy into the place taken because of the writers)
4. Imagination: How bold or novel may be the place drawn in the paper? Does it express well-thought through and innovative ground that is new?
5. Range: How scope that is much brand brand new scientific studies are exposed by this paper? Just just just What effect could it have on current areas and concerns?
6. Significant contrast: could be the paper well-situated pertaining to past work, both place documents (taking the exact same or opposing side for a passing fancy or comparable problems) and relevant theoretical or experimental work?
7. Substance: Does the paper have sufficient substance for a paper that is full-length? Could be the problem adequately crucial? Would be the arguments sufficiently varied and thoughtful?
8. General suggestion: there are numerous submissions that are good for slots at COLING 2018; essential could it be to feature this 1? Please be decisive—it is way better to vary from other reviewers rather than grade every thing in the centre.

A study paper provides an organized summary of the literary works up to now for a particular subject that assists the reader comprehend the kinds of concerns being inquired about this issue, the different approaches which were used, the way they relate solely to one another, and just what further research areas they open. A conference-length study paper should be in regards to a adequately concentrated subject that it could try this effectively with into the web web page limits.

1. Relevance: may be the paper highly relevant to COLING?
2. Readability/clarity: could be the paper generally speaking an easy task to follow and well organized?
3. Organization: Does the paper arrange the literature that is relevant a narrative and recognize common strands of inquiry?
4. Scope: Does the paper determine a fairly concentrated area to review?
5. Thoroughness: because of the region identified to survey, does the paper address all the literature that is relevant? Could be the literary works evaluated represented accurately?
6. Outlook: Does the paper recognize areas for future work and/or demonstrably mention exactly what is certainly not yet managed inside the literary works surveyed?
7. Context: Does the paper situate research that is current within its historic context? (We don’t expect papers in the first place Pa?ini, yet during the exact same time one thing that just cites work from 2017 most likely does not capture exactly how current work pertains to the larger image.)