Evaluation of automatic annotation of syntactic dependencies
Considering the importance that syntactic dependencies have been assuming in Natural Language Processing (NLP) tasks and, consequently, in linguistic studies focused on the automatic processing of languages, we present here a qualitative evaluation of a recently released gold-standard treebank for t...
| Autores: | , |
|---|---|
| Tipo de recurso: | artículo |
| Estado: | Versión publicada |
| Fecha de publicación: | 2024 |
| País: | Brasil |
| Institución: | Universidade Federal do Paraná (UFPR) |
| Repositorio: | Revista da ABRALIN (Online) |
| Idioma: | portugués |
| OAI Identifier: | oai:ojs.revista.abralin.org:article/2114 |
| Acceso en línea: | https://revista.abralin.org/index.php/abralin/article/view/2114 |
| Access Level: | acceso abierto |
| Palabra clave: | Anotação linguística Dependências sintáticas Treebank de língua portuguesa AValiação de dependências sintáticas |
| Sumario: | Considering the importance that syntactic dependencies have been assuming in Natural Language Processing (NLP) tasks and, consequently, in linguistic studies focused on the automatic processing of languages, we present here a qualitative evaluation of a recently released gold-standard treebank for the Portuguese language, with the objective of identifying (a) the linguistic patterns that are more difficult for automatic annotators, (b) the reasons that can lead them to make mistakes in these analyses, and (c) to expand the possibilities of dialogue between the linguistic studies and computational linguistics. The syntactic annotation was performed according to the guidelines of the Universal Dependencies (UD) project, and the evaluation of the annotation was performed using open source tools, in three steps: firstly, we performed an intrinsic evaluation of a syntactic dependencies annotation model, assuming that this type of evaluation reflects the consistency of the corpus annotation with which the model was trained; then, we detail the results of this evaluation, presenting the number of correct answers for each individual linguistic class, which gave us an overview of the linguistic difficulties for automatic learning and, also, information regarding the confidence in the automatic analysis of each linguistic classification. Finally, we selected the classes with the highest number of errors and analyzed all the wrong cases. The results suggest that, on the linguistic side, we can already count on consistent analyzes and in quantity, at least apparently, enough. As far as the quality of automatic parsers is concerned, the room for linguistic improvements is getting smaller and smaller. |
|---|