ReLDI-NormTagNER-hr 2.0 is a manually annotated corpus of Croatian tweets. It is meant as a gold-standard training and testing dataset for tokenisation, sentence segmentation, word normalisation, morphosyntactic tagging, lemmatisation, and named entity recognition of non-standard Croatian. Each tweet is also annotated for its automatically assigned standardness levels (T = technical standardness, L = linguistic standardness). As an update to version 1.1, 2.0 adds annotations for named entities. Authors Nikola Ljubešić, Tomaž Erjavec, Maja Miličević, Tanja Samardžić Availability For local use, a full-text version of...

This package is a Java reimplementation of four previously published stemming algorithms for Serbian and Croatian: The greedy and the optimal subsumption-based stemmer for Serbian, by Vlado Kešelj and Danko Šipka A refinement of the greedy subsumption-based stemmer, by Nikola Milošević A "Simple stemmer for Croatian v0.1", by Nikola Ljubešić and Ivan Pandžić All the stemmers expect the input text to be formatted in UTF-8. Their outputs are also UTF-8 encoded. Author Vuk Batanović Availability The package and a more extensive documentation can be downloaded...

A tool for automatic lemmatisation (returning the base or dictionary form of an inflected word). The tool looks up the hrLex/srLex lexicons and uses a predictive model for lemmatising OOVs (out of vocabulary words) which was trained on available corpora and lexicons. Author Nikola Ljubešić Availability The lemmatiser is freely available in three forms: For local use, the code and models of the lemmatiser can be downloaded from this GitHub repository. The lemmatiser web service can be used online, via our web interface that...

hr500k is a reference training corpus of Croatian that consists of 900 documents divided into 24 794 sentences, or 506 457 tokens. It is an extension of previous training corpora for Croatian, such as SETimes.HR and SETimes.HR+. The corpus is manually annotated on the following levels: Token, sentence, and document segmentation Morphosyntax Lemmas Dependency syntax Semantic roles Named entities The entire corpus was annotated with regard to morphosyntax and lemmas. The set of morphosyntactic tags used in the corpus follows the revised...

hrLex is an inflectional lexicon of Croatian. The size of the lexicon is 103,077 lemmas, or 4,970,520 surface forms. Each entry in the lexicon consists of a (word form, lemma, MSD, absolute frequency, in-million frequency) quintuple, e.g.: (ženu, žena, Ncfsa, 54158, 0.038746). The frequencies were estimated on the Croatian web corpus hrWaC. The set of morphosyntactic tags used in the lexicon follows the MULTEXT-East V5 tagset for Croatian, available here. Authors Nikola Ljubešić, Filip Klubička Availability For local use, hrLex can be downloaded as a raw text...

hrWaC is a web corpus collected from the .hr top-level domain. The 2.1 version of the corpus contains 1.4 billion tokens. The corpus is automatically annotated on the diacritic restoration, morphosyntax and lemma layers. The dependency syntax layer will be added in version 2.2. The set of morphosyntactic tags used in the corpus follows the revised MULTEXT-East V5 tagset for Croatian and Serbian, available here. Authors Nikola Ljubešić, Filip Klubička Availability For local use, a full-text version of hrWaC can be downloaded here. hrWaC can also...

A tool for automatic diacritic restoration on text with potentially missing diacritics (e.g. it turns kuca into kuća if necessary). Reported accuracy of the tool: 99.5% on standard language and 99.2% on non-standard language. Authors Nikola Ljubešić, Tomaž Erjavec, Darja Fišer Availability The tool is freely available in two forms: The code and models of the tool can be downloaded from this GitHub repository. Our web service can be accessed from of our Python library, which can also be downloaded from the CLARIN.SI GitHub...

A tool for automatic tokenisation (dividing text into words and sentences). It was engineered through iterative runs on representative datasets and features modes for both standard and non-standard language. Authors Nikola Ljubešić, Tomaž Erjavec Availability The tokeniser is freely available in three forms: For local use, the tokeniser can be downloaded from this GitHub repository. The tokeniser can be used online, via our web interface that can be found here. Our web service can be accessed from of our Python library, which can also...

A tool for automatic annotation on the morphosyntactic level. It is capable of tagging both Croatian and Serbian as models for both languages are present in the tool. The tagger is based on the CRF algorithm trained on a 500,000-token Croatian training corpus and the hrLex/srLex lexicons for each respective language. The set of morphosyntactic tags used in the corpus follows the revised MULTEXT-East V5 tagset for Croatian and Serbian, available here. Accuracies calculated on test sets for each language: Croatian: 92.53% Serbian: 92.33% Author Nikola...