python - Tagging spanish text with Unicode characters not possible with NLTK? -

-

I'm trying to parse some Spanish sentences that contain non-Eski characters (for example. Peilicula (film), anension (meditation), etc.).

I am reading lines from a file which is encoded with UTF-8. Here is a sample of my script: nltk.corpus as imported as unit from import cess_esp to import nltk import system <- p> 

  # - * - coding: utf- 8 - * - nltk import BigramTagger as b = f (code_sentences', encoding = 'utf-8' W + ') for line in ITER (F): output_line = "current line content before tagging-> + str (line.decode (' utf-8 ',' replace ')) print output_line results_file.write (output_line. Encode ('utf8')) output_line = "Ungram Tagger->" print output_line results_file.w Rite (output_line) s = line.decode ('utf-8', 'replace') output_line = tagger.uni.tag (s.split ()) print output_line results_file.write (str (output_line) .encode ('utf8' )) F.close () results_file.close ()

On this line: 

  output_line = tagger.uni.tag (ssplit () )

I am getting this error: 

  /usr/local/lib/python2.7/dist-packages/nltk- 2.0.4- Py2.7.egg / nltk / tag / sequential.py: 138: UnicodeWarning: Unicode equivalent comparison Unicode failed to change both debates - interpreted them as having uneven return self._context_to_tag.get (reference)

Here are some output days for a simple sentence Received: 

  Existing line content before tagging- & gt; Tengo una cuza u cad queen hablo a ethenian mejan en la line media horra y corton la laalada !! Unigrammed Tagger- & gt; [(Utengo ',' vmip1s0 '), (' u'na ',' dfff) ', (ukua', 'ncfsfxa'), (u'i ',' cc '), (u'kada ',' 0 CC 0 '), (U.K.,' Preet CNNA '), (U'HoLlo', 'VMP1S'), (U'A ',' SPS & '), (Utensi \ Xf3n ', none) (U'me', 'pp1cs000'), (you'jan ',' vmip3p0 '), (u'n', 'espind'), (yula ',' dff 0, 0), (U'line '(U'I', 'CC'), (U 'Karta', None), (Yula ',' Da0fs0 '), (u'llamada !!', none )]

If I understood from right ... the process is correct ... I have utf-8 Change the code, and then Unicode to UTF-8 again ... ... I do not understand this error.

What is wrong with any idea? < / P>

Thank you, Alejandro

Edit: Problems get ... Basically the Spanish SAS_PAP corpus is encoded with Latin-2 encoding correctly. See the code below to be able to train 

  tag_saint = (has been sent to [word.decode ('Latin2'), tag) (word, tag) in cass.tagged_sents ()) tagger = UT (Tag_Assent) # Training A Tagger

Corpus encoding to use the class to ask for a better method, thus you do not need to know it beforehand.

Maybe there is something wrong with your tagger object or how your file has been read. I have rewritten the part of my code and it runs without error: 

  # - * - coding: utf-8 - * - import urlib2, nltk as ct from nltk. Corpus import codec tagger from cess_esp = ut (cess.tagged_sents ()) URL = 'https://db.tt/42Lt5M5K' wings = urllib2.urlopen (url) .read () bt as nltk import from BigramTagger ut Imported from NTTK import UnigramTagger as word_tokenize. . ('Tagger.out' 'W', 'UTF8') for the fin.split line in decode ('UTF8') fout = codecs.open ('\ n'): Print & gt; & Gt; Fout, current line content before "tagging->,", line print & gt; & Gt; Fout, "Unigram tagger- & gt;", print & gt; & Gt; Fout, tagger.tag (word_tokenize (line)) print & gt; & Gt; [Out]:

Comments

Post a Comment

Popular posts from this blog

import - Python ImportError: No module named wmi -

-
I followed the instructions for downloading WMI for Python When I try to run the code for C.Win32_Service (StartMode = "Auto", state = "Stopped") import wmi c = wmi.WMI (): if Raw_input ("restart% s?"% S.Caption) .upper () == "Y": s.StartService () I got the error traceback (most recent call end): file ". \", Line 1, edit: I am using Python 2.7.6 EDIT2: I am running 64-bit Windows 2008 R2, and I've downloaded WMI-1.4.9.zip (md5). I removed the content and put it in D: Python \ Tools \ Scripts I executed dragon setup.py.install I said D : \ Python \ tools \ scripts to% PATH% and when I execute the code for C.Win32_Service (StartMode = "Auto", state = "Stopped") import wmi c = Wmi.WMI (): If raw_input ("Restart% s?"% S. Caption). () == "Y": s.StartService () error me Gets traceback (most recent call final): File ". See \", line 1, & lt; Modu...
Read more

Editing Python Class in Shell and SQLAlchemy -

-
I am working on the terminal on a shell script after this tutorial SQLAlchemy tutorial. I need to type gt; & Gt; & Gt; Sqlalchemy import column, integer, string & gt; & Gt; & Gt; Class user (base): __tablename__ = 'users' id = column (integer, primary_key = true) name = column (string) absolute name = column (string) password = column (string) def __repr __ (self): return "& lt ; User (name = '% s', absolute name = '% s', password = '% s') & gt; "% (Self.name, self.fullname, self.password) After the problem I have typed the password = column (string) I hit twice and changed .... >> I then took everything back but then an error was thrown because the class already exists ... I am not completely sure how to fix it. How do I open that slip in the shell script and edit it (Add DF to DRR) The error thrown below: / user / giripeters / TFSQLLX / Lib / python2.7 / site-packages / sqlalchemy / ext / decla...
Read more

lua - HowTo create a fuel bar -

-
I would like to create a fuel bar (empty with full tank). When the user touches the screen, the fuel once Dairejh these is my code: lifeBar = display.newImage ( "fuel_bar1.png") lifeBar.anchorX = 0 lifeboats. ======================== Lifetime Y = 37 screens Group: The Insert function (called Laifbar) However, I fuel Kansmoshn (equivalent to "pressed" variable to reduce the fuel tank "true") Have used. When pressed == is true, it means that the user is touching the screen: processed (fuel) processed () if life & gt; 0 and pressed == true remains = life - life 1 Life = life =. Wide-1 remains Value.text = string.format ("% d", life) End of this This function is active as the entrance event: runtime: addEventListener ( "Antrfrem" fuel Snsadna) it is very well, but my question is: pressed variable each time 5 seconds (once = = True) How is it possible to reduce the vitality width? I tried to add it to my touch even...
Read more