PythonÍøÂçÅÀ³æ³öÏÖÂÒÂëÎÊÌâµÄ½â¾ö·½·¨


            ¹ØÓÚÅÀ³æÂÒÂëÓкܶà¸÷ʽ¸÷ÑùµÄÎÊÌ⣬ÕâÀï²»½öÊÇÖÐÎÄÂÒÂ룬±àÂëת»»¡¢»¹°üÀ¨Ò»Ð©ÈçÈÕÎÄ¡¢º«ÎÄ ¡¢¶íÎÄ¡¢²ØÎÄÖ®ÀàµÄÂÒÂë´¦Àí£¬ÒòΪ½â¾ö·½Ê½ÊÇÒ»Öµģ¬¹ÊÔÚ´Ëͳһ˵Ã÷¡£



ÍøÂçÅÀ³æ³öÏÖÂÒÂëµÄÔ­Òò



Ô´ÍøÒ³±àÂëºÍÅÀÈ¡ÏÂÀ´ºóµÄ±àÂë¸ñʽ²»Ò»Ö¡£

ÈçÔ´ÍøҳΪgbk±àÂëµÄ×Ö½ÚÁ÷£¬¶øÎÒÃÇץȡϺó³ÌÐòÖ±½ÓʹÓÃutf-8½øÐбàÂë²¢Êä³öµ½´æ´¢ÎļþÖУ¬Õâ±ØÈ»»áÒýÆðÂÒÂë ¼´µ±Ô´ÍøÒ³±àÂëºÍץȡÏÂÀ´ºó³ÌÐòÖ±½ÓʹÓô¦Àí±àÂëÒ»ÖÂʱ£¬Ôò²»»á³öÏÖÂÒÂë; ´ËʱÔÙ½øÐÐͳһµÄ×Ö·û±àÂëÒ²¾Í²»»á³öÏÖÂÒÂëÁË



×¢ÒâÇø·Ö





  Ô´Íø±àÂëA¡¢

  ³ÌÐòÖ±½ÓʹÓõıàÂëB¡¢

  ͳһת»»×Ö·ûµÄ±àÂëC¡£

  



ÂÒÂëµÄ½â¾ö·½·¨



È·¶¨Ô´ÍøÒ³µÄ±àÂëA,±àÂëAÍùÍùÔÚÍøÒ³ÖеÄÈý¸öλÖÃ

1.http headerµÄContent-Type

»ñÈ¡·þÎñÆ÷ header µÄÕ¾µã¿ÉÒÔͨ¹ýËüÀ´¸æÖªä¯ÀÀÆ÷һЩҳÃæÄÚÈݵÄÏà¹ØÐÅÏ¢¡£ Content-Type ÕâÒ»ÌõÄ¿µÄд·¨¾ÍÊÇ "text/html; charset=utf-8"¡£ 



2.meta charset



<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />  



3.ÍøҳͷÖÐDocument¶¨Òå







<script type="text/javascript"> 

if(document.charset){ 

 alert(document.charset+"!!!!"); 

 document.charset = 'GBK'; 

 alert(document.charset); 

} 

else if(document.characterSet){ 

 alert(document.characterSet+"????"); 

 document.characterSet = 'GBK'; 

 alert(document.characterSet); 

} 





ÔÚ»ñÈ¡Ô´ÍøÒ³±àÂëʱ£¬ÒÀ´ÎÅжÏÏÂÕâÈý²¿·ÖÊý¾Ý¼´¿É£¬´ÓÇ°Íùºó£¬ÓÅÏȼ¶ÒàÊÇÈç´Ë¡£

ÒÔÉÏÈýÕßÖоùûÓбàÂëÐÅÏ¢ Ò»°ã²ÉÓÃchardetµÈµÚÈý·½ÍøÒ³±àÂëÖÇÄÜʶ±ð¹¤¾ßÀ´×ö



°²×°£º pip install chardet



¹Ù·½ÍøÕ¾£º http://chardet.readthedocs.io/en/latest/usage.html



Python chardet ×Ö·û±àÂëÅжÏ



ʹÓà chardet ¿ÉÒԺܷ½±ãµÄʵÏÖ×Ö·û´®/ÎļþµÄ±àÂë¼ì²â ËäÈ»HTMLÒ³ÃæÓÐcharset±êÇ©£¬µ«ÊÇÓÐЩʱºòÊDz»¶ÔµÄ¡£ÄÇôchardet¾ÍÄÜ°ïÎÒÃÇ´óæÁË¡£

chardetʵÀý







import urllib 

rawdata = urllib.urlopen('//www.zwyuanma.com/').read() 

import chardet 

chardet.detect(rawdata) 

{'confidence': 0.99, 'encoding': 'GB2312'} 





chardet¿ÉÒÔÖ±½ÓÓÃdetectº¯ÊýÀ´¼ì²âËù¸ø×Ö·ûµÄ±àÂë¡£º¯Êý·µ»ØֵΪ×ֵ䣬ÓÐ2¸öÔªËØ£¬Ò»¸öÊǼì²âµÄ¿ÉÐŶȣ¬ÁíÍâÒ»¸ö¾ÍÊǼì²âµ½µÄ±àÂë¡£



ÔÚ¿ª·¢×ÔÓÃÅÀ³æ¹ý³ÌÖÐÈçºÎ´¦Àíºº×Ö±àÂë?

ÏÂÃæËù˵µÄ¶¼ÊÇÕë¶Ôpython2.7,Èç¹û²»¼Ó´¦Àí£¬²É¼¯µ½µÄ¶¼ÊÇÂÒÂ룬½â¾öµÄ·½·¨Êǽ«html´¦Àí³ÉͳһµÄutf-8±àÂë Óöµ½windows-1252±àÂ룬ÊôÓÚchardet±àÂëʶ±ðѵÁ·Î´Íê³É







import chardet 

a='abc' 

type(a) 

str 

chardet.detect(a) 

{'confidence': 1.0, 'encoding': 'ascii'} 

 

 

a ="ÎÒ" 

chardet.detect(a) 

{'confidence': 0.73, 'encoding': 'windows-1252'} 

a.decode('windows-1252') 

u'\xe6\u02c6\u2018' 

chardet.detect(a.decode('windows-1252').encode('utf-8')) 

type(a.decode('windows-1252')) 

unicode 

type(a.decode('windows-1252').encode('utf-8')) 

str 

chardet.detect(a.decode('windows-1252').encode('utf-8')) 

{'confidence': 0.87625, 'encoding': 'utf-8'} 

 

 

a ="ÎÒÊÇÖйúÈË" 

type(a) 

str 

{'confidence': 0.9690625, 'encoding': 'utf-8'} 

chardet.detect(a) 

# -*- coding:utf-8 -*- 

import chardet 

import urllib2 

#ץȡÍøÒ³html 

html = urllib2.urlopen('//www.zwyuanma.com/').read() 

print html 

mychar=chardet.detect(html) 

print mychar 

bianma=mychar['encoding'] 

if bianma == 'utf-8' or bianma == 'UTF-8': 

 html=html.decode('utf-8','ignore').encode('utf-8') 

else: 

 html =html.decode('gb2312','ignore').encode('utf-8') 

print html 

print chardet.detect(html) 





python´úÂëÎļþµÄ±àÂë

pyÎļþĬÈÏÊÇASCII±àÂ룬ÖÐÎÄÔÚÏÔʾʱ»á×öÒ»¸öASCIIµ½ÏµÍ³Ä¬ÈϱàÂëµÄת»»£¬Õâʱ¾Í»á³ö´í£ºSyntaxError: Non-ASCII character¡£ÐèÒªÔÚ´úÂëÎļþµÄµÚÒ»ÐÐÌí¼Ó±àÂëָʾ£º







# -*- coding:utf-8 -*- 

 

print 'ÖÐÎÄ' 



ÏñÉÏÃæÄÇÑùÖ±½ÓÊäÈëµÄ×Ö·û´®ÊÇ°´ÕÕ´úÂëÎļþµÄ±àÂë'utf-8'À´´¦ÀíµÄ

Èç¹ûÓÃunicode±àÂë,ÒÔÏ·½Ê½£º



s1 = u'ÖÐÎÄ' #u±íʾÓÃunicode±àÂ뷽ʽ´¢´æÐÅÏ¢



decodeÊÇÈκÎ×Ö·û´®¾ßÓеķ½·¨£¬½«×Ö·û´®×ª»»³Éunicode¸ñʽ£¬²ÎÊýָʾԴ×Ö·û´®µÄ±àÂë¸ñʽ¡£

encodeÒ²ÊÇÈκÎ×Ö·û´®¾ßÓеķ½·¨£¬½«×Ö·û´®×ª»»³É²ÎÊýÖ¸¶¨µÄ¸ñʽ¡£

¸ü¶àÄÚÈÝÇë²Î¿¼×¨Ì⡶pythonÅÀÈ¡¹¦ÄÜ»ã×Ü¡·½øÐÐѧϰ¡£

ÒÔÉϾÍÊDZ¾ÎĵÄÈ«²¿ÄÚÈÝ£¬Ï£Íû¶Ô´ó¼ÒµÄѧϰÓÐËù°ïÖú£¬Ò²Ï£Íû´ó¼Ò¶à¶àÖ§³ÖÖÐÎÄÔ´ÂëÍø¡£