We crawl data from the Internet and the final step will consider how to store the data. If the amount of data is not large, you will not choose to store it in the database. Instead, you choose to store it in a file, such as a text file, CSV file, xls file, and so on. Because the file is easy to carry and intuitive.
Python as a glue language, of course, goes without saying. However, in the process of writing data, an error is often caused by having Chinese characters in the data source. Most people have scalp coding problems.
Let me first talk about encoding-related knowledge. There are many encoding methods: UTF-8, GBK, ASCII and so on.
The ASCII code is a set of character codes developed in the United States in the 1960s. It mainly regulates the relationship between English characters and binary bits. English vocabulary is simple in composition and consists of 26 letters. One byte can be used to represent a letter symbol. Add a variety of symbols, use 128 characters to meet the encoding requirements.
Different countries have different languages. At the same time, the number of text components is much larger than English letters. According to incomplete statistics, the number of Chinese characters is approximately 100,000, and 3,000 Chinese characters are used daily. Obviously, ASCII encoding can not meet the demand. Therefore, Chinese characters are encoded in GBK and two bytes are used to represent a Chinese character. Simplified Chinese encoding is GBK2312.
What is UTF-8 encoding? This requires first talking about Unicode. The purpose of Unicode is to unify various encodings. Because each country has its own coding method. If one encoding encoding is used, another encoding decoding is used. This can cause garbled conditions. But Unicode is just a set of symbols. It only specifies the binary code of the symbol, but it does not specify how this binary code should be stored. UTF-8 is the most widely used Unicode implementation on the Internet.
Therefore, if we want to write data to a file, it is better to specify the encoding format as UTF-8.
In the Python standard library, there is a library called csv that handles csv read and write operations. Specific examples of use are as follows:
Import csv
Import codecs
# codecs is a natural language code conversion module
fileName = 'PythonBook.csv'
# Specify the encoding as utf-8, avoid Chinese garbled writing csv file
With codecs.open(fileName, 'w', 'utf-8') as csvfile:
# Specify the head display item of the csv file
Filednames = ['title', 'author']
Writer = csv.DictWriter(csvfile, fieldnames=filednames)
Books = []
Book = {
'TItle': 'Swordsman',
'author': 'Jin Yong',
}
Books.append(book)
Writer.writeheader()
For book in books:
Try:
Writer.writerow({'title':book['TItle'], 'author':book['author']})
Except UnicodeEncodeError:
Print("Encoding error, this data cannot be written to file, ignore this data directly")
This method writes data to the CSV file line by line, so it is less efficient. If you want to bulk write data to a CSV file, you need to use pandas library.
Pandas is a third-party library, so it needs to be installed before use. Installation via pip is the simplest and most convenient.
Pip install pandas
The usage of bulk writing data using pandas is as follows:
Import pandas as pd
fileName = 'PythonBook.csv'
Number = 1
Books = []
Book = {
'TItle': 'Swordsman',
'author': 'Jin Yong',
}
# pandas will write 50 pieces of data to a file each time there are enough books.
Books.append(book)
Data = pd.DataFrame(books)
# write csv file, 'a+' is the append mode
Try:
If number == 1:
Csv_headers = ['title', 'author']
Data.to_csv(fileName, header=csv_headers, index=False, mode='a+', encoding='utf-8')
Else:
Data.to_csv('fileName, header=False, index=False, mode='a+', encoding='utf-8')
Number = number + 1
Except UnicodeEncodeError:
Print("Encoding error, this data cannot be written to file, ignore this data directly")
Replace 221 Lever Connector,Lever Connectors,Compact Connectors,Compact Splicing Connector
Guangdong Ojun Technology Co., Ltd. , https://www.ojunconnector.com