Bash utf8 encode string ubuntu. Follow asked Sep 19, 2012 at 6:27.

Bash utf8 encode string ubuntu #!/bin/bash username=username password=password echo Encoding username/password Bash scripting: Encoding string to base64 into a variable [duplicate] Ask Question Asked 3 years, 9 months ago. This is correct. Modified 3 years, BASH: Convert Unicode Hex to String. txt, utf8 format. txt another. Hot Network Questions How can I control LED brightness from an MCU without using PWM You have to ensure that the issued sed command is UTF-8 encoded. When you have encoding values you aren't familiar with, take the time to check out the ranges in the common encoding schemes to determine what encoding a value belongs to. Let’s see what it does: 00000000: e2 If your locale's charset (see output of locale charmap) is not UTF-8 (but can represent all the characters in the encoded string), add a | iconv -f UTF-8. I don't know what you mean by "how many strings are represented by a bash or python script". It is used to encode Unicode characters in groups Pardus, Ubuntu, Mint. String objects in Java use the UTF-16 encoding that can't be modified *. Possibly related: stackoverflow. You can do this with iconv "by hand", or you can use file: $ file utf8. bash ascii to hex. Fortunately, current versions of mysql will automatically upgrade from varchar(n) to the text data type if you attempt to alter a varchar(n) field to larger than the feasible byte size (while issuing a warning). Does anybody know how I can achieve this? http; post; curl; encoding; webserver; Share. But I guess the commands are not executing properly. Hot Network Questions Merge two Short Answer. Modified 3 years, 9 months ago. If you want a deep dive into converting numbers to chars: look here to see an article from Greg's wiki (BashFAQ) about ASCII encoding in Bash! Don't rely on regexes: JSON has some strange corner-cases with \u escapes and non-BMP code points. Usually, this stuff is really easy to do with sed, but I'm having trouble at getting sed to recognize the special characters. You see it if you bash into it and execute locale: sudo docker exec -i -t yowsup3 Python3 utf8 codecs not decoding as expected in Docker ubuntu: I'm trying to write a bash script to convert all special characters inside a file (é, ü, ã, etc) into latex format (\'e, \"u, \~a, etc). For it to print it UTF-8 encoded regardless of the user's locale, you'd need LC_CTYPE=en_US. e. So far I have tried a simple bash file containing python -m base64 -d $1 but this command expects a filename not a string. I run the follow In Debian you can also use: encguess: $ encguess test. bas64 /path/to/file > output. You could also try this: echo $var | iconv -f What you have isn't quite "UTF-8 text". parse, encode urls UTF8 - shells script. sub This script will try to fix the encoding of any I have a file in us-ascii format that I want to pass to UTF-8. UTF-8 and then rebooting. results in: Zvaigzdes aukstybej uzges or auoOUA The most common encodings for Swedish are iso-8859-1 and utf-8. Provided by: libunicode-utf8-perl_0. 54. utf8" Notice the difference between ". The "Q" marks Quoted-Printable mode, and "B" marks Base64 mode. Ask Question Asked 3 years, 10 months ago. A customer sent me a csv file that Linux recognizes as "unknown-8bit" ( Any ideas on what I can do to make this work with paths with UTF8 chars and whitespace? How can fix UTF-8 string usage in bash? 1. Stack Overflow for Teams Where developers & technologists share private knowledge with coworkers; Advertising & Talent Reach devs & technologists worldwide about your product, service or employer brand; OverflowAI GenAI features for Teams; OverflowAPI Train & fine-tune LLMs; Labs The future of collective knowledge sharing; About the company I would like to write a bash script to decode a base64 string. If this doesn't matter to you (i. echo 'latin string' | iconv -t utf-8 > yourfileInUTF8 Edit: As suggested by damadam I removed the -f option since the string typed in terminal uses the default encoding-f, --from-code=NAME encoding of original text How do I set my . If it cannot represent The default character encoding is UTF-8 (Unicode), though almost all (quite possibly all on a default install) file names are regular ASCII characters, common to most encodings. You actually want plain UTF-8 text as output, as it's what Linux uses for "special characters" everywhere. csv outputs the chinese characters fine on terminal. This is because it is a binary-encoded native JavaScript string, not a UTF8-encoded string. 59-1build1_amd64 NAME Unicode::UTF8 - Encoding and decoding of UTF-8 encoding form SYNOPSIS use Unicode::UTF8 qw[decode_utf8 I have a question about converting UTF-8 to CP1252 in Ubuntu with PHP or SHELL. txt Encode with line wrapping: echo 'Long string. UTF-8 ENV LANGUAGE en_US:en ENV LC_ALL en_US. Yes, those are lower. txt should do it. UTF-8 ENV LANG en_US. txt > another. In this tutorial, we’ll discuss converting different Unicode types to it's a real 80x25 textmode terminal, so you can't use more than 256 characters. /%&/g' tr -d \\n removes the linefeeds that are added by xxd -p after every 60 characters. Improve this question. Inputting some text and then using Enter and then Ctrl+D to signify end of file then causes md5sum to spit out the MD5 of the raw text you entered (including that Enter, it's a CR, IIRC). Commented Mar 26 Save this script as fix-encoding. Specifically, the command uses printf to print the result of the encode_base64() function, formatted as a string. I am French so in my language I have letters like: é è à . To encode and decode text files using base64 in Bash, use the following syntaxes: Encode text incorporating pipe command: echo -n 'Sample string' | base64 Encode with text file: base64 file1. The html sample in the original post contains em/en dash characters which are not part of the ISO-8859-1 layout, so it certainly uses another encoding, probably Windows-1252 (which is a superset of ISO-8859-1 including the dash characters) since the OP reported to use Ubuntu through the Windows subsystem layer. ) The -A n flag is short for --address-radix=n, with n being short One of the servers I quite often ssh to uses western encoding instead of utf-8 (and there's no way I can change that). 2 or above)'s printf %b '\u2586' will print the U+2586 ( ) character in the locale charset (0xe2 0x96 0x86 if it's UTF-8, 0xa8 0x7d if it's GB18030, etc) or as \u2586 if the locale has no such character. $ echo 123456789 | base64 MTIzNDU2Nzg5Cg== And then when I did the same on base64encode, I got this s = "├" (in your UTF-8 encoded source file) assigns the character \u251C to the first position of s, a UTF-8 encoded string. sh "this is my test string" # => U2FsdGVkX18fjSHGEzTXU08q+GG2I4xrV+GGtfDLg Using Ubuntu, encrypt passwords in In my Python script running at the command line on Ubuntu, it's selecting UTF8-encoded content from a MySQL database. txt -o output_ascii. ; The od program is the "octal dump" program. Alternatively, change your terminal and shell settings to UTF-8 UTF-8 (Unicode Transformation Format) is an 8-bit Unicode conversion format. is in file testutf8. base64 - base64 encode/decode data and print to standard output Encode hex echo "hello" | xxd -p 68656c6c6f0a Decode hex echo "68656c6c6f0a" | xxd -r -p Encode base64 echo "hello" | . With encoded string, you can pipe an echo command into base64 as given a UTF-8 encoded file called a which contains this example: We’re not a different species “All alone?” Jeth mentioned. encode('utf-8') b'Six of one, \xc2\xbd dozen of Bash stores strings as byte strings, and performs operations according to the current LC_CTYPE setting. utf8 There is an existing post on Unix & Linux about including Unicode characters in the Bash prompt, but the method it gives for using the UTF-16 code (syntax \uXXXX) doesn't work for me. If not in UTF-8, you do like this: The UTF-8 (Unicode Transformation Format) is an 8-bit Unicode conversion format. Use framebuffer console if you want real utf-8. UTF-8 The usual docker images don't specify a locales. Possibly through an erroneous line in my . Convert string to hexadecimal on command line. ; file only guesses at the file encoding and may be wrong (especially in cases where special characters only appear In this guide, we will describe what character encoding and cover a few examples of converting files from one character encoding to another using a command line tool. For instance, on macOS you can choose the terminal encoding and optionally set the locale environment variables at terminal startup in That means that the UTF8 encoding is: e2 98 a0 Or, in HEX to avoid errors: 0xE2 0x98 0xA0. txt US-ASCII As it is a perl script, it can be installed on most systems, by installing perl or the script as standalone, in case perl has already been installed. If you haven't used the encoding in Gedit before, scroll down to Add @TimTisdall Oh, you're talking about upper bounds. After the conversion to UTF-16 you can hexdump in groups of 4 bytes and post-process the dump with sed to insert the leading \u. Running command: iconv -f UTF8 -t US-ASCII//TRANSLIT testutf8. bash_profile, the only thing that helped was: localectl set-locale en_US. Percent-encode all characters except ASCII alphanumeric characters in How can a text string be turned into UTF-8 encoded bytes using Bash and/or common Linux command line utilities? For example, in Python one would do: "Six of one, ½ dozen of the other". Then finally, we will look at how to convert several files I'm having some trouble with displaying UTF-8 characters correctly on bash. Additionally, specifying UTF-16BE or UTF-16LE doesn't prepend a byte-order mark, so I first convert to UTF-16, which uses a platform In bash, how can I convert an hex-encoded string like this: 2e2f65202d6b2022616622 or \x2e\x2f\x65\x20\x2d\x6b\x20\x22\x61\x66\x22 (or something similiar) Convert string to encoded hex. Could it be an issue with my zshprofile or bash_profile? In my zshrc profile: vim ~me/. , you aren't converting strings represented in Unicode from another system or are fine with JavaScript's native UTF This means that files and strings which contain only 7-bit ASCII characters have the same encoding under both ASCII and UTF-8 . txt will. Question - will it not base64 -d converts into byte array ? The string dGVzdA== is base64 encoded string test. Therefore by association, the base64 encoding maps in a 1:1 fashion to its hex string representation. As @JdeBP said, the terminal does not use the locale environment variables to determine its encoding. Hot Network Questions How to eliminate variables in ODE system? @Ehryk - I don't think that there is much to have to come to agreement on. A bash solution is optimal, but a C/C++ code solution or guide My quick poke at the --help for md5sum demonstrates that the command:. Bash - Hex to String. utf8" and ". Displaying UTF8 stings in Ubuntu's terminal with a Python script. ' | base64 -w 0 Decode text file: base64 -d encodedfile. I want to encode them to base64 with the format of username:password. Modified 2 years, (this time on Ubuntu, but that shouldn't matter): cat any. 0: $ recode -l | grep -iw html HTML-i18n 2070 RFC2070 HTML_4. The displayed strings have an encoding problem, as they don't show the Convert a Base64 encoded UTF8 file with Unix line endings to Base64 encoded Latin 1 file with Dos line gc -en string in. txt How do I convert this UTF-8 text to text with special characters in bash? What you have isn't quite "UTF-8 text". For example, the string: Žvaigždės aukštybėj užges or äüöÖÜÄ. Hot Network Questions Possibly file detects some other content type for those files. Right now Python uses UTF-8 for the FS encoding but sticks to ASCII for the default encoding :-(>>> import sys >>> sys. sh encode example. sh myfile. bash_profile was: # ⚠️ Better not do this export LANG="en_US. 4. I can't share the url but this is how I am sending the request. From the bash shell. I am trying to use the following code where hola. Share. Technically bash (4. I am using a Mac if that's relevant. txt ascii. iconv will use whatever input/output encoding you specify regardless of what the contents of the file are. You can try to use the file command to detect a file's type/encoding. UTF-8 #export LOCALE=UTF-8 export LC_CTYPE=C export LANG=C cat myfile. – Ruud Helderman. UTF-8 printf (or any other locale available on the FROM ubuntu RUN apt update && apt -y install locales && locale-gen en_US. Check it with locale. You can do this with a script or small program that selectively interprets JSON string character escapes. My chaotic back and forth running would include trying an other browser, check caching tools, deployment and so on. getdefaultencoding() 'ascii' >>> sys. zshrc. txt Decoding Strings. the codepoint of iso8859-1 characters are the same as in The response of the below curl command is a json and it has some japanese characters which it is showing in utf-8. Environment : Ubuntu 10. For example, in Ubuntu/Debian, it is: sudo dpkg-reconfigure console-setup. ; The -n flag tells echo to not generate a new line at the end of the "Hello". Stack Exchange network consists of 183 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers. 3; a file csv with letters (œ, à, ç) Methods used : $ perl -MMIME::Base64 -ne 'printf "%s",encode_base64($_)' <<< "Hello, World!" SGVsbG8sIFdvcmxkIQo= The -n flag is for reading the input line by line, whereas the -e flag is for executing the provided command enclosed in quotes for each line. How to printf "`" and "%" character inside array on aliasses? Hot Network Questions Alternative (to) freehub body replacement for FH-M8000 rear hub Denial of boarding or ticketing issue - best path forward End An example of basic conversion from source encoding to target coding as the output: $ iconv -f [SOURCE_ENCODING] -t [TARGET_ENCODING] [INPUT_FILE] -o [OUTPUT_FILE] By using the iconv command, we convert a Txt file written in UTF-8 to a text file written in ASCII by operating: $ iconv -f UTF-8 -t ASCII input_utf8. print unicode characters in terminal with python 3. txt: UTF-8 Unicode text If that is not enough, I can offer you the Python script I wrote for this answer here, which scans complete files and tries to decode them using a specified character set. Then, I want to print the string to the console. Also check the sources for new String, getBytes, InputStreamReader, OutputStreamReader without Charset. txt. So there is no need to restart bash: just set the LC_CTYPE or LC_ALL variable to your desired locale. So if you need UTF-8 data, then you need a byte[]. I think I need to use an UTF-8 encoding for the POST request. – ciis0. Watch out for issues with the byte order / Now I want to write UTF-8 encoded content to the console. 11. Therefore, to standardize our data, we might need to convert them into UTF-8. txt: ISO-8859 text ISO-8859-1 maps every byte to a character, with the 80. srt my3rdfile. Even though the standard says a byte-order-mark isn't recommended for UTF-8, there can be legitimate confusions between UTF-8 and ASCII without a byte-order mark. base64 /path/to/file. You can then run bash yourfile. URL encoding a string in bash script. An index will also have a lower worst-case upper bound, and that may present other Base64 Encoding a File. Hot Network Questions If the string is already stored in a variable you can use bash's parameter expansion, specifially ${parameter,,pattern} (available since bash 4. Percent-encode all characters xxd -p|tr -d \\n|sed 's/. Character Encoding Issue on Ubuntu/Bash. Follow asked Sep 19, 2012 at 6:27. Here we use od instead of xdd because od is usually pre-installed. UTF-8". This is the hex representation of the ASCII letter Z. For example: " test&quo Is there a Linux command I could pipe to that specializes in decoding this type of data? linux; bash; scripting; sed; html; Share. echo unicode character as utf8 from file. If iconv cannot convert the file due to invalid UTF-8 sequences, it will return with a non-zero exit code. txt | python3 -c "import urllib. Improve this I used apt install recode on Ubuntu 22. Let's take this arrow as an example: The easiest way to specify the encoding when opening the file is to use the Open Files dialog. 35. com/questions/12649896/ Check that you have the locales You can use iconv to convert the encoding of the file: iconv -f ascii -t utf16 file2. txt Encode and save encoded text to a file: base64 file2. This is not what Git wants to use on Windows: we assume that char * variables point to strings encoded in UTF-8. txt utf8. the utf8 parameter of nr2char did not exist in 7. 0), where parameter is the name of your variable and pattern is ommitted: There is a better way to do that: encode the charset information in the filename, and apache will output the proper encoding header based on that. This is possible thanks to the AddCharset lines in the conf file, such as the line below: conf/httpd. You can do this in one of two ways: Write the sed command to a file, ensure the file is UTF-8, and execute it as a script: file yourfile should say UTF-8 Unicode text. Character encoding is a way of telling a computer how to interpret raw zeros and ones into real characters. 1. So if I do: echo 5a | command_im_looking_for I will see a solitary letter Z: Z Any simple way in bash to handle escaped UTF8 int? 13. 57. In Java I can decode every byte in the range 00. Less to type and no piping! And avoiding your plaintext BASH: Convert Unicode Hex to String. /fix-encoding. Ascii/Hex convert in bash. If it cannot represent all the characters, you could try | iconv -f UTF-8 -t //TRANSLIT to get an approximation. How can fix UTF-8 string usage in bash? 2. txt test. When I try that with windows-1252 I get garbage for the values bobince listed. On the bottom left, you will see an drop-dwon option for Character encoding. 5 bash terminal. Parse hex values into string for shell script. To base64 encode a file. ashiaka ashiaka. conf: AddCharset UTF-8 . encode. 0 h h4 HTML The default encoding for HTML 4 is Latin-1. Bash takes care of your locale settings. 69. it was using the wrong I'm trying to do a very simple task: take a unicode-aware wstring and convert it to a string, encoded as UTF8 bytes, and then the opposite way around: take a string containing UTF8 bytes and convert it to unicode-aware wstring. There is no single-byte charset other than ASCII that is a subset of UTF-8 as the UTF-8 encoding of characters other than the ASCII one is on 2 bytes or more. On a modern system, iconv -f iso-8859-1 -t utf-8 file. txt > encoded_file. txt latin1. /encode. The only thing that can have a different encoding is a byte[]. I can decode the PowerShell output in Linux, but I can't encode a string in Linux and get the same result as PowerShell. hex to ascii in bash. It matters what encoding the original string was used. If you 100% always only expect UTF-8 encoded text files, you can check with iconv, if a file is valid UTF-8: iconv -f utf-8 -t utf-16 out. txt | Out-File -en utf8 out. If your locale's charset (see output of locale charmap) is not UTF-8 (but can represent all the characters in the encoded string), add a | iconv -f UTF-8. txt >/dev/null. Ask Question Asked 12 years, 5 months ago. 8. setting the console font permanently will depend on your distribution. The problem is, I need it cross-platform and I need it work with Boost and I just can't seem to figure a way to make it work. The probably problematic line in . But like I said, it appears to work fine without the extra parameter; tested with some characters outside the ASCII range. When we write text to a file, the words and sentences Unicode escape sequences \uXXXX are used to encode Unicode characters. Unicode special characters not working on Mac OSX 10. 04 ; PHP 5. There is a pseudo UTF-8 I'm making a parser (1 csv to 3 csv) script and I have a problem. The large text file contains Arabic texts in UTF-8 format. txt $ file ascii. It is used to encode Unicode characters in groups of 8-bit variable byte numbers. For example I type decode QWxhZGRpbjpvcGVuIHNlc2FtZQ== and it prints Aladdin:open sesame and returns to the prompt. sh decode. If you have a String that contains unexpected data, then the problem is at some earlier place that incorrectly converted some binary data to a String (i. Literals of the form \uXXXX correspond to UTF-16 encoding. To decode with base64 you need to use the --decode flag. recode supports only HTML 4. You can view the UTF settings with the Proficient in bash scripting, Ansible, and AWX central server management, he handles server When converting your file, you should be sure it contains a byte-order mark. 04. I've started writing a bash script to connect to this server, so I won't have to type out the entire address every time, but I would like to improve this script so it also changes the encoding of the terminal window correctly. If it succeeds, that encoding is a potential candidate. If I understand your "render" notation, you mean a simple way to translate the six-byte escape sequence into a UTF-16BE character, in addition to being able to actually render on the console. sh and use it like this:. I'd happily take less than an hour to write such a procedure if this is the @blueray, no windows-1258 and 1252 are not subsets of UTF-8 (though they are supersets of ASCII like most charsets still in use these days). OS X uses the BSD tr and produces a nice result: We're not a different species “All alone?” Jeth mentioned. You may want to write the stdout to file instead. To discover the original encode of your file you can do. psv is the new file in utf-8 format: # getting enco How can I check in bash whether a variable contains a valid UTF-8 string without any it fails to reject strings that contain the UTF-8 encoding of characters with code points a python3 script (at least with python 3. Note that strings are zero terminated - it's impossible to store the zero byte. (specifically, JSON will encode one code-point using two \u escapes) If you assume 1 escape sequence translates to 1 code point, you're doomed on such text. 9F range being the C1 control characters. Background : Converting a csv file from UTF-8 to CP1252 in Ubuntu with PHP or SHELL, copy file from Ubuntu to Windows, open file with nodepad++. To workaround this issue you must take a look at the program at this site (also good for a deep look into the issue of converting numbers to chars. To opt-in (so that you don't accidentally break older code when you've implemented workarounds like the previous answers), simply set the WSL_UTF8 environment variable with Basically, I have a series of large dataframes, and I wish to convert one of the column vectors in all of these datasets from a string of characters, say for example: ASDFSDFSAFDSA, to its equivalent in ASCII format. On a crappy old PuTTY or ancient linux distro, iconv -f utf-8 -t iso-8859-1 file. txt: UTF-8 Unicode text $ file latin1. sh #!/usr/bin/env bash echo $1 | openssl aes-256-cbc -a -salt -pass 1 | openssl aes-256-cbc -d -a -pass pass:somepassword make files executable chmod +x encode. txt: ASCII text utf8. And look for "charset=" We’re only interested in the LC_CTYPE (locale character type) “codeset” part, “UTF-8”, which tells the shell how to interpret byte sequences as code points. Bash has an extension called "bash arrays" that allows to create an array. BASH: Convert Unicode Hex to String. 4,064 9 9 gold Stack Exchange Network. – that other guy Note that for bash 4. sh, give it execute permission using chmod +x fix-encoding. Running shell scripts with special chars in unix. This will output a very long, base64 encoded string. Your input, instead, is You only need to know from what encode you are converting. Ubuntu uses the GNU tr and produces this nasty result: Most tools that attempt to make this distinction will try and decode the file as utf-8 (as that's the more strict encoding), and if that fails, then fall back to iso-8859-1. Suppose I have string, returned in Json, like 004100610061 Is is string. The encoding HTML is an alias for HTML_4. Sorry my answer did come seeing Let's say that I have a string 5a. * All UCS characters greater than 0x7f are encoded as a multibyte sequence consisting only of bytes in the range 0x80 to 0xfd, so no ASCII byte can appear as part of another character and there are no problems with, for example, '\0' or '/'. If you specify the wrong input encoding, the output will be garbled. UTF8 in my bash_profile. Similarly a string of ascii characters maps in a 1:1 fashion to its hex string representation. I want to do this in bash because they are too large to process in R. $ dpkg -S /usr/bin/encguess perl: /usr/bin/encguess Anybody knows why this difference and some solution to my problem? I think that the problem is related to Unix's UTF-8 encoding, but I can't find a solution. But if \u00A7 is mangled too, most likely some web filter might be guilty: a minifier or such. Is bash's expansion of unset parameters to the empty string documented anywhere? echo -n "Hello" | od -A n -t x1 Explanation: The echo program will provide the string to the next command. 5 on Ubuntu) won't even start if it has a byte sequence that is the encoding of code points over 0x110000 Ubuntu Bash - print specific word from a file. How can I tell the command to read the file using iso, or UTF-8 encoding? I was encoding (from terminal) into base64. (We will be providing a flag to tell it to dump it in hexadecimal instead of octal. A string of ascii characters maps in a 1:1 fashion to its base64 encoding. This is the only thing I have uncommented: export LC_CTYPE=C export LANG=C In bash_profile: #export LANG=en_US. Your input, instead, is MIME encoded UTF-8. sh file into UTF-8? I tried: But this doesn't seem to work. Step-1: Show current UTF-8 settings. . txt Note: The possible enumeration values are "Unknown, String in I'm passing data to a bash script and sometimes that data includes special entities. txt my2ndfile. – Boldewyn LANG=en_EN. I want to split a large tagged text file into smaller ones so that I can manipulate them in Python. psv is my original file and hola1. 3. Visit Stack Exchange Your example code will now work properly with an opt-in fix in the latest WSL Preview release (), but note that (at least while in Preview) it is only available to Windows 11 users. So you can convert the string dGVzdA== to the string test: Tools xxd - make a hexdump or do the reverse. FF to a String using ISO-8859-1, then re-encode it to get the original bytes back. That is, the values between the space (HEX 20) and the Line-Feed (Hex 0A). I need to find a Linux shell command which will take a hex string and output the ASCII characters that the hex string represents. file -i yourfile. This has changed in HTML 5. 0. The ANSI variant assumes that the strings are encoded according to whatever is the current locale. The terminal can however let applications that interact it know its encoding by setting the locale environment variables. Improve this answer. Using a full JSON parser from the language of your choice is considerably more robust: -r (--raw-output) outputs the contents of strings instead of JSON string literals. 2 the Unicode code points from 0x80 to 0xFF are encoded incorrectly (bash bug). txt should then have the desired encoding. md5sum - will then give a prompt for simple input. So you have to convert your string into that encoding first before hexdumping it. Note that if you store a string in a variable or function, what matters is the encoding at the time the variable is expanded or the relevant command in Try this command line to get what you want, '2325', $ <<<'e28ca5' xxd -r -p | iconv -t unicode | hexdump 0000000 feff 2325 0000004 My original answer got some comments, that recode does not work for UTF-8 encoded HTML files. getfilesystemencoding() 'UTF-8' I thought the best (clean) way to do this was setting the PYTHONIOENCODING environment variable. qhenl egt rhc rff ykhkb cfv iewqr vrycfra iumosy usmsbtb