Hello, Since there are f*cking windows mail programs around which does not handle utf8 email I had to look into how can I have utf8 frontend and iso-8859-2 email encoding. It Is Hell. It seems that OTRS was coded on the assumption that either the world is utf8 or it isn't, coexistence denied. Halfway I thought EncodeObject is a helpful thing, but it turned out that I am unable to grasp the point what it is good for. First, what I've thought. I believed it is the charset property of an object, so if I have, say, message in 8859-15 then I can set its encodingObject to know that it's 8859-15, so if I need it in, say, 8859-2, then I simply tell the poor fella to Convert( $to_my_charset ) and it happens. Nope. EncodeObject is somehow tracking utf8'ness, but seems to be impossible to use it to support any encoding. For example Sendmail.pm uses: $Self->{EncodeObject}->SetIO(\*MAIL); Which should set handle to utf8 if the object is in utf8, and leave it alone otherwise. But it does something magical: if $DefaultCharset is utf8 then it sets handle to utf8. No matter what I set anywhere. I believe this is wrong: EncodeObject should have an internal knowledge of its encoding (that's why it's an object, isn't it?) and act on that. So if my web frontend is in utf8 I'd set $article->{EncodeObject}->my_encoding('utf8'), and if I want to send an email in 8859-2, then I could say $new_article = $Self->{EncodeObject}->Convert('8859-2') or like and I could go on and Send() it using Charset 8859-2. Due to time constraints I am not able to completely analyse otrs object structure so probably encoding could be attached some different way (this means that my examples might have been stupid ones), but at the end of the chain (like having the mail in Sendmail.pm or probably around the web frontend) the result ought to know what its own encoding is and if the output module wants a new encoding then it should be able to convert it easily. And Encode should use Encode::FB_HTMLCREF as "CHECK" value in encodings/decodings. I didn't dare to start recoding EncodeObject since I don't yet grasp what it is EXACTLY designed for. -- Now using M2, Opera's e-mail client: http://www.opera.com/m2/
So here's the patch to be able to force the email charset no matter the frontend charset is. Please be aware that this was not thoroughly tested and is a rude hack (that means it completely lacks of design). EncodeObject probably should be fixed instead. It seems to work nevertheless, and doesn't choke if the mail would contain characters not possible to represent in the output charset (due to FB_HTMLCREF). Set $Self->{EmailCharsetForced} = 'iso-8859-2'; (or whatever your email charset preference is) in Config.pm for the best effects. And say a curse on bill gates today, and a blessing on martin ;) --- ./System/Email/Sendmail.pm-20041011 2004-10-11 16:42:59.000000000 +0200 +++ ./System/Email/Sendmail.pm 2004-10-11 17:13:20.000000000 +0200 @@ -64,7 +64,11 @@ # send mail if (open( MAIL, "| $Self->{Sendmail} $Arg")) { # set handle to binmode if utf-8 is used - $Self->{EncodeObject}->SetIO(\*MAIL); + if( !defined( $Param{Charset} ) || $Param{Charset} =~ /utf-?8/i ) { + # if Charset is not defined, act on DefaultCharset + # if it is utf8, then go on and hope that DefaultCharset it utf8, too :-@ + $Self->{EncodeObject}->SetIO(\*MAIL); + } print MAIL ${$Param{Header}}; print MAIL "\n"; print MAIL ${$Param{Body}}; --- ./System/Ticket/Article.pm-20041011 2004-10-11 13:05:02.000000000 +0200 +++ ./System/Ticket/Article.pm 2004-10-11 17:14:48.000000000 +0200 @@ -15,6 +15,7 @@ use Mail::Internet; use Kernel::System::StdAttachment; use Kernel::System::Crypt; +use Encode; use vars qw($VERSION); $VERSION = '$Revision: 1.70.2.1 $'; @@ -1178,6 +1179,16 @@ $Param{$_} = encode_mimewords($Param{$_}, Charset => $Charset) || ''; } } + # encode if email charset forced + if( defined($Self->{ConfigObject}->Get('EmailCharsetForced')) ) { + #$Self->{EncodeObject} = Kernel::System::Encode->new( ConfigObject => $Self->{ConfigObject} ); + #my $newbody = $Self->{EncodeObject}->Convert( + # Text => $Param{Body}, + # From => $Charset, + # To => $Self->{ConfigObject}->Get('EmailCharsetForced') ); + $Param{Body} = Encode::encode( $Self->{ConfigObject}->Get('EmailCharsetForced'), $Param{Body}, Encode::FB_HTMLCREF ); + $Charset = $Self->{ConfigObject}->Get('EmailCharsetForced'); + } # check bcc if ($Self->{SendmailBcc}) { $Param{Bcc} .= ", $Self->{SendmailBcc}"; @@ -1486,6 +1497,7 @@ Subject => $Param{Subject}, Header => $head->as_string(), Body => $Entity->body_as_string(), + Charset => $Charset, )) { # write article to fs if (!$Self->ArticleWritePlain( @@ -1573,6 +1585,7 @@ Bcc => $Self->{SendmailBcc}, Header => $HeaderObject->as_string(), Body => $BodyAsSting, + # Charset => $Charset, )) { return; } @@ -2087,7 +2100,7 @@ To => $GetParam{From}, Cc => $Cc, RealName => $Param{Realname}, - Charset => $Param{Charset}, + Charset => $Self->{ConfigObject}->Get('EmailCharsetForced') || $Param{Charset}, Subject => $Subject, UserID => $Param{UserID}, Body => $Param{Body}, --- ./System/Email.pm-20041011 2004-10-11 16:59:17.000000000 +0200 +++ ./System/Email.pm 2004-10-11 17:00:06.000000000 +0200 @@ -212,6 +212,7 @@ ToArray => \@ToArray, Header => \$Param{Header}, Body => \$Param{Body}, + Charset => \$Param{Charset}, ); } -- Now using M2, Opera's e-mail client: http://www.opera.com/m2/
Hi Peter, currently OTRS is just using one charset (in<->core (incl. db)<->out). Anyway, it would be a good feature to force outgouing emails to an other charset. Ot better, to put different charsets to in,core and out (So Kernel::System::Encode needs to be improved). Your patch is very good. :) Just one question (after that I'll start to implement it). What is FB_HTMLCREF for? To probe the new charset? Thanks! :) -Martini Peter Gervai wrote:
So here's the patch to be able to force the email charset no matter the frontend charset is. Please be aware that this was not thoroughly tested and is a rude hack (that means it completely lacks of design).
EncodeObject probably should be fixed instead.
It seems to work nevertheless, and doesn't choke if the mail would contain characters not possible to represent in the output charset (due to FB_HTMLCREF).
Set $Self->{EmailCharsetForced} = 'iso-8859-2'; (or whatever your email charset preference is) in Config.pm for the best effects.
And say a curse on bill gates today, and a blessing on martin ;)
--- ./System/Email/Sendmail.pm-20041011 2004-10-11 16:42:59.000000000 +0200 +++ ./System/Email/Sendmail.pm 2004-10-11 17:13:20.000000000 +0200 @@ -64,7 +64,11 @@ # send mail if (open( MAIL, "| $Self->{Sendmail} $Arg")) { # set handle to binmode if utf-8 is used - $Self->{EncodeObject}->SetIO(\*MAIL); + if( !defined( $Param{Charset} ) || $Param{Charset} =~ /utf-?8/i ) { + # if Charset is not defined, act on DefaultCharset + # if it is utf8, then go on and hope that DefaultCharset it utf8, too :-@ + $Self->{EncodeObject}->SetIO(\*MAIL); + } print MAIL ${$Param{Header}}; print MAIL "\n"; print MAIL ${$Param{Body}}; --- ./System/Ticket/Article.pm-20041011 2004-10-11 13:05:02.000000000 +0200 +++ ./System/Ticket/Article.pm 2004-10-11 17:14:48.000000000 +0200 @@ -15,6 +15,7 @@ use Mail::Internet; use Kernel::System::StdAttachment; use Kernel::System::Crypt; +use Encode;
use vars qw($VERSION); $VERSION = '$Revision: 1.70.2.1 $'; @@ -1178,6 +1179,16 @@ $Param{$_} = encode_mimewords($Param{$_}, Charset => $Charset) || ''; } } + # encode if email charset forced + if( defined($Self->{ConfigObject}->Get('EmailCharsetForced')) ) { + #$Self->{EncodeObject} = Kernel::System::Encode->new( ConfigObject => $Self->{ConfigObject} ); + #my $newbody = $Self->{EncodeObject}->Convert( + # Text => $Param{Body}, + # From => $Charset, + # To => $Self->{ConfigObject}->Get('EmailCharsetForced') ); + $Param{Body} = Encode::encode( $Self->{ConfigObject}->Get('EmailCharsetForced'), $Param{Body}, Encode::FB_HTMLCREF ); + $Charset = $Self->{ConfigObject}->Get('EmailCharsetForced'); + } # check bcc if ($Self->{SendmailBcc}) { $Param{Bcc} .= ", $Self->{SendmailBcc}"; @@ -1486,6 +1497,7 @@ Subject => $Param{Subject}, Header => $head->as_string(), Body => $Entity->body_as_string(), + Charset => $Charset, )) { # write article to fs if (!$Self->ArticleWritePlain( @@ -1573,6 +1585,7 @@ Bcc => $Self->{SendmailBcc}, Header => $HeaderObject->as_string(), Body => $BodyAsSting, + # Charset => $Charset, )) { return; } @@ -2087,7 +2100,7 @@ To => $GetParam{From}, Cc => $Cc, RealName => $Param{Realname}, - Charset => $Param{Charset}, + Charset => $Self->{ConfigObject}->Get('EmailCharsetForced') || $Param{Charset}, Subject => $Subject, UserID => $Param{UserID}, Body => $Param{Body}, --- ./System/Email.pm-20041011 2004-10-11 16:59:17.000000000 +0200 +++ ./System/Email.pm 2004-10-11 17:00:06.000000000 +0200 @@ -212,6 +212,7 @@ ToArray => \@ToArray, Header => \$Param{Header}, Body => \$Param{Body}, + Charset => \$Param{Charset}, ); }
On Mon, 11 Oct 2004 18:17:57 +0200, Martin Edenhofer
currently OTRS is just using one charset (in<->core (incl. db)<->out).
I see, that's "core charset"! :)
Anyway, it would be a good feature to force outgouing emails to an other charset. Ot better, to put different charsets to in,core and out (So Kernel::System::Encode needs to be improved).
Doesn't seem much hard to do, I'll check if time permits. For me it seems to be good idea to associate a charset with every larger text object (articles, notifications, etc) so it would be possible to even set a mail charset for every customer, for example. But that's just for the design, I don't think it's urgent to implement. So far my patch would work for email (after maybe some cleaning).
What is FB_HTMLCREF for? To probe the new charset?
If you don't use any value it's FB_DEFAULT, which mean it dies if there's a char not encodeable in the output encoding, like a hungarian o" to be converted in 8859-1, or chinese spam quoted to 8859-2. HTMLCREF created HTML embeds from those non-ecodeable characters, so instead of dying it generates ő which is slightly better. Peter -- Now using M2, Opera's e-mail client: http://www.opera.com/m2/
On Mon, 11 Oct 2004 17:25:16 +0200, Peter Gervai
participants (2)
-
Martin Edenhofer -
Peter Gervai