Here's the short-story on the current situation with our mailing list to Usenet gateway:
* Our Usenet host rejects multipart/alternative messages
because they are technically illegal Usenet posts
* This means that some emails do not reach comp.lang.ruby
(several messages each day according to the logs)
* We don't like this
To solve this, we want to enhance the gateway to convert multipart/alternative messages into something we can legally post to Usenet. I have two thoughts on this strategy:
1. If possible, we should gather all text/plain portions of an email and post those with a content-type of text/plain
2. If that fails, we can just post the original body but force the content-type to text/plain for maximum compatibility
Now I need all of you email and Usenet experts to tell me if that's a sane strategy. If another approach would be better, please clue me in.
I've pretty much made it this far. The code at the bottom of this message is the mail_to_news.rb script used by the gateway rewritten using this strategy.
If you aren't familiar with the gateway code, you can get details from the articles at:
http://blog.grayproductions.net/categories/the_gateway
There's one problem left I know I haven't solved correctly. Help me figure out a decent strategy for this last piece and we can deploy the new code.
The outstanding issue is how to handle character sets for the constructed message. You'll see in the code below that I just pull the charset param from the original message, but after looking at a few messages, I realize that this doesn't make sense. For example, here are the relevant portions of a recent post that wasn't gated correctly:
Content-Type: multipart/alternative; boundary=Apple-Mail-18-445454026
--Apple-Mail-18-445454026
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed
As you can see, the overall email doesn't have a charset but each text portion can. If we are going to merge these parts, what's the best strategy for handling the charset?
I thought of trying to convert them all to UTF-8 with Iconv, but I'm not sure what to do if a type doesn't declare a charset or when Iconv chokes on what is declared? Please share your opinions.
If you are feeling really adventurous, rewrite the relevant portion of the code below which I will bracket with a FIX ME comments.
Here's the script:
#!/usr/bin/env ruby
# written by James Edward Gray II <james@grayproductions.net>
$KCODE = "u"
GATEWAY_DIR = File.join(File.dirname(__FILE__), "..").freeze
$LOAD_PATH << File.join(GATEWAY_DIR, "config") << File.join(GATEWAY_DIR, "lib")
require "tmail"
require "servers_config"
require "nntp"
require "logger"
require "timeout"
# prepare log
log = Logger.new(ARGV.shift || $stdout)
log.datetime_format = "%Y-%m-%d %H:%M "
# build incoming and outgoing message object
incoming = TMail::Mail.parse($stdin.read)
outgoing = TMail::Mail.new
# skip any flagged messages
if incoming["X-Rubymirror"].to_s == "yes"
log.info "Skipping message ##{incoming.message_id}, sent by news_to_mail"
exit
elsif incoming["X-Spam-Status"].to_s =~ /\AYes/
log.info "Ignoring Spam ##{incoming.message_id}: " +
"#{incoming.subject}–#{incoming.from}"
exit
end
# only allow certain headers through
%w[from subject in_reply_to transfer_encoding date].each do |header|
outgoing.send("#{header}=", incoming.send(header))
end
outgoing.message_id = incoming.message_id.sub(/\.+>$/, ">")
%w[X-ML-Name X-Mail-Count X-X-Sender].each do |header|
outgoing[header] = incoming[header].to_s if incoming.key?header
end
# doctor headers for Ruby Talk
outgoing.references = if incoming.key? "References"
incoming.references
else
if incoming.key? "In-Reply-To"
incoming.reply_to
else
if incoming.subject =~ /^Re:/
outgoing.reply_to = "<this_is_a_dummy_message-id@rubygateway>"
end
end
end
outgoing["X-Ruby-Talk"] = incoming.message_id
outgoing["X-Received-From"] = <<END_GATEWAY_DETAILS.gsub(/\s+/, " ")
This message has been automatically forwarded from the ruby-talk mailing list by
a gateway at #{ServersConfig::NEWSGROUP}. If it is SPAM, it did not originate at
#{ServersConfig::NEWSGROUP}. Please report the original sender, and not us.
Thanks! For more details about this gateway, please visit:
http://blog.grayproductions.net/categories/the_gateway
END_GATEWAY_DETAILS
outgoing["X-Rubymirror"] = "Yes"
# translate the body of the message, if needed
if incoming.multipart? and incoming.sub_type == "alternative"
### FIX ME ###
# handle multipart/alternative messages
# extract body
body = ""
extract_text = lambda do |message_or_part|
if message_or_part.multipart?
message_or_part.each_part { |part| extract_text[part] }
elsif message_or_part.content_type == "text/plain"
body += message_or_part.body
end
end
extract_text[incoming]
if body.empty?
outgoing.body = "Note: the content-type of this message was altered by " +
"the gateway.\n\n#{incoming.body}"
else
outgoing.body = "Note: non-text portions of this message were stripped " +
"by the gateway.\n\n#{body}"
end
# set the content type of the new message
outgoing.set_content_type( "text", "plain",
"charset" => incoming.type_param("charset") )
### END FIX ME ###
else
%w[content_type body].each do |header|
outgoing.send("#{header}=", incoming.send(header))
end
end
log.info "Sending message ##{incoming.message_id}: " +
"#{incoming.subject}–#{incoming.from}…"
log.info "Message looks like:\n#{outgoing.encoded}"
# connect to NNTP host
begin
nntp = nil
Timeout.timeout(30) do
nntp = Net::NNTP.new( ServersConfig::NEWS_SERVER,
Net::NNTP::NNTP_PORT,
ServersConfig::NEWS_USER,
ServersConfig::NEWS_PASS )
end
rescue Timeout::Error
log.error "The NNTP connection timed out"
exit -1
rescue
log.fatal "Unable to establish connection to NNTP host: #{$!.message}"
exit -1
end
# attempt to send newsgroup post
unless $DEBUG
begin
result = nil
Timeout.timeout(30) { result = nntp.post(outgoing.encoded) }
rescue Timeout::Error
log.error "The NNTP post timed out"
exit -1
rescue
log.fatal "Unable to post to NNTP host: #{$!.message}"
exit -1
end
log.info "… Sent. nntp.post() result: #{result}"
end
__END__
Thanks for the help.
James Edward Gray II
But if there are plans to have the list itself do it, that's great.