243833 - Attachments of Content-Type: application/pkcs7-mime that contain non-standard contents fail to decode and are not shown

Status: RESOLVED FIXED

(MailNews Core :: MIME, defect)

Description

[Enrico Caruso]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; it-IT; rv:1.6) Gecko/20040113
Build Identifier: Mozilla/5.0 (Windows; U; Windows NT 5.1; it-IT; rv:1.6) Gecko/20040113

When received a message with one or more file attachment of type
application/pkcs7-mime (.p7m) is impossible to see the file and to save it.
Here are some headers extracted from messages received:
==============EXAMPLE 1==============
Sent with MS Outlook
=====================================
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----=_NextPart_000_0046_01C439B9.DFB4D640"
X-Priority: 3 (Normal)
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook, Build 10.0.2627
X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4939.300
...
This is a multi-part message in MIME format.

------=_NextPart_000_0046_01C439B9.DFB4D640
Content-Type: multipart/alternative;
	boundary="----=_NextPart_001_0047_01C439B9.DFB4D640"


------=_NextPart_001_0047_01C439B9.DFB4D640
Content-Type: text/plain;
	charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
...
------=_NextPart_000_0046_01C439B9.DFB4D640
Content-Type: application/pkcs7-mime;
	name="wf24m001.p7m"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
	filename="wf24m001.p7m"
....
==============EXAMPLE 2==============
Sent signed with Mozilla on WindowsXP
=====================================
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; it-IT; rv:1.6) Gecko/20040113
X-Accept-Language: it, en-us, en
MIME-Version: 1.0
...
Content-Type: multipart/signed; protocol="application/x-pkcs7-signature";
micalg=sha1; boundary="------------ms050407090205000104070508"

This is a cryptographically signed message in MIME format.

--------------ms050407090205000104070508
Content-Type: multipart/mixed;
 boundary="------------070403020802060305020301"

This is a multi-part message in MIME format.
--------------070403020802060305020301
Content-Type: multipart/alternative;
 boundary="------------000301010902010409030609"


--------------000301010902010409030609
...
--------------070403020802060305020301
Content-Type: application/pkcs7-mime;
 name="TestFlusso.txt.p7m"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
 filename="TestFlusso.txt.p7m"
...
--------------ms050407090205000104070508
Content-Type: application/x-pkcs7-signature; name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="smime.p7s"
Content-Description: S/MIME Cryptographic Signature
...


Reproducible: Always
Steps to Reproduce:
1. On the windows platform which defaults .p7m with a content type of
application/pkcs7-mime use outlook or Mozilla to compose a message and attach a
file with .p7m extension;
2. Send the message to a user with Mozilla Mail on Windows or other platform;


Actual Results:  
The user which receives the message is not able to see the file attached

Expected Results:  
The user should have a way to see/save the file/s attached.

Comment 1

[Mike Cowperthwaite]

*** Bug 260841 has been marked as a duplicate of this bug. ***

Comment 2

[Frank Wein [:mcsmurf]]

->MIME i think

Comment 3

[Adam Guthrie]

*** Bug 304431 has been marked as a duplicate of this bug. ***

Comment 4

[marcoleli]

Even version 1.5.0.9 has this bug.

Comment 5

[giorgio]

... and version 3 alpha 1 (20070206) too.

Comment 6

[Peter]

Attachment: file for test (SEND/RECEIVE)

When send or receive this file it's impossible to see the file and to save it.

Comment 7

[Peter]

Comment on attachment 297336 [details]
file for test (SEND/RECEIVE)

When send or receive this file it's impossible to see the file and to save it. Please help

Comment 8

[Laurent Blin]

The problem still goes on with 2.0.0.6.
Is there any way to show the p7m file as attachement?

Comment 9

[guia78]

This problem in very important for the Pubblic Amministration because use the exstension p7m for digital company.
In the most newspaper it has been described the problem:
http://www.pc-facile.com/news/thunderbird_problema_enti_pubblici/56350.htm
http://www.hwupgrade.it/news/software/mozilla-thunderbird-e-il-problema-degli-allegati-firmati-digitalmente_24462.html

The solution is the disable the sting "application/pkcs7-mime" in thunderbird.exe 

Please the resolve this problem

Comment 10

[intendentedelleacque]

After many many tests, I believe to have found out a way to solve this problem, until TB developers give an official answer about this bug: everybody can try this extension https://nic-nac-project.org/~kaosmos/p7mHandler-en.html 

Notice that - if this extension works well as it seems to do - the problem is fixable just adding a line in compreg.dat file.

Comment 11

[guia78]

Very good, but the problem is the source of Thunderbird. It is incorrect. The plug-in is a solution temporary.

Comment 12

[intendentedelleacque]

After last changes in my extension, I can give some details more.

At the present Thunderbird seems to be able to decode/decrypt messages when their primary content-type is application/pkcs7-mime or application/x-pkcs7-mime.

But if a message has another primary content-type (for example text-plain) and it has an attachment with content-type application/pkcs7-mime or application/x-pkcs7-mime, the attachment is hidden.

Besides, if there are other attachments, Thunderbird can fail to open them.

So it seems that at the present Thunderbird is able to handle properly content-type application/pkcs7-mime or application/x-pkcs7-mime just if it's the primary one, while it fails in other cases, probably because of wrong parsing/decoding. Since it fails, the attachment is hidden and also other attachments parsing can become wrong.

The last version of my extension tries to fix this, enabling/disabling the native support for application/pkcs7-mime or application/x-pkcs7-mime, according to the primary-content of the message.

Of course this is a workaround, until the problem will be fixed in core code.

Comment 13

[Nelson Bolyard (seldom reads bugmail)]

I think this has never been mentioned in mozilla's mozilla.dev.tech.crypto newsgroup until today.  :-/

Comment 14

[spt]

application/x-pkcs7-signature should be application/pkcs7-signature.  application/x-pkcs7-mime should be application/pkcs7-mime.  As per RFC 3850 (http://www.ietf.org/rfc/rfc3851.txt) section 3.

Comment 16

[paultt]

problem still there
this prevents some PA related environment to adopt thunderbird :(
just to incorporate the p7m-handler extension is too difficult?

Comment 17

[paultt]

there's another strange thing about this bug: when you choose forward or edit as new on the message hiding such attachment, the attachment shows itself again, correctly as an attachment.

the attachment type is application/pkcs7-mime
if you save the attach to disk, then open a new mail and attach the saved file to the new mail, now the attachmente type becomes application/octet-stream (which probably colud be more correct), and so on the attachment becomes visible on receiving that mail

Comment 18

[[:Aureliano Buendía]]

this bug is very annoying: in Italy now is on use an "standard" named PEC
(see draft http://www.ietf.org/id/draft-gennai-smime-cnipa-pec-06.txt)

Since this issue is still here all italian user that use  PEC can't use TB because all attachments are hidden (except than in source code).

Comment 19

[Nelson Bolyard (seldom reads bugmail)]

First, the Thunderbird people made it clear that they have no interest in
making any further enhancements to the S/MIME capabilities of Thunderbird.
They're not even taking contributions of source code patches that implement
these things.  Bug fixes are still accepted, AFAIK.

Second, if they WERE to work on this issue, they'd need to see ENTIRE S/MIME
messages that demonstrate the issues, not merely the binary contents of some
P7m attachment.

Comment 20

[Francesco Lodolo [:flod]]

(In reply to comment #18)
> Since this issue is still here all italian user that use  PEC can't use TB
> because all attachments are hidden (except than in source code).

A little more information about this. This is really a major problem in Italy:
* In order to start a new company, or change the existing identifying data in public registries, you need a PEC account. All professionals (i.e. lawyers) had to provide a PEC account before November 2009 to their professional order, while all existing companies must have a PEC account before January 2012.
* .p7m attachments are mandatory for some kind of communications, for example to send yearly Financial Statement to the local Chamber of Commerce.

Comment 21

[Nelson Bolyard (seldom reads bugmail)]

Please, no more messages advocating action on this issue.  This is not a 
discussion forum.  This is a bug report.  The comments and attachments 
that belong here are those that provide direct technical information that 
leads directly to diagnosis and/or solution.  Advocacy belongs elsewhere,
such as in the Thunderbird newsgroups and mailing lists. 

If you can contribute a sample SMIME email message that demonstrates this 
issue, please do so.  You may email such a message directly to me, if you 
wish.  I don't promise that it will lead to any resolution of this issue,
but I can promise that this issue will not be resolved without such sample 
messages.  To facilitate that, I will send a signed email to each of the 
people who have added comments to this bug report.

Comment 22

[Nelson Bolyard (seldom reads bugmail)]

Oh, to make it clear, I will take any email that you send to me to demonstrate this issue and attach it to this bug, where it will become readable by all 
bugzilla readers.  Bear that in mind as you decide what to send me.

Comment 23

[intendentedelleacque]

Attachment: Message with p7m attachment

Comment 24

[intendentedelleacque]

Steps to reproduce the bug:

1) Import the eml file named at #23
2) select it --> you will not see any attachment



"Smart p7m support" extension test:

1) install this extension written by me https://nic-nac-project.org/~kaosmos/p7mHandler-en.html
2) select the message above --> you will be able to see, open and save the p7m attachment

For more details about the way this extension works, please see #12

Comment 26

[Nelson Bolyard (seldom reads bugmail)]

This bug is about encrypted emails.  TB will only attempt to handle an encrypted message provided that the person on whose behalf it is reading the
message is a designated recipient of the encrypted message.  Sending me a message with a p7m attechement won't do me any good unless the message is addressed to me, using my email certificate for encryption.

Someone sent me a signed email message.  It contained only a .p7s component, 
no p7m component.  It was quite curiously constructed.  The person sent it 
to me, but it went to an SMTP relay that generated a new email message, 
with his orignal email message as an attachment to it.  The new email 
message did not appear to be sent by the original person, but by the relay.
It was signed, not by the original sender, but by the relay.  The signed 
message (from the relay) appeared to be a correctly signed email message in
my email client.  The signature verified OK, although it was a signature 
from the relay and not from the original sender.  consequently, I was unable
to send an encrypted reply to the original sender, because I did not have 
a certificate for that sender, but only for the relay. I did not wish to 
correspond with the relay, and did not.

In any case, so far, based on the emails sent to me, I have not yet seen any
misbehavior attributable to the TB client.

Comment 27

[Francesco Lodolo [:flod]]

(In reply to comment #26)
> In any case, so far, based on the emails sent to me, I have not yet seen any
> misbehavior attributable to the TB client.

Did you try the message attached to this bug?
https://bugzilla.mozilla.org/attachment.cgi?id=454091

Just try to open it with Thunderbird (no attachment) and forward it (attachment is magically there).

Comment 28

[Nelson Bolyard (seldom reads bugmail)]

Did you read comment 26 paragraph 1?

Comment 29

[Francesco Lodolo [:flod]]

(In reply to comment #28)
> Did you read comment 26 paragraph 1?

Yes, but I don't see the point (my fault?). The summary of this bug says "Unable to see/detach .p7m attachments of Content-Type: application/pkcs7-mime" and that's exactly what happens if you try what I wrote in comment 27.

Comment 30

[[:Aureliano Buendía]]

(In reply to comment #26)
Ciao Nelson.
I'm the person which has sent the message using my account with PEC relay.

> Someone sent me a signed email message.  It contained only a .p7s component, 
> no p7m component.  It was quite curiously constructed.  The person sent it 
> to me, but it went to an SMTP relay that generated a new email message, 
> with his orignal email message as an attachment to it.  The new email 
> message did not appear to be sent by the original person, but by the relay.
> It was signed, not by the original sender, but by the relay.  The signed 
> message (from the relay) appeared to be a correctly signed email message in
> my email client.  The signature verified OK, although it was a signature 
> from the relay and not from the original sender.  

That's right: it is exactly that! The individual who did not send any certificate, but the relay certifies that the sending and delivery.

I hate this "standard" but many of us have to use it for work.

As suggested in comment #27, if you just do a test with the attached email into this bug (comment #23) you will see that the attachment is not displayed, unless you "edit as new" the message.

This bug is very serious because it actually prevents the use of TB in the Italian Public Administration.

Let us know if we can do.

Comment 31

[paultt]

i understand tb which isn't able to verify/open the attachment by itself.
it's wrong that when it fails nor:
1. say it failed something or
2. (better) simply show the 'wrong' attachment to be shown as an attachment

Comment 32

[Matteo]

Nelson Bolyard (:MisterSSL) wrote:
> This bug is about encrypted emails.

This isn't exact, even if it can be generated by how TB handles encrypted emails.

This bug is about attachments, mainly with the .p7m extension, signed or encrypted by external applications, and that will be opened by external applications (such as this one, from italian DoD: http://www.difesa.it/pki-ff/manuale-verifica-firma/apri_p7m.htm ).

So we simply need to transfer those files as they are, like any other attachment; no matter if email recipient and signing/encrypting certificate don't correspond.

I hope this clarifies that old bug.

Thank You.
Matteo

Comment 33

[Dan Mosedale (:dmosedale, :dmose)]

(In reply to comment #19)
> First, the Thunderbird people made it clear that they have no interest in
> making any further enhancements to the S/MIME capabilities of Thunderbird.
> They're not even taking contributions of source code patches that implement
> these things.

While S/MIME is not currently a high priority for us, the statement that we're not accepting source code patches implementing features is not correct.  There are logistical issues around there being not many people around at all who have experience with the S/MIME code.  Not surprisingly, bigger patches are harder for to get landed.  

The fix described in comment 32 is almost certainly one we would take, I think.

Comment 34

[Nelson Bolyard (seldom reads bugmail)]

> the statement that we're not accepting source code patches implementing 
> features is not correct.

https://bugzilla.mozilla.org/show_bug.cgi?id=380624
https://bugzilla.mozilla.org/show_bug.cgi?id=386313

Comment 35

[Dan Mosedale (:dmosedale, :dmose)]

(In reply to comment #34)
> > the statement that we're not accepting source code patches implementing 
> > features is not correct.
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=380624

The patch in this bug was given ui-review-, and is awaiting a new patch with the various concerns addressed, as explicitly stated in the last few comments in the bug.

> https://bugzilla.mozilla.org/show_bug.cgi?id=386313

This one had fallen off the radar; sorry about that!  I've just talked to Bryan, and he put some ui-review comments in that bug and it's now awaiting those to be addressed.

Comment 36

[Nelson Bolyard (seldom reads bugmail)]

A translated form of the page at the URL given in comment 32 is
http://babelfish.yahoo.com/translate_url?doit=done&tt=url&intl=1&fr=bf-home&trurl=http%3A%2F%2Fwww.difesa.it%2Fpki-ff%2Fmanuale-verifica-firma%2Fapri_p7m.htm&lp=it_en&btnTrUrl=Translate

This bug tells me that, while MOST mail clients (MUAs) that send encrypted 
SMIME messages expect the receiving MUA to decrypt them, some expect those
the encrypted part of the message to be treated purely as a binary attachment, 
like a binary file, and be saved to disk, without any attempt to decrypt by
the MUA.  

I'm sure this is further complicated by the possibility of an encrypted email being "forwarded as attachment".  I believe most SMIME-capable MUAs will 
decrypt the enclosed attached email while reading the enclosing email, if the MUS'a user is a designated receipient of the enclosed attachment.  

It's not clear to me that the decision on how to treat the encrypted part of
the mail should be made by the sender of the email.  It's quite possible 
that one recipient's MUA would handle the complete message itself, and 
another recipient's MUA would was to use an external "helper program" for 
that task.  Seems to me the sender should not force the recipient to handle
the encrypted part in one way or another.  

I suspect that, in this case, the TB MUA decided that this particular 
encrypted part should be decrypted line any other encrypted email for which
the local user was clearly a designated recipient, but then failed to 
decrypt it for some reason.  But that's conjecture.  The attachments provided for this bug to date do not enable TB developers to tell what's really going on, because the encrypted messages were not encrypted using any of the TB developer's email certificates, and those messages were not addressed to any 
of the TB developers.

Comment 37

[Matteo]

I think some diagnosys about conditions that causes TB to fail to display attachment is in Comment 12 https://bugzilla.mozilla.org/show_bug.cgi?id=243833#c12 by the person that wrote the smartp7m extension.

I'm not a developer, so I don't know if this can help; but this bug is very easy to reproduce:
-Compose a new mail message and attach the .p7m attachment in this bug;
-Save the message as draft and close it;
-Go to draft folder and select the message: where's the attachment gone??
-Double click the message to compose it again: here it is!

'Edit message as new' was one of the first work-around to recover the attachment, before the extension came out.

Regards.
Matteo

Comment 38

[Nelson Bolyard (seldom reads bugmail)]

Someone on the cc list of this bug has been sending me messages through this
PEC system.  These messages are signed only, and contain NO p7m attachment,
so they fail to reproduce the problem that is the subject of this bug.
Worse, when I try to reply to them with a signed reply, I get a bounce message 
which reads:

  62.149.152.101 failed on DATA command.  Remote host said: 
  554 Messaggio rifiutato dal sistema. Indirizzo destinatario sconosciuto o 
  non abilitato alla ricezione di posta non certificata.

Babelfish translates that as:
  Message refused from the system. Address adressee disowned or not qualified 
  to the reception of mail not certifyd. 

Now, this raises an important Mozilla policy issue.  Mozilla is expressly 
NOT committed to supporting closed PKI systems.  If this non-standard PEC 
system can be made to accept communications then it may be worthy of more 
time, else (IMO) it is not.

Comment 39

[Matteo]

IMHO, including PEC in this bug is misleading, since TB handles PEC messages well (as long as they have no .p7m attachment); there are even tutorials to configure TB with PEC (http://www.tol.it/doc/posta/videoguide/pec_thunderbird.swf ).

PEC is a global system, it involves MUA as well as SMTP, it must reside on exclusive domain (i.e., a domain or subdomain must handle only PEC accounts and not also normal accounts), and PEC accounts are not always setup to receive messages from normal accounts (even if signed with standard signature; this is the error You received).

The fact is that PEC, for its purpose of being used with P.A., is often used to send signed attachments, so it's affected by this bug maybe often than normal messages, but in the same way as normal messages; for me, absurd and complicated (sigh!) architecture of PEC is not a matter in this bug and can really complicate analysis.

This bug is present in all messages with a .p7m attachment opened by TB, PEC or not.

Regards.
Matteo

Comment 40

[logan]

This issue is also present with files that have a '.ent' extension from the older versions of entrust. Thunderbird is treating attachments that is has no business dealing with as a part of its own protocol.

Comment 41

[logan]

(In reply to comment #40)
> This issue is also present with files that have a '.ent' extension from the
> older versions of entrust. Thunderbird is treating attachments that is has no
> business dealing with as a part of its own protocol.

At this point, I'm left assuming that it's more to do with the headers, and Mozilla not keeping up with changes in encryption technology than anything else. Seems a lot like a legacy crypto system in dire need of an update, and nic-nac's smart p7m management is on the right track.

Comment 42

[Nelson Bolyard (seldom reads bugmail)]

> Seems a lot like a legacy crypto system in dire need of an update,

Yeah, it's the MIME part that is is most dire need of updating here, IMO.

But I expect that if/when an update is delivered, it will simply remove 
all S/MIME support entirely.

Comment 43

[logan]

That would be a preferable alternative, at least in terms of American, and evidently, Italian public administration. It's easy enough to export a .pfx from a third party app and import it into Thunderbird, but that doesn't help us get it backed up on the servers when someone is sending a file from the field, nor does it help when you have hundreds of people in an office that all need it done. At least make S/MIME support opt-in so the die-hards can find it and the rest of us won't have to bend over backwards just to fit Thunderbird into our work environment.

Comment 44

[paultt]

my 3 cents:
is it not easier to simply display the attachment as an attachment when tb isn't able to decrypt it?
it's a sure fail anyway if it hides the attachment, it's unquestionably wrong not to let the user save it. PEC or not PEC, administration or not administration.

Comment 45

[Nikolay Shopik]

http://tools.ietf.org/html/rfc6109 - Italian Certified Electronic Mail, they now have informational RFC, JFI

Comment 46

[David :Bienvenu]

Forgive me if this is a dumb idea (I'm not the smime expert), but it would seem that if the parent part content-type is not something like:

Content-Type: multipart/signed; protocol="application/pkcs7-signature"; 

or if it's multipart/mixed,

then we should either not be invoking the s/mime plugin content handlers, or the content handlers should just pass through the content.

Comment 47

[David :Bienvenu]

Attachment: possible fix

I've requested a try server build for people to try - the builds will show up here in a few hours:

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/tryserver-builds/[e-mail]-c57ecbfec8c8

I'm especially interested in knowing that messages that are signed with .p7m files are still handled correctly with this patch.

Comment 48

[David :Bienvenu]

try server builds failed - I'll update the bug if I get a successful try server build.

Comment 49

[David :Bienvenu]

Comment on attachment 527655 [details] [diff] [review]
possible fix

Kaie, does this look reasonable to you?

Comment 50

[David :Bienvenu]

thx to Standard8, who fixed the try server, builds are starting to show up here - http://stage.mozilla.org/pub/mozilla.org/thunderbird/tryserver-builds/[e-mail]-8ab6c8cd0790/

To people on the cc list, if you're interested in seeing this fix go forward, it would be very helpful if you could try one of these builds and report back.

Comment 51

[[:Aureliano Buendía]]

(In reply to comment #50)
> it would be very helpful if you could try one of these builds and report back.

I don't find win32 builds

Comment 52

[David :Bienvenu]

sorry, our build configs have been in a state of flux - try this http://stage.mozilla.org/pub/mozilla.org/thunderbird/try-builds/[e-mail]-8ab6c8cd0790/

Comment 53

[[:Aureliano Buendía]]

(In reply to comment #52)
> sorry, our build configs have been in a state of flux - try this
> http://stage.mozilla.org/pub/mozilla.org/thunderbird/try-builds/[e-mail]-8ab6c8cd0790/

tested attached testcase ( _Message with p7m attachment_ ) on try build on work fine for me: now I can see the encrypted file *.doc.p7m :-D

Comment 54

[Giacomo Magnini]

I can finally see and save the p7m attachment (Linux64).
Thanks for taking care of this peculiar bug, David.

Comment 55

[Francesco Lodolo [:flod]]

The build has been tested by at least 3 people on out support forum, I personally tested the OS X 64bit version and I can finally see the attachment in the test mail attached to this bug.

Comment 56

[David :Bienvenu]

Kai, is there a realistic chance that you'll get a chance to review this before we ship 3.3 in June, or should I try to find a different reviewer?

Comment 57

[David :Bienvenu]

Attachment: fix with mozmill test - backed out

Perhaps asuth can review this.

One caveat - the test_attachment.js test is failing on my machine, with and without my patches, in multiple trees, when run as a standalone test. It opens the compose window and the attach dialog, and then just sits there. But I think the test change is right, in that the test should fail w/o the patch, and succeed with it.

Comment 58

[gianni.giusti]

The build has been tested by at least 2 people on windows xp. I see the attachment.

Comment 59

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 529882 [details] [diff] [review]
fix with mozmill test - backed out

Test looks fine and mozmill passed on the try server:
http://arbpl.visophyte.org/?tree=ThunderbirdTry&pushid=644

Comment 60

[David :Bienvenu]

fixed on trunk - http://hg.mozilla.org/comm-central/rev/8b8f6d0418b2

Comment 61

[Kai Engert (:KaiE:)]

Have you tested that it works in all scenarios?

I'm reading the original code as this:
- if mimetype is application/(x-)pkcs7-mime
  then let's start to assume it's encrypted (catch-all),
  and by default, let's handle it by
    mimeEncryptedCMSClass

- only if it's a cert-only-message or a compressed-message,
  then let's hand it by something else.

According to RFC 3851, extension .p7m can be two different things:
- a signed message
- an encrypted message

Your patch changes that to say:
- if we see extension .p7m,
  don't handle it as an encrypted message, but using that other handler

So, my question is, have you tested both possible kinds of messages?


(a)
Have you tested that on receiving a signed-only message that uses the .p7m format, the message will still be decoded, and the user interface will still show the signature icon?

(b)
Have you tested that on receiving an encrypted only message that uses the .p7m format, that the message will be correctly decrypted and the user interface still displays the encryption icon?

Comment 62

[David :Bienvenu]

(In reply to comment #61)
> Have you tested that it works in all scenarios?
Thx for the feedback. I don't have examples of the kind of messages you're talking about - does anybody cc'ed on this bug have examples?

My understanding is that the mimetype of the containing part would be multipart/signed or multipart/encrypted and that would control whether we would treat the message as signed/encrypted.

Comment 63

[Kai Engert (:KaiE:)]

I have looked through my archive of sample messages.

I have lots of encrypted messages that use the .p7m type.
It's actually what Thunderbird produces by default when creating an encrypted message.

I propose, one of you who is proposing this change, should run a build with this patch applied, and exchange signed/encrypted emails with me, and test, if you can read my encrypted emails with both patch applied and not applied.


Regarding SignedData messages that use .p7m, I can't find an example of that right now.

Comment 64

[Kai Engert (:KaiE:)]

(In reply to comment #63)
> 
> Regarding SignedData messages that use .p7m, I can't find an example of that
> right now.

Ok, the attached sample message appears to be of that type

Comment 65

[Kai Engert (:KaiE:)]

(In reply to comment #62)
> 
> My understanding is that the mimetype of the containing part would be
> multipart/signed or multipart/encrypted and that would control whether we would
> treat the message as signed/encrypted.

mimetype application/pkcs7-mime is a binary blob.

multipart is only used when you want to include the message part in plaintext, and want to deliver the signature as a separate part, enabling non-s/mime apps to display the plaintext part.

An alternative encoding is to combine signature and message in a binary blob: That's actually what the attached messages are doing. There is no more mime or multipart inside, it's DER encoded binary data.

When dealing with encrypted data, to my knowledge, you never use multipart. The encrypted contained is always a binary DER encoded blob. (Only inside that one, after decrypting, you might find either multipart of DER encoded signature, or something else).

I'll attach an encrypted sample message.
(That won't help for testing, because only I can decrypt it. If you want real testing, someone with a patched build needs to exchange signed/encrypted messages with me.)

Comment 66

[Kai Engert (:KaiE:)]

Attachment: sample encrypted message (only kai can decrypt)

Comment 67

[Kai Engert (:KaiE:)]

regarding DER:
see http://en.wikipedia.org/wiki/Distinguished_Encoding_Rules

Comment 68

[Kai Engert (:KaiE:)]

I invite anyone on this bug to send me a S/MIME signed message. Signed-only. To the email address shown in this bug. Please mention bug 243833 in the subject.
I'll reply with an encrypted email.

Comment 69

[David :Bienvenu]

(In reply to comment #68)
> I invite anyone on this bug to send me a S/MIME signed message. Signed-only. To
> the email address shown in this bug. Please mention bug 243833 in the subject.
> I'll reply with an encrypted email.

done.

Comment 70

[Kai Engert (:KaiE:)]

Comment on attachment 529882 [details] [diff] [review]
fix with mozmill test - backed out

r-, because David told me this regression reading encrypted message.

I propose to reopen and backout.

Comment 71

[David :Bienvenu]

Attachment: fix for regression showing signed/encrypted messages

this displays correctly all the messages Kai sent me, and doesn't regress the mozmill test.

Kai suggested that mimecms should have the smarts about what to show, but I could not for the life of me get it to treat the .p7m file as an attachment in the case of a multipart/mixed message. I believe it's very tricky to fool libmime that a part should be treated as an external part, certainly beyond my libmime chops.

All this patch does is for mime multipart messages, set a flag in the parse state that we're in a multipart part, and treat p7m files as external. I believe multipart/signed messages will still be handled correctly, but I don't have any to test with.

Comment 72

[David :Bienvenu]

multipart/signed worked fine with this patch (if you send yourself a message with html in it, we create a multipart/signed message).

Comment 73

[Kai Engert (:KaiE:)]

reopening because of regression

Comment 74

[Kai Engert (:KaiE:)]

Attachment: Test message - p7m is single content (not multipart)

This is a variation of the earlier attached message.

Instead of having a p7m signature attachment, the p7m attachment is the single content.

In other words, not a multipart message.
The patch for this bug should be tested with this message, too.

Comment 75

[David :Bienvenu]

(In reply to comment #74)
> Created attachment 530588 [details]
> Test message - p7m is single content (not multipart)
>
Thx for the test case. This test case doesn't work in 3.1, or with the most recent patch, but it does work with the patch that regressed signed/encrypted non-multipart messages.

If I'm reading this correctly, this is evidence that the core mime code can't tell the difference between a cert attached as a message body and an encrypted message, if the content type is application/pkcs7-mime. In which case, we have a few choices:

1. Accept the most recent patch, which fixes this bug, and doesn't regress anything that has ever worked in Thunderbird, as far as we know (but doesn't work with the most recent test case). And then file a separate bug for #2.

2. Someone (not me, unfortunately) can figure out how to make s/mime trick the core mime code into doing the right thing based on the content of the p7m data. This involves s/mime knowledge that I don't have (telling the difference between a p7m attachment and der-encoded binary data), and then telling libmime to either treat the data as an attachment or a message.

3. Back out the checked in patch and mark the bug won't fix.

I'd vote for 1 since it would all the Italian Public Administration to use TB 3.3 and doesn't regress anything that ever worked (essentially, the perfect is the enemy of the good). But for now, I'll back out the checked in patch.

Comment 76

[[:Aureliano Buendía]]

(In reply to comment #75)
> I'd vote for 1 
+1

>(essentially, the perfect is the enemy of the good). 
:-D

Comment 77

[David :Bienvenu]

Comment on attachment 529882 [details] [diff] [review]
fix with mozmill test - backed out

I've backed this out - I'm not marking it obsolete because the test part would still be valid if we did fix this.

Comment 78

[David :Bienvenu]

backout changeset - http://hg.mozilla.org/comm-central/rev/165a011f1c2a

Comment 79

[Kai Engert (:KaiE:)]

Comment on attachment 530201 [details] [diff] [review]
fix for regression showing signed/encrypted messages

r=kaie 

I'm fine with this patch, if it doesn't regress.

I wish we had more tests. I'm not sure I was able to think of all possible kind of messages.

David, did you also get my email from my gmx.de address (yesterday), and the attached message worked as expected, and showed the encryption icon?

Comment 80

[David :Bienvenu]

(In reply to comment #79)

> I wish we had more tests. I'm not sure I was able to think of all possible
> kind of messages.

I've been thinking what it would take to add tests for this. We'd need the test messages, and a cert db with the public key of the sender and the private key/cert of the recipient, right? I think I could write a test that imported a cert db into the profile, loaded the test messages, and verified the results, but I'd need the cert db, and it would have to be certs that we didn't mind being publicly available (i.e., test certs). I assume we don't have to worry about cert expiration, because we wouldn't be constructing messages, just reading them. Kai, if you'd be up for creating the test messages and an appropriate cert db, I could write the xpcshell tests that verify the test messages.
> 
> David, did you also get my email from my gmx.de address (yesterday), and the
> attached message worked as expected, and showed the encryption icon?

I got one from 5/4 with an attached encrypted e-mail that worked fine, but nothing from yesterday. I don't see anything in my junk folder either.

Comment 81

[David :Bienvenu]

Oh, and I'll hold off on landing this until we figure out what happens with the last test case you mentioned.

Comment 82

[David :Bienvenu]

Attachment: fix with unit test restored

Comment 83

[Kai Engert (:KaiE:)]

> > David, did you also get my email from my gmx.de address (yesterday), and the
> > attached message worked as expected, and showed the encryption icon?
> 
> I got one from 5/4 with an attached encrypted e-mail that worked fine

I guess that's the one I meant.

Comment 84

[David :Bienvenu]

new fix checked in -
http://hg.mozilla.org/comm-central/rev/9bd80ef2728e

I'll file a follow-on bug for the p7m file as message body/attachment

Comment 85

[David :Bienvenu]

bug 655535 filed for the remaining issue (#2 above)

Comment 86

[Kai Engert (:KaiE:)]

Attachment: opaque signed message - important test case - mailbox file

David, please test this:
- download zip attachment, unzip
- place file wmopaque into ~/.thunderbird/*/Mail/Local\ Folders

This message was created using "Windows Mail" (the Outlook Express successor).

This message:
- has a p7m body
- the message is NOT encrypted, just signed
- the message correctly works in Thunderbird

Can you please test the patch does not regress this kind of message?

Comment 87

[David :Bienvenu]

that test case works the same for me on the trunk and with 3.1.10 - the body says "signed". The envelope says the message is signed, but the signature is not valid. Perhaps I don't have that cert, but it behaves the same way w/ and w/o the patch, so it doesn't seem to be a regression.

Comment 88

[Kai Engert (:KaiE:)]

(In reply to comment #87)
> that test case works the same for me on the trunk and with 3.1.10 - the body
> says "signed". 

great


> The envelope says the message is signed, but the signature is
> not valid. Perhaps I don't have that cert,

that's fine. I created it using a non-trusted test cert.


> but it behaves the same way w/
> and w/o the patch, so it doesn't seem to be a regression.

Great!

Comment 89

[Kai Engert (:KaiE:)]

Some more explanations, starting with a quick summary:

This bug is about signed S/MIME messages.

When creating signed messages, an application has the choice of
- using multipart (content plus detached signature)
- using so called "opaque signing",
  where the message is "encoded before signing"

When Thunderbird signs messages, it always uses the multipart approach.

Other applications may use opaque signing.

One application that I know of supporting opaque signing, which I have used in the past for testing, was Outlook Express. I've learned that Windows 7 still offers this application, as part of Windows Live Essentials, now called Windows (live) Mail.

Somewhere deep in the advanced preferences, Outlook Express / Windows Mail offer a checkbox, that controls whether signed messages will be sent as multipart or opaque (encode before signing).

I know Mozilla Mail has always been able to decode such messages correctly, and show them as "signed".


If we compare the p7m mime parts provided by the reporter of this bug, with the p7m parts produced by Windows Mail, they are described identically at the MIME level. Both are base64 encoded, have mime-type application/pkcs7-mime or application/x-pkcs7-mime (supposed to be equivalent), content-dispositio attachment, and have file extension p7m.

But as we know, the original test case of this bug does not work, while my latest Windows Mail product does work. So, where's the difference?

The difference is the contents of the p7m part.

I decoded both p7m objects with bas64, then derdump.
I can see that both contain equivalently encoded pkcs7-signed-data object, with a pkcs7-data content.

Now let's look at the pkcs7-data content.


Let me simplify the previous part.
Both attachments contain a valid pkcs7 blob.
That pkcs7 blob is a container, that contains signature information, and data.
The container is similarly structured, so let's look at differences in the contained data.


The contents of the p7 container in the original/italian sample message appears to be "Raw Binary Data".

The contents of the p7 container from Windows Mail contains: "A full Mime message with content types" !!!


In my opinion:

The messages originally reported as nonworking are not well formed.

In a message that is supposed to carry meta information that describes each part of the content, you shouldn't simply drop in a binary stream, without content-type description, and expect it to work correctly.

I suspect those messages are out of spec.

Comment 90

[Kai Engert (:KaiE:)]

(In reply to comment #89)
> I suspect those messages are out of spec.

From
  http://www.ietf.org/rfc/rfc3851.txt
section 3:
  The data to be secured is always a canonical MIME entity.

From my understanding, a MIME entity is defined as a content-describing header, plus a body.


In my understanding a raw binary blob is not a MIME entity.

I'd conclude this bug may actually be WONTFIX and the real solution is, the application who produced those messages should be fixed to transform the binary blob into a MIME entity, prior to performing the S/MIME encoding.


However, David, maybe you'll decide that you want to be lenient, and still make it work.

I think that means: Whenever we detect a content piece, that is not a MIME entity, then do not make statements about its type, but treat it as an application/octect type attachment.

(Maybe that's what your patch does already.)

Comment 91

[Serge Gautherie (:sgautherie)]

http://hg.mozilla.org/releases/comm-2.0/rev/d30ecad050e6

Comment 92

[Serge Gautherie (:sgautherie)]

(In reply to comment #91)
> http://hg.mozilla.org/releases/comm-2.0/rev/d30ecad050e6

Arf,
http://hg.mozilla.org/releases/comm-2.0/rev/37de639a6b44
"back out fix for bug 243833 since it caused regressions"

Comment 93

[Giacomo Magnini]

Why is this still resolved fixed if it was backed out?
I tried to open the test email in the bug with today's SM 2.1 build and the attachment doesn't show up.

Comment 94

[David :Bienvenu]

(In reply to comment #93)
> Why is this still resolved fixed if it was backed out?
> I tried to open the test email in the bug with today's SM 2.1 build and the
> attachment doesn't show up.

it's marked fixed because a different fix was landed. I can't speak to SM builds but the attached message is working in TB trunk builds.

Comment 95

[intendentedelleacque]

David, thanks a lot to work on this.
There is still a problem: the p7m attachments are not showed in printing, i guess because of http://mxr.mozilla.org/comm-1.9.2/source/mailnews/mime/emitters/src/nsMimeHtmlEmitter.cpp#532
Could you kindly check this?

Comment 96

[David :Bienvenu]

Attachment: backout fix

Unfortunately, due to regressions, this is going to have to get backed out for Miramar - requesting approval. I'll backout on the trunk now.

Comment 97

[David :Bienvenu]

backed out - http://hg.mozilla.org/comm-central/rev/95ae2e8fd683

Comment 98

[intendentedelleacque]

(In reply to comment #95)
> David, thanks a lot to work on this.
> There is still a problem: the p7m attachments are not showed in printing, i
> guess because of
> http://mxr.mozilla.org/comm-1.9.2/source/mailnews/mime/emitters/src/
> nsMimeHtmlEmitter.cpp#532
> Could you kindly check this?

Since this depends from a different part of code, I0ve filled a new bug  https://bugzilla.mozilla.org/show_bug.cgi?id=659244

Comment 99

[Ludovic Hirlimann [:Usul]]

I've been told that the italian government was using :
http://www.openpec.org/eng/index.shtml (OpenPEC is "THE" open source solution for Certified Email, in italian "Posta Elettronica Certificata" (PEC).) so we even have the source, maybe we should investigate and file a bug and get it fixed.

Comment 100

[Flavio Fanton]

HI,

in a few words, the Italian Certified Electronic Mail (PEC) is a traditional 
e-mail system where the provider signs the emails sent by local users 
and generates new new mail signed certifying the acceptance and delivery.

The signature on the email is a PKCS#7 detach structure and then the 
signature informations are in simime.p7s attach that the security system of TB
seem to manage properly.

PEC is just an RFC, if you want:
http://datatracker.ietf.org/doc/rfc6109/

The use of PEC has helped the spread of digital signature regulated by
Italian laws but in this case the digital signature is a PKCS#7 nodetach 
structure - p7m extension.

So this bug is not a PEC problem but an important problem for italian users.

With openssl you can create a signed file nodetach quickly:
openssl smime -sign -signer cert.pem -inkey key.pem -outform DER -in test.txt -nodetach -out test.txt.p7m

I do not know the TB code but let me know how I can help.


-- flazan

Comment 101

[David :Bienvenu]

backed out from miramar as well.

Comment 102

[David :Bienvenu]

Flazan, thx for your offer of help - this comment is the one that best reflects our understanding of the issue: #c90 - essentially, if the attachment were of type application/octet-stream, users could save/detach it.

Comment 103

[David :Bienvenu]

https://bugzilla.mozilla.org/show_bug.cgi?id=243833#c90

Comment 104

[Kai Engert (:KaiE:)]

Flazan:

In short, if we decode your p7m, we see binary data.

But we expect something else. We expect full MIME encoded data.

So instead of 
  p7m -> binary data

we expect
  p7m -> MIME entity with content type -> binary data

Comment 105

[intendentedelleacque]

(In reply to comment #104)
> Flazan:
> 
> In short, if we decode your p7m, we see binary data.

Yes this is the scenario with p7m attachments in Italy.

My extension "Smart P7M Support" works in this way: if the main content-type of the message is NOT one of the following ones

- multipart/encrypted
- x-pkcs7-mime
- pkcs7-mime

then the TB native support for encrypted content is disabled and so the p7m attachment is showns in attachemnts list.
Otherwise, in the 3 cases above, the support is enabled and TB decodes the message.
Maybe this system is not perfect, but I had no complaints about it.
Could this system be used in the core code?

As alternative, it would be possible that the attachment is shown when decoding fails?

Comment 106

[Flavio Fanton]

When you send an email (no signed/encrypted) with a p7m attachment file 
with TB, it has to build a MIME structure that treats the file as a 
normal binary file then TB has to use
Content-Type: application / octet-stream;
and encode file in base64

With openssl try with
openssl smime -sign -signer cacert.pem -inkey key.pem -outform DER \
-in test.txt -nodetach | base64 -w 78 > test.txt.p7m.base64

now you have to add 

Content-Type: application/octet-stream; name...
Content-Transfer-Encoding: base64
Content-Disposition: ...
...

You can verify mime entity (test.txt.p7m.base64) with:
base64 -d test.txt.p7m.base64 | openssl smime -verify -inform DER \
-noverify -out test.txt.2

Outlook instead uses Content-Type: application/pkcs7-mime;
with base64 encoded for files with extension p7m
but, I believe, application/pkcs7-mime is S/MIME compliants only (not MIME).

However, TB has to manage application/pkcs7-mime; attachment properly 
and to consider it as normal binary files (such as application/octet-stream;)
converted to base64 for transport: in this moment TB
does not care the attachment file contains a PKCS7 structure.


-- flazan

Comment 107

[Kai Engert (:KaiE:)]

Flazan, I don't understand and I am confused.

Why do you try to explain to use what Thunderbird should do when producing messages?

This bug is not about messages that Thunderbird produces.

This bug is about messages that PEC produces, incorrect messages from PEC.

In my understanding, PEC should do what you describe in comment 16 when creating a message.

The problem is that PEC produces p7m messages without mime content type information.

Comment 108

[Flavio Fanton]

Sorry Kay, I try to explain better.

The situation is:
sometimes (with Outlook mail) TB does not show the attachment with p7m extension.

It's important to known:
. PEC never produces messages with p7m attachments.
. PEC produces S/MIME messages only with p7s attachment.

Well, I believe this bug is not related to S/MIME or PEC but only
to messages with attachment p7m:
MIME parsing of TB has to consider the entity with mime 
Content-Type: application/pkcs7-mime
like Content-Type: application/octet-stream 
(both are binary data);

To produce a p7m signed file you can use:
openssl smime -sign -signer cert.pem -inkey key.pem -outform DER -in test.txt -nodetach -out test.txt.p7m

If you can download test.txt.p7m, can verify with
openssl smime -verify -inform DER -in test.txt.p7m -noverify -out test.txt

However if you want, I can give you a test PEC account; contact me by private email.

--flazan

Comment 109

[Flavio Fanton]

Kai, 2 simple considerations more.

1. Outlook should not use Content-Type: application/pkcs7-mime 
in MIME messages, it's about S/MIME messages only
(see http://www.iana.org/assignments/media-types/application/index.html)

2. to verify what in c106 and c108:
send a messagge with p7m attachment with Outlook and receive it with TB 
(you do not see p7m attachment, ok?).

Now you save the message in your local pc and modify it:
replace 
Content-Type: application/pkcs7-mime;
with
Content-Type: application/octet-stream;

Save the messagge and reopen it with TB: now you can see p7m attachment and 
manage it correctly.

I hope this help you.

--flazan

Comment 110

[David :Bienvenu]

Flazan, are you saying that Outlook is the offending party here, that it's the program generating the application/pkcs7-mime content type for a simple p7m attachment?

Comment 111

[Flavio Fanton]

I think so. 

Perhaps others MUA do the same, I don't known.

In any case TB shuld be resolved this situations because to much people use Outlook.


--flazan

Comment 112

[Matteo]

I don't know for the last TB versions, but that was the behaviour of TB too, at least until 3.2a1.
See comment #37 on how to reproduce this bug on TB with a message generated by TB itself.

Regards.
Matteo

Comment 113

[Kai Engert (:KaiE:)]

Flazan, in your comment 108, if your input test.txt is not a MIME message, then you are producing an invalid p7m container.

Whoever is manually creating p7m containers, based on non-MIME data, and attaching it to messages, is breaking expectations.


I think, I hope, I finally understand what this bug is about.

This bug is NOT about email programs creating S/MIME messages.
This bug is NOT about Thunderbird being unable to display S/MIME messages.

As I have shown further above, MS Mail produces messages that Thunderbird can show correctly.
As I know from my past work and testing, the same applies to Outlook Express.


This bug seems to be about the following very special, and unexpected (for me), scenario:

- a user creates a p7m file
- the p7m file is NOT according to standards,
  because the data contained in the p7m is NOT a MIME entity,
  rather it's something else, for example binary data.
- user uses Outlook to compose an email message
- the user attaches the non-standard p7m file that was created
  externally by someone else
- the user sends the message

In this scenario, the following happens:

- Outlook creates a multipart message
- Outlook makes content-type decisions for each part
- for the p7m part, Outlook decides that the content-type
  should be application/pkcs7-mime
- when receiving with Thunderbird, 
  we detect content-type application/pkcs7-mime
- because application/pkcs7-mime means "this is an S/MIME message"
  we process it using our S/MIME decoder
- we are unable to find a MIME entity inside, and we fail

Is the above correct?


If the answer is yes, then my opinion is:

As Flazan said, it's Outlook who makes the decision to use
content-type application/pkcs7-mime.

In my understanding, Windows uses a mapping from file extensions
to mime types.

Can we blame Outlook?
I think, no, we cannot.

I'm not a friend of the Windows world, but if Windows defines
that .p7m is mapped to application/pkcs7-mime,
then that's the way it is, and users have to follow that.

If a user produces non-standard .p7m files (not containing MIME),
then you are breaking the system.

Can we blame Thunderbird?
I think, no, we cannot.

If you are sending application/pkcs7-mime with bad data,
then you cannot expect the software to behave correctly.

So, who is to blame?

I think you should blame the users who produce the non-standard .p7m files, attach them to email files.

You cannot expect that to work, because .p7m / application/pkcs7-mime has a special meaning in email messages.

The users should:
- either use a different file extension for their non-standard files
  (in order to work around Outlook behaviour)
- or wrap the confusing non-S/MIME attachments
  in an archive, for example, create a zip file that contains your
  non-standard files, and send the zip file by email.

If you really think you should be able to send confusing .p7m attachments in email, without the above workarounds, then in my opinion:

Talk to Microsoft, ask them to make Outlook smarter when deciding about the content-type used when creating email messages.


So, having said the above, in my opinion, this bug should be resolved INVALID.


However, if Thunderbird developers really really want to support this broken incoming data, then the following could be done:

- whenever you receive data that says it's application/pkcs7-mime,
  then try to decode it as usual

- after decoding, if the contents are NOT a MIME entity,
  then display the application/pkcs7-mime as a binary attachment.

But I don't know whether that's easy to do.

Comment 114

[Kai Engert (:KaiE:)]

If you don't mind, we are curious WHY the users are doing this.

Why are users creating the non-MIME p7m files?

Comment 115

[Matteo]

(In reply to comment #113)
> This bug is NOT about email programs creating S/MIME messages.
> This bug is NOT about Thunderbird being unable to display S/MIME messages.

This is correct (it's what I said in comment #32); as I said in comment #112, the following isn't correct:
> - user uses Outlook to compose an email message
since it happens also using TB to composing a message with a .p7m not-mime attachment (now I checked also with last TB nightly).


> The users should:
> - either use a different file extension for their non-standard files
> - or wrap the confusing non-S/MIME attachments in an archive and send the zip file by email.

Add the following:
- switch from TB to Outlook, OE or Windows Mail, who all show the attachment;
- go on with the Smart P7M extension, hoping that it will be always developed through future TB versions (only for skilled users, since it's not even in the addons website).


.p7m is the extension choosen by the Italian PA for digital signed (in PKCS#7 format, they say) official documents; documents are signed with appropriate software (and sometimes hardware, like smart cards) and signature will be verified by appropriate software (for example http://www.difesa.it/pki-ff/manuale-verifica-firma/Pagine/apri_p7m.aspx from Italian DoD).
MUAs are just the couriers used to transfer the digital signed document from PA to citizens and from citizens to PA, and from citizens to CPA which then trasmits the file to PA; in this scenario, MUAs aren't supposed to decode or check the .p7m signature.

I don't know which whould be the correct extension for this task, if there is one.

> - after decoding, if the contents are NOT a MIME entity,
>   then display the application/pkcs7-mime as a binary attachment.

That would be good; is maybe what others MUAs do?


Regards.
Matteo

Comment 116

[Flavio Fanton]

(In reply to comment #113)
> This bug seems to be about the following very special, and unexpected (for
> me), scenario:
> 
> - a user creates a p7m file
> - the p7m file is NOT according to standards,
>   because the data contained in the p7m is NOT a MIME entity,
>   rather it's something else, for example binary data.
> - user uses Outlook to compose an email message
> - the user attaches the non-standard p7m file that was created
>   externally by someone else
> - the user sends the message
> 
> In this scenario, the following happens:
> 
> - Outlook creates a multipart message
> - Outlook makes content-type decisions for each part
> - for the p7m part, Outlook decides that the content-type
>   should be application/pkcs7-mime
> - when receiving with Thunderbird, 
>   we detect content-type application/pkcs7-mime
> - because application/pkcs7-mime means "this is an S/MIME message"
>   we process it using our S/MIME decoder
> - we are unable to find a MIME entity inside, and we fail
> 
> Is the above correct?

Yes

> 
> If the answer is yes, then my opinion is:
> 
> As Flazan said, it's Outlook who makes the decision to use
> content-type application/pkcs7-mime.
> 
> In my understanding, Windows uses a mapping from file extensions
> to mime types.
> 
> Can we blame Outlook?
> I think, no, we cannot.

I wrote to MS and I'm waiting for a response but this 
colud be a solution far.

> 
> Can we blame Thunderbird?
> I think, no, we cannot.
> 
> If you are sending application/pkcs7-mime with bad data,
> then you cannot expect the software to behave correctly.

Yes, but let me say if TB manage this unusual situations it 
confirms its flexibility and many Italian user can continue 
using it.

> 
> However, if Thunderbird developers really really want to support this broken
> incoming data, then the following could be done:
> 
> - whenever you receive data that says it's application/pkcs7-mime,
>   then try to decode it as usual
> 
> - after decoding, if the contents are NOT a MIME entity,
>   then display the application/pkcs7-mime as a binary attachment.
> 
> But I don't know whether that's easy to do.

This is very important to us, I hope the patch could be applied.


--flazan

Comment 117

[logan]

This issue occurs even when the .p7m file is attached by a sender using Thunderbird. The program generating these files is Entrust and is used by governments across America due to the fact that we have to use NIST certified encryption applications. We can import the user's certificate (private key I'm assuming) into Thunderbird and it will handle the attachment correctly, but doing this for thousands upon thousands of users is not realistic.

Comment 118

[logan]

The final sentence in my previous post should begin "We can import the recipient's certificate into the recipient's Thunderbird instance."

Comment 119

[David :Bienvenu]

It has proven very difficult to fix this issue without causing regressions, and it's difficult code to work with in the first place. We would appreciate contributors' efforts to fix it, but we need to have unit tests to make sure that we're not regressing anything.

Comment 120

[logan]

All I can really attribute is .p7m files generated by the application in question. There's been discussion about switching to Outlook, but I've generally been able to head that off for our agency. We're using the smart .p7m manager right now, but I had to comb through the source code line by line and modify the update link just to get that approved. I have my fingers crossed that something doesn't pop up where that addon has to be updated and I have to spend half a day combing through the code again. A checkbox in the options to toggle .p7m handling would be a blessing at this point.

Comment 121

[Nelson Bolyard (seldom reads bugmail)]

Please read
http://www.tech-archive.net/Archive/Outlook/microsoft.public.outlook.interop/2004-03/0134.html

Comment 122

[Nelson Bolyard (seldom reads bugmail)]

RFC 2633 says:

   MIME Type                                File Extension

   Application/pkcs7-mime (signedData,      .p7m
   envelopedData)

   Application/pkcs7-mime (degenerate       .p7c
   signedData "certs-only" message)

   Application/pkcs7-signature              .p7s

Comment 123

[Mark Banner (:standard8)]

(In reply to comment #119)
> It has proven very difficult to fix this issue without causing regressions,
> and it's difficult code to work with in the first place. We would appreciate
> contributors' efforts to fix it, but we need to have unit tests to make sure
> that we're not regressing anything.

Given the issues that we've had around fixing this, it feels like we are not able to block on this as there is no clear solution in this case. Therefore setting not blocking.

Comment 124

[Ludovic Hirlimann [:Usul]]

As a bonus point there are two extensions dealing with this specific issue they can be found at :
https://addons.mozilla.org/it/thunderbird/addon/thunderpec/

and 

http://nic-nac-project.de/~kaosmos/p7mHandler.html

Comment 125

[andihartmann]

Please urgently apply this patch 

https://bug243833.bugzilla.mozilla.org/attachment.cgi?id=534526 

to revert http://hg.mozilla.org/releases/comm-release/rev/9bd80ef2728e

to seamonkey too, because the absence of this patch breaks the handling of mime encrypted mails in enigmail:

Write a mail and save it with "encryption on" to Drafts folder. Close the mail window and go to Drafts folder and try to reopen this mail. Enigmail will fail because of this not reverted patch in Seamonkey!

Comment 126

[andihartmann]

To Comment 125: Sorry, I forget to write, that I meant Seamonkey 2.2.

Comment 127

[Jens Hatlak (:InvisibleSmiley)]

(In reply to comment #125)
> revert http://hg.mozilla.org/releases/comm-release/rev/9bd80ef2728e
> 
> to seamonkey too, because the absence of this patch breaks the handling of
> mime encrypted mails in enigmail

AFAICS, the patch is only applied on comm-release but neither comm-beta (SM 2.3), comm-aurora (SM 2.4) or comm-central (SM 2.5) so this should fix itself automatically with the next SeaMonkey version. Note that this patch affected shared code, so it would usually have affected both TB and SM, but in this case TB was not affected because they used comm-miramar for TB 5.

That said, I'll relnote this issue for SM 2.2.

Comment 128

[Matilda]

In Thunderbird 17.0.5 ann SeaMonkey 2.17 (this one just released) the "p7m" issue seems finally fixed.
Thank you so much!!! :)

Comment 129

[Matilda]

Forget it, unfortunately.
On a computer (OS: Windows XP Pro), SeaMonkey can manage attachments "p7m".
In another computer (the same operating system and the same configuration & addons of the browser), SeaMonkey continues to have the issue.
I will do further testing ...

Comment 130

[giulia]

Attachment: this patch will enable enduser to change p7m default handling in tb

I would like to submit a patch for 'solving' this bug that affects italian users.

As you can see in the attached code, it is based on David Bienvenu first approach; I only added an hidden preference (in my test case mailnews.p7m_external) to turn the additional code on/off:

1) if the pref is false (or unset), TB behaves as usual (tb users wordwide will not be affected by this additional code)
2) if the pref is true, the additional code provides changes to show the p7m attachment

The patch covers also bug 659244

Regards

Giulia

Comment 131

[Siuccia]

(In reply to giulia from comment #130)
> Created attachment 745850 [details] [diff] [review]
this patch will enable
> enduser to change p7m default handling in tb

I would like to submit a patch
> for 'solving' this bug that affects italian users.

As you can see in the
> attached code, it is based on David Bienvenu first approach; I only added an
> hidden preference (in my test case mailnews.p7m_external) to turn the
> additional code on/off:

1) if the pref is false (or unset), TB behaves as
> usual (tb users wordwide will not be affected by this additional code)
2) if
> the pref is true, the additional code provides changes to show the p7m
> attachment

The patch covers also bug 659244

Regards

Giulia

----
Resolved!!!
Thank you very mouch for your solution!!!

Comment 132

[nikcol]

Thanks Giulia,
  your solution works even if 'need to create the. dll for each version of thunderbird.

Comment 133

[giulia]

review ping?

Comment 134

[giulia]

review ping ?

Could we try to get this in tb 24 ?

thanks

Giulia

Comment 135

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

comment 199 in the bug makes me nervous about trying to be the reviewer for something I don't understand in a feature I don't use of a product I very rarely use.

Mark, can you help get this issue resolved one way or another. There has been a ton of discussion about it and it seems to be very important to some set of users. Comment 199 makes me feel like I have no clue as to what the implications of the changes are.

Comment 136

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Brian Smith (:briansmith), was [e-mail] (:bsmith) from comment #135)
> comment 199 in the bug makes me nervous about trying to be the reviewer for
> something I don't understand in a feature I don't use of a product I very
> rarely use.

ER, comment 119.

Comment 137

[Mark Banner (:standard8)]

So I definitely think we need unit tests before we can land this. There's examples of tests around in the code base already, and covering all of the examples give, plus any more that can be found would be a good first start. Usul may also have some more examples around.

Comment 138

[Mark Banner (:standard8)]

I probably should say that I don't have much confidence in this area either. However, tests with real-life examples should at least give us some confidence in what we're changing as we'll be able to see the results, and if there is regressions found, then testing and changing without breaking the new cases will be much easier.

Comment 139

[giulia]

Attachment: This patch will enable enduser to change p7m default handling in tb

this patch update reflects the changes in comm-central source tree.

As the previous code, it is based on David Bienvenu first approach; I only added an hidden preference (in my test case mailnews.p7m_external) to turn the additional code on/off:

1) if the pref is false (or unset), TB behaves as usual (tb users wordwide will not be affected by this additional code)
2) if the pref is true, the additional code provides changes to show the p7m attachment

The patch covers also bug 659244

Regards

Giulia

Comment 140

[giulia]

Attachment: test case for the proposed patch

The js attachment contains the test case for the proposed patch. 
The test is performed in three steps on a test message containing a 'strange' p7m attachment (Content-Type: application/pkcs7-mime):

first step) TB default behaviour: the attachment is discarded
second step) setting the preferences to true: the attachment is presented to the user
third step) setting the preference to false: the attachment is discarded (TB default behaviour)

Giulia

Comment 141

[giulia]

As I said in comment 134, could we try to get this patch in tb 24 ?

Tipically, as stated in previous comments, these attachments are carried inside PEC emails.

The table inside the web page http://www.digitpa.gov.it/pec/statistichepec, reports the number of PEC accounts (the third column, 5.416.535 ) and the number of PEC messages exchanged (the last column, 116.580.245) in the latest two-month period measurement (March and April).

Thanks

Giulia

Comment 142

[giulia]

review ping?

Giulia

Comment 143

[giulia]

review ping?

thanks

Giulia

Comment 144

[giulia]

Attachment: Test case for the proposed patch (tb24)

review ping?

I applied the proposed patch on tb24 source tree and, after recompiling Thunderbird under Ubuntu, I tested it successfully.  

The new test-case works under tb24

Giulia

Comment 145

[giulia]

Review ping?

Giulia

Comment 146

[Mark Banner (:standard8)]

Attachment: Combined patch for enabling preference

Sorry for the long delay in getting to this.

I've combined the two patches into one and done a few minor tweaks for our standard coding styles.

I think it is fine to land this as-is, to make it easier for folks to test and find issues. I would suggest that we keep this bug open, and use it as a tracking bug for the issues, until such time that we can be reasonably confident it is working correctly all-round.

Comment 147

[giulia]

Thanks for review !!

Giulia

Comment 148

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/comm-central/rev/d5169e22e52a

Comment 149

[Piviul]

I have just installed TB 24.5 but TB can't show any attachment in the p7mTest.eml downloaded from the comment 23: there is a regression?

Piviu

Comment 150

[giulia]

Hi

the patch is available since TB 27 

Giulia

Comment 151

[Piviul]

thanks Giulia!

Piviul

Comment 152

[Piviul]

...I have just installed TB 30b1 but TB can't show any attachment in the p7mTest.eml downloaded from the comment 23. :(

Piviul

Comment 153

[giulia]

Hi

in order to modify Thunderbird default behaviour, you have to define a new preference (mailnews.p7m_external as described in comment 139). You can find additional information in the blog thunderpec.wordpress.com

Giulia

Comment 154

[Piviul]

Thanks Giulia, in effect seems that, setting mailnew.p7m_external to true in TB 30b1, solves the problem. But I've not understand if there is a way for TB 24.5 to manage correctly p7m attachment installing ThunderPEC or ThunderMon.

Thanks a lot

Piviul

Comment 155

[Wayne Mery (:wsmwk)]

(In reply to Mark Banner (:standard8) from comment #146)
>...
> I think it is fine to land this as-is, to make it easier for folks to test
> and find issues. I would suggest that we keep this bug open, and use it as a
> tracking bug for the issues, until such time that we can be reasonably
> confident it is working correctly all-round.

Are we satisfied there are no more issues?

Comment 156

[giulia]

All is fine.

I'm in contact with 'happy' italian people moving from TB24 to TB31 in order to use a bug243833-free Thunderbird version

Thanks

Giulia

PS. This patch solves also bug659244

Comment 157

[nikcol]

 'problem solved'....;-)

Comment 159

[Wayne Mery (:wsmwk)]

(In reply to Mark Banner (:standard8) from comment #146)
> Created attachment 819510 [details] [diff] [review]
> Combined patch for enabling preference
> 
> Sorry for the long delay in getting to this.
> 
> I've combined the two patches into one and done a few minor tweaks for our
> standard coding styles.
> 
> I think it is fine to land this as-is, to make it easier for folks to test
> and find issues. I would suggest that we keep this bug open, and use it as a
> tracking bug for the issues, until such time that we can be reasonably
> confident it is working correctly all-round.

Mark, given the previous comments, shall we close this fixed?

Comment 160

[Mark Banner (:standard8)]

Yeah, might as well.

Comment 161

[ced]

To the ones affected by this bug, i've added some notes to #890355 (to me the same bug) since this is already closed and became far too big to handle.

Comment 163

[giulia]

In order to simplify italian users, the P7MON Thunderbird extesnsion is available on the Mozilla addon site (https://addons.mozilla.org/it/thunderbird/addon/p7mon/)

This extension creates and sets the new hidden preference at installation time, so no user intervention is needed

Giulia