Decryption seems to be limited to the first 1,000,000,000 bytes


I have a 1.6gig capture (SSAS DB sync via HTTPS).
When I attempt to decrypt it, it creates a destination file of 1,000,000,000 bytes (~953Meg).

Aside: I've not had a sucessful decrypt yet, but when I get some detail on that, I'll log it as a separate issue, but it could be because it's getting to the 1,000,000,001st byte and crashing.

file attachments


PaulLong wrote Dec 17, 2012 at 1:53 PM

Yes, you are correct. I can raise it up to 2gig, but there is a limit which is somewhat variable. I will try to get an update posted, however in the meantime you can build the project. If that's not an option for you, let me know and we could work to get you a private version.


CraigHumphrey wrote Dec 17, 2012 at 7:31 PM

Thanks for your response Paul, I probably could manage to compile this if I dug deep enough into cobwebs of my VS.Net skills, but if you've got a moment to build a 2gig edition, that would be a real time-saver.


wrote Dec 17, 2012 at 9:31 PM

PaulLong wrote Dec 17, 2012 at 9:31 PM

I've attached an x64 zip file with an install MSI. Hopefully this will allow you to save up to 2gigs. Please let me know if it works for you.


CraigHumphrey wrote Dec 18, 2012 at 8:55 PM

Thanks for that Paul. I gave it a go and it converted my 1.6gig (1,692,539,750 bytes) HTTPS cap into a 2gig (2,151,018,561 bytes) HTTPS cap file...

Either I messed something up or the decrypt isn't working correctly.

Note: I can see correctly decrypted packets in the debug file, but the "decrypted" cap file still appears as HTTPS and appears to be encrypted when I load it into NetMon.

I'll try again with a small cap, just in case it's a size related issue.

CraigHumphrey wrote Dec 18, 2012 at 9:43 PM

Hey Paul,

just confirming, if I take the first 100 or so frames of my 1.6gig cap file and decrypt it, it decrypts correctly.
But if I try the whole thing, the resulting cap file seems to be still encrypted.

Not sure if this is related to the massive size of my original cap file.

let me know if you want me to log this as a separate issue.


CraigHumphrey wrote Dec 19, 2012 at 12:40 AM

One more thing for today... I noticed that it's pre-allocating the size of the decrypted file.
Even with smaller source CAP files, it's creating a 2gig file up front.

I also just noticed today that the original encrypted frames are still in the decrypted cap file, plus the decrypted version of each frame. Which suggests my 1.6gig cap probably needs a 3.2+gig output file.

Just thinking outloud :)

CraigHumphrey wrote Dec 19, 2012 at 12:56 AM

heh heh, it gets better.
where data spans more than one SSL/TLS frame, in the decrypted cap file, you get the original "continuation" frame, a combined TLS frame and a combined decrypted HTTP frame.

So that's three copies of the data. My 1.6gig cap probably needs more like 5gig...

FYI I just ran a capture against a smaller SSAS HTTPS DB sync, only 12meg of traffic.
While it pre-allocated a 2gig output file, once it was done, the decypted cap file was 26.5meg, so a bit more than double. So this makes no sense about output size...

wrote Dec 19, 2012 at 2:23 PM

PaulLong wrote Dec 19, 2012 at 2:23 PM

You are right, the expert includes the original messages plus each layer of reassembly. Unfortunately the .cap file format has a limit since it uses a DWORD for each frame to reference the offset in the file. So 4 gigs is the absolute maximum addressable size, but as the frame table grows, this size shrinks.

Also the allocation of the size of 2gigs at first is how the API works. I'm not sure why, but let me know if this is a problem.

The only solution I could think of was to write a chained capture instead. I've attached that version. Of course if you want to see it altogether you could filter our all the DecryptHeader messages and use NMCap to create a file using all the chained files as input.


CraigHumphrey wrote Dec 20, 2012 at 12:01 AM

Thanks Paul,

that worked better.

I now have a 2gig and a 1.2gig decrypt file.

However, only the first 100 or so frames have been sucessfully decrypted.
It looks like the agregated TLS frames are still generated (though later they seem to have changed to SSL2) but not HTTP frames.

Is there a limit to the frame/packet size that can be decrypted? These SSAS sync packets often seem to be >64K.


CraigHumphrey wrote Dec 20, 2012 at 1:57 AM

One last correction...

There are correctly decrypted HTTP frames/packets later on in the CAPs, but they get fewer and fewer.

Let me know if there's anything I can send through that might help figure this out...

PaulLong wrote Dec 20, 2012 at 4:33 PM

I don't think there is any limit on the message size, but anything possible. I'm actually only maintaining the code right now so there could be some pieces I don't understand.

Can you tell me something about the traffic you are trying to decrypt? Is it one long conversation, or multiple separate TLS conversations?

wrote Feb 14, 2013 at 7:16 PM