[Oisf-users] Alerts on long running flow

[Eric Urban]

Hello,

I am trying to make sense of behavior we saw in our environment related to
alerts on a long running flow and am hoping for some input on what may be
going on.

The somewhat short version of the story is that in our environment we have
two sensor sets that are configured to receive identical copies of
traffic.  The sets are identical in terms of hardware, OS, Suricata version
(4.0.5), and Suricata configuration.  This week we saw a large difference
in the number of alerts between sensor sets and these alerts were all
produced from a single flow of SMB traffic.

I confirmed this flow to be going to both sensors in our connection logs,
and also captured traffic of this long running connection directly on the
sensor not producing alerts.  During my investigation, I ran into something
that could potentially explain what happened if there were a few packets
dropped on one side.  I found this while running the packet captures
through Suricata in offline/pcap mode.  It appears the point at which the
flow is captured determines whether or not subsequent alerts are
generated.  I understand that in many cases capturing midstream traffic can
determine whether alerts are triggered, especially where rules have content
modifiers that require content be at a specific point in the payload.
However, my confusion comes from a test where I ran these captures through
Suricata with the only difference of one packet at the beginning, which
ends up causing a different SMB message to start the capture.  In each of
these cases neither packet should match the particular SMB rule generating
these alerts.

What I am seeing may be working as designed, but I am having trouble
feeling comfortable that something isn't wrong.

Additional details of this situation are below, but a summary of some of
the things I am looking for are:
1. Is the payload of a single connection always treated as one or are there
some cases where it is split into multiple payloads?  This is important
because it determines how rules may trigger alerts where keywords like
depth are used.

2. Does Suricata treat connections starting with SMB Close Response
messages in a special way?  I could not find anything in the code
indicating it does.

3. Does the problem as I have described it below sound like this is an
issue with the rule itself, how rules are used to inspect the traffic, or
something else?  The fact one sensor set saw the traffic and the other
didn't suggests dropped packets, but I am looking for things beyond just
environment based on what am I seeing running these captures through
Suricata.


Here is my long story about a long running connection:

We had a long running connection of about 7 days generate as many as a few
thousand alerts per hour only on one of our sensor sets.  The connection
was being used to transfer data via SMB. It was being treated by Suricata
as a single flow so has the same flow id throughout.

I gathered two separate packet captures that were run about 10 minutes
apart on the sensor not generating alerts. A host filter was used to
capture only the desired traffic.  Each capture ended up being about 500MB
in size.  When I ran the first capture through an instance of Suricata
using pcap/offline mode, it triggered a several hundred SMB alerts, which
were mostly for Emerging Threats SIDs 2025709
<http://doc.emergingthreats.net/bin/view/Main/2025709> "SMB2 Create Req for
DLL" and some also for 2025705
<http://doc.emergingthreats.net/bin/view/Main/2025705> "SMB2 Create Req for
.ps1".  When I ran the second capture through Suricata, to my surprise
there were zero alerts generated.  I inspected the second capture with
Wireshark to look for differences that might explain why it produced no
alerts and found there were definitely DLL files being transferred after an
SMB2 Create Request.  As a test I removed a chunk of packets from the
beginning of the second capture in case something was throwing it off and
ran it through again.  That did the trick and several hundred alerts were
generated.  I then removed only the first packet, which happened to be an
SMB2 Close Response.  I ran this modified capture through Suricata and sure
enough there were alerts.  So I decided to try this with a few other SMB2
Close Responses in case something was up with the first one and I was able
to reproduce the situation each time where that specific message type at
the start of the capture resulted in no alerts, even with several hundred
situations of DLL files being transferred from a Create Request.  I also
tested using another SMB message type at the start the capture in case it
wasn't just the Close Response ones affecting it.  I found a case where
trimming a single packet of a Close Response resulted in the capture
starting with a Read Request SMB message.  The capture starting with that
message produced all expected alerts when I ran it through Suricata.

Here is the full Emerging Threats rule with SID 2025709 from
http://doc.emergingthreats.net/bin/view/Main/2025709 (formatted for
readability):
alert tcp any any -> $HOME_NET 445
(msg:"ET POLICY SMB2 NT Create AndX? Request For a DLL File - Possible
Lateral Movement";
flow:established,to_server;
content:"SMB"; depth:8;
content:"|05 00|"; distance:8; within:2;
content:"|00 2E 00|d|00|l|00|l|00|"; nocase; distance:0;
metadata: former_category POLICY;
classtype:bad-unknown;
sid:2025709;
rev:2;
metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit,
attack_target SMB_Client, deployment Internal, signature_severity Minor,
created_at 2018_07_16, updated_at 2018_07_16;)

The first Content has a depth content modifier of 8 which according to
Suricata documentation at
https://suricata.readthedocs.io/en/suricata-4.0.5/rules/payload-keywords.html#depth
means that "The number after depth designates how many bytes from the
beginning of the payload will be checked".  After that we see another
Content of |05 00| that must exist a distance of 8 bytes away and those
next 2 bytes should be the 05 and 00, which is looking for a Create
Response type SMB message,.  Finally, after that another Content exists
looking essentially for the string ".dll" that can exist at any point after
that in the payload since the distance is 0.

A Close Response has the type of 0x06.  There is a reference of these at
https://github.com/OISF/suricata/blob/ec77632e84a106ddbcd0baef4e4368b4fe5c5f9e/src/app-layer-smb.h#L97
and another reference at https://wiki.wireshark.org/SMB2.  I had one
capture start with a Close Response and the other I trimmed only that frame
so the capture started with a Read Request.  A Read Request is of type
0x08, so has a 08 at a distance of 8 away from "SMB".  Neither one of these
message types should match this rule, yet somehow starting the capture with
one or the other changes the behavior significantly.

It is now not clear to me what constitutes a payload inside of a
connection.  If there is only one payload per connection, then how do any
of these situations have alerts triggered since SMB with the following 05
00 bytes do not start any of these?  If there are multiple payloads per
connection, which I would assume is handled by the application layer
detectors, then why is this Close Response causing all detections to fail?

I did try adjusting the rule signature protocol to use smb instead of tcp
in the hope that could make a difference, but it did not look to change
anything for me.

I enabled debug with Suricata and searched for any log messages with smb
but did not see any differences.

I also tried running the captures through 3.1 and the same behavior was
present.  I was able to recreate this using a vanilla config that only had
the HOME_NET, rule location, vlan.use-for-tracking values modified, so I
don't feel anything specific to our configuration is causing this to occur.

Thank you for reading!

-- 
Eric Urban
University Information Security | Office of Information Technology |
it.umn.edu
University of Minnesota | umn.edu
eurban at umn.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20181010/72585448/attachment.html>