[Oisf-users] Alerts on long running flow

[Andreas Herz]

Hi Eric,

sorry for the rather late reply. Would you mind creating a redmine issue
with that? And are you able to share the pcap (maybe personal to one of
the OISF)?

It might be (just my guess) rather hard to determine the root cause of
that and how to fix it.

On 10/10/18 at 15:03, Eric Urban wrote:
> Hello,
> 
> I am trying to make sense of behavior we saw in our environment related to
> alerts on a long running flow and am hoping for some input on what may be
> going on.
> 
> The somewhat short version of the story is that in our environment we have
> two sensor sets that are configured to receive identical copies of
> traffic.  The sets are identical in terms of hardware, OS, Suricata version
> (4.0.5), and Suricata configuration.  This week we saw a large difference
> in the number of alerts between sensor sets and these alerts were all
> produced from a single flow of SMB traffic.
> 
> I confirmed this flow to be going to both sensors in our connection logs,
> and also captured traffic of this long running connection directly on the
> sensor not producing alerts.  During my investigation, I ran into something
> that could potentially explain what happened if there were a few packets
> dropped on one side.  I found this while running the packet captures
> through Suricata in offline/pcap mode.  It appears the point at which the
> flow is captured determines whether or not subsequent alerts are
> generated.  I understand that in many cases capturing midstream traffic can
> determine whether alerts are triggered, especially where rules have content
> modifiers that require content be at a specific point in the payload.
> However, my confusion comes from a test where I ran these captures through
> Suricata with the only difference of one packet at the beginning, which
> ends up causing a different SMB message to start the capture.  In each of
> these cases neither packet should match the particular SMB rule generating
> these alerts.
> 
> What I am seeing may be working as designed, but I am having trouble
> feeling comfortable that something isn't wrong.
> 
> Additional details of this situation are below, but a summary of some of
> the things I am looking for are:
> 1. Is the payload of a single connection always treated as one or are there
> some cases where it is split into multiple payloads?  This is important
> because it determines how rules may trigger alerts where keywords like
> depth are used.
> 
> 2. Does Suricata treat connections starting with SMB Close Response
> messages in a special way?  I could not find anything in the code
> indicating it does.
> 
> 3. Does the problem as I have described it below sound like this is an
> issue with the rule itself, how rules are used to inspect the traffic, or
> something else?  The fact one sensor set saw the traffic and the other
> didn't suggests dropped packets, but I am looking for things beyond just
> environment based on what am I seeing running these captures through
> Suricata.
> 
> 
> Here is my long story about a long running connection:
> 
> We had a long running connection of about 7 days generate as many as a few
> thousand alerts per hour only on one of our sensor sets.  The connection
> was being used to transfer data via SMB. It was being treated by Suricata
> as a single flow so has the same flow id throughout.
> 
> I gathered two separate packet captures that were run about 10 minutes
> apart on the sensor not generating alerts. A host filter was used to
> capture only the desired traffic.  Each capture ended up being about 500MB
> in size.  When I ran the first capture through an instance of Suricata
> using pcap/offline mode, it triggered a several hundred SMB alerts, which
> were mostly for Emerging Threats SIDs 2025709
> <http://doc.emergingthreats.net/bin/view/Main/2025709> "SMB2 Create Req for
> DLL" and some also for 2025705
> <http://doc.emergingthreats.net/bin/view/Main/2025705> "SMB2 Create Req for
> .ps1".  When I ran the second capture through Suricata, to my surprise
> there were zero alerts generated.  I inspected the second capture with
> Wireshark to look for differences that might explain why it produced no
> alerts and found there were definitely DLL files being transferred after an
> SMB2 Create Request.  As a test I removed a chunk of packets from the
> beginning of the second capture in case something was throwing it off and
> ran it through again.  That did the trick and several hundred alerts were
> generated.  I then removed only the first packet, which happened to be an
> SMB2 Close Response.  I ran this modified capture through Suricata and sure
> enough there were alerts.  So I decided to try this with a few other SMB2
> Close Responses in case something was up with the first one and I was able
> to reproduce the situation each time where that specific message type at
> the start of the capture resulted in no alerts, even with several hundred
> situations of DLL files being transferred from a Create Request.  I also
> tested using another SMB message type at the start the capture in case it
> wasn't just the Close Response ones affecting it.  I found a case where
> trimming a single packet of a Close Response resulted in the capture
> starting with a Read Request SMB message.  The capture starting with that
> message produced all expected alerts when I ran it through Suricata.
> 
> Here is the full Emerging Threats rule with SID 2025709 from
> http://doc.emergingthreats.net/bin/view/Main/2025709 (formatted for
> readability):
> alert tcp any any -> $HOME_NET 445
> (msg:"ET POLICY SMB2 NT Create AndX? Request For a DLL File - Possible
> Lateral Movement";
> flow:established,to_server;
> content:"SMB"; depth:8;
> content:"|05 00|"; distance:8; within:2;
> content:"|00 2E 00|d|00|l|00|l|00|"; nocase; distance:0;
> metadata: former_category POLICY;
> classtype:bad-unknown;
> sid:2025709;
> rev:2;
> metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit,
> attack_target SMB_Client, deployment Internal, signature_severity Minor,
> created_at 2018_07_16, updated_at 2018_07_16;)
> 
> The first Content has a depth content modifier of 8 which according to
> Suricata documentation at
> https://suricata.readthedocs.io/en/suricata-4.0.5/rules/payload-keywords.html#depth
> means that "The number after depth designates how many bytes from the
> beginning of the payload will be checked".  After that we see another
> Content of |05 00| that must exist a distance of 8 bytes away and those
> next 2 bytes should be the 05 and 00, which is looking for a Create
> Response type SMB message,.  Finally, after that another Content exists
> looking essentially for the string ".dll" that can exist at any point after
> that in the payload since the distance is 0.
> 
> A Close Response has the type of 0x06.  There is a reference of these at
> https://github.com/OISF/suricata/blob/ec77632e84a106ddbcd0baef4e4368b4fe5c5f9e/src/app-layer-smb.h#L97
> and another reference at https://wiki.wireshark.org/SMB2.  I had one
> capture start with a Close Response and the other I trimmed only that frame
> so the capture started with a Read Request.  A Read Request is of type
> 0x08, so has a 08 at a distance of 8 away from "SMB".  Neither one of these
> message types should match this rule, yet somehow starting the capture with
> one or the other changes the behavior significantly.
> 
> It is now not clear to me what constitutes a payload inside of a
> connection.  If there is only one payload per connection, then how do any
> of these situations have alerts triggered since SMB with the following 05
> 00 bytes do not start any of these?  If there are multiple payloads per
> connection, which I would assume is handled by the application layer
> detectors, then why is this Close Response causing all detections to fail?
> 
> I did try adjusting the rule signature protocol to use smb instead of tcp
> in the hope that could make a difference, but it did not look to change
> anything for me.
> 
> I enabled debug with Suricata and searched for any log messages with smb
> but did not see any differences.
> 
> I also tried running the captures through 3.1 and the same behavior was
> present.  I was able to recreate this using a vanilla config that only had
> the HOME_NET, rule location, vlan.use-for-tracking values modified, so I
> don't feel anything specific to our configuration is causing this to occur.
> 
> Thank you for reading!
> 
> -- 
> Eric Urban
> University Information Security | Office of Information Technology |
> it.umn.edu
> University of Minnesota | umn.edu
> eurban at umn.edu

> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> 
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/


-- 
Andreas Herz