[Oisf-users] Alerts on long running flow

Wed Oct 31 21:23:02 UTC 2018

Thanks, Andreas.  I created
https://redmine.openinfosecfoundation.org/issues/2656 and attached pcaps to
the ticket.

-- 
Eric Urban
University Information Security | Office of Information Technology |
it.umn.edu
University of Minnesota | umn.edu
eurban at umn.edu

On Mon, Oct 29, 2018 at 4:23 PM Andreas Herz <andi at geekosphere.org> wrote:

> Hi Eric,
>
> sorry for the rather late reply. Would you mind creating a redmine issue
> with that? And are you able to share the pcap (maybe personal to one of
> the OISF)?
>
> It might be (just my guess) rather hard to determine the root cause of
> that and how to fix it.
>
> On 10/10/18 at 15:03, Eric Urban wrote:
> > Hello,
> >
> > I am trying to make sense of behavior we saw in our environment related
> to
> > alerts on a long running flow and am hoping for some input on what may be
> > going on.
> >
> > The somewhat short version of the story is that in our environment we
> have
> > two sensor sets that are configured to receive identical copies of
> > traffic.  The sets are identical in terms of hardware, OS, Suricata
> version
> > (4.0.5), and Suricata configuration.  This week we saw a large difference
> > in the number of alerts between sensor sets and these alerts were all
> > produced from a single flow of SMB traffic.
> >
> > I confirmed this flow to be going to both sensors in our connection logs,
> > and also captured traffic of this long running connection directly on the
> > sensor not producing alerts.  During my investigation, I ran into
> something
> > that could potentially explain what happened if there were a few packets
> > dropped on one side.  I found this while running the packet captures
> > through Suricata in offline/pcap mode.  It appears the point at which the
> > flow is captured determines whether or not subsequent alerts are
> > generated.  I understand that in many cases capturing midstream traffic
> can
> > determine whether alerts are triggered, especially where rules have
> content
> > modifiers that require content be at a specific point in the payload.
> > However, my confusion comes from a test where I ran these captures
> through
> > Suricata with the only difference of one packet at the beginning, which
> > ends up causing a different SMB message to start the capture.  In each of
> > these cases neither packet should match the particular SMB rule
> generating
> > these alerts.
> >
> > What I am seeing may be working as designed, but I am having trouble
> > feeling comfortable that something isn't wrong.
> >
> > Additional details of this situation are below, but a summary of some of
> > the things I am looking for are:
> > 1. Is the payload of a single connection always treated as one or are
> there
> > some cases where it is split into multiple payloads?  This is important
> > because it determines how rules may trigger alerts where keywords like
> > depth are used.
> >
> > 2. Does Suricata treat connections starting with SMB Close Response
> > messages in a special way?  I could not find anything in the code
> > indicating it does.
> >
> > 3. Does the problem as I have described it below sound like this is an
> > issue with the rule itself, how rules are used to inspect the traffic, or
> > something else?  The fact one sensor set saw the traffic and the other
> > didn't suggests dropped packets, but I am looking for things beyond just
> > environment based on what am I seeing running these captures through
> > Suricata.
> >
> >
> > Here is my long story about a long running connection:
> >
> > We had a long running connection of about 7 days generate as many as a
> few
> > thousand alerts per hour only on one of our sensor sets.  The connection
> > was being used to transfer data via SMB. It was being treated by Suricata
> > as a single flow so has the same flow id throughout.
> >
> > I gathered two separate packet captures that were run about 10 minutes
> > apart on the sensor not generating alerts. A host filter was used to
> > capture only the desired traffic.  Each capture ended up being about
> 500MB
> > in size.  When I ran the first capture through an instance of Suricata
> > using pcap/offline mode, it triggered a several hundred SMB alerts, which
> > were mostly for Emerging Threats SIDs 2025709
> > <http://doc.emergingthreats.net/bin/view/Main/2025709> "SMB2 Create Req
> for
> > DLL" and some also for 2025705
> > <http://doc.emergingthreats.net/bin/view/Main/2025705> "SMB2 Create Req
> for
> > .ps1".  When I ran the second capture through Suricata, to my surprise
> > there were zero alerts generated.  I inspected the second capture with
> > Wireshark to look for differences that might explain why it produced no
> > alerts and found there were definitely DLL files being transferred after
> an
> > SMB2 Create Request.  As a test I removed a chunk of packets from the
> > beginning of the second capture in case something was throwing it off and
> > ran it through again.  That did the trick and several hundred alerts were
> > generated.  I then removed only the first packet, which happened to be an
> > SMB2 Close Response.  I ran this modified capture through Suricata and
> sure
> > enough there were alerts.  So I decided to try this with a few other SMB2
> > Close Responses in case something was up with the first one and I was
> able
> > to reproduce the situation each time where that specific message type at
> > the start of the capture resulted in no alerts, even with several hundred
> > situations of DLL files being transferred from a Create Request.  I also
> > tested using another SMB message type at the start the capture in case it
> > wasn't just the Close Response ones affecting it.  I found a case where
> > trimming a single packet of a Close Response resulted in the capture
> > starting with a Read Request SMB message.  The capture starting with that
> > message produced all expected alerts when I ran it through Suricata.
> >
> > Here is the full Emerging Threats rule with SID 2025709 from
> > http://doc.emergingthreats.net/bin/view/Main/2025709 (formatted for
> > readability):
> > alert tcp any any -> $HOME_NET 445
> > (msg:"ET POLICY SMB2 NT Create AndX? Request For a DLL File - Possible
> > Lateral Movement";
> > flow:established,to_server;
> > content:"SMB"; depth:8;
> > content:"|05 00|"; distance:8; within:2;
> > content:"|00 2E 00|d|00|l|00|l|00|"; nocase; distance:0;
> > metadata: former_category POLICY;
> > classtype:bad-unknown;
> > sid:2025709;
> > rev:2;
> > metadata:affected_product Windows_XP_Vista_7_8_10_Server_32_64_Bit,
> > attack_target SMB_Client, deployment Internal, signature_severity Minor,
> > created_at 2018_07_16, updated_at 2018_07_16;)
> >
> > The first Content has a depth content modifier of 8 which according to
> > Suricata documentation at
> >
> https://suricata.readthedocs.io/en/suricata-4.0.5/rules/payload-keywords.html#depth
> > means that "The number after depth designates how many bytes from the
> > beginning of the payload will be checked".  After that we see another
> > Content of |05 00| that must exist a distance of 8 bytes away and those
> > next 2 bytes should be the 05 and 00, which is looking for a Create
> > Response type SMB message,.  Finally, after that another Content exists
> > looking essentially for the string ".dll" that can exist at any point
> after
> > that in the payload since the distance is 0.
> >
> > A Close Response has the type of 0x06.  There is a reference of these at
> >
> https://github.com/OISF/suricata/blob/ec77632e84a106ddbcd0baef4e4368b4fe5c5f9e/src/app-layer-smb.h#L97
> > and another reference at https://wiki.wireshark.org/SMB2.  I had one
> > capture start with a Close Response and the other I trimmed only that
> frame
> > so the capture started with a Read Request.  A Read Request is of type
> > 0x08, so has a 08 at a distance of 8 away from "SMB".  Neither one of
> these
> > message types should match this rule, yet somehow starting the capture
> with
> > one or the other changes the behavior significantly.
> >
> > It is now not clear to me what constitutes a payload inside of a
> > connection.  If there is only one payload per connection, then how do any
> > of these situations have alerts triggered since SMB with the following 05
> > 00 bytes do not start any of these?  If there are multiple payloads per
> > connection, which I would assume is handled by the application layer
> > detectors, then why is this Close Response causing all detections to
> fail?
> >
> > I did try adjusting the rule signature protocol to use smb instead of tcp
> > in the hope that could make a difference, but it did not look to change
> > anything for me.
> >
> > I enabled debug with Suricata and searched for any log messages with smb
> > but did not see any differences.
> >
> > I also tried running the captures through 3.1 and the same behavior was
> > present.  I was able to recreate this using a vanilla config that only
> had
> > the HOME_NET, rule location, vlan.use-for-tracking values modified, so I
> > don't feel anything specific to our configuration is causing this to
> occur.
> >
> > Thank you for reading!
> >
> > --
> > Eric Urban
> > University Information Security | Office of Information Technology |
> > it.umn.edu
> > University of Minnesota | umn.edu
> > eurban at umn.edu
>
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> > List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >
> > Conference: https://suricon.net
> > Trainings: https://suricata-ids.org/training/
>
>
> --
> Andreas Herz
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20181031/fcafa35e/attachment-0001.html>