[Oisf-users] HTTP and DNS alert and captures not working

Sun Jun 26 21:41:34 UTC 2016

On Sun, Jun 26, 2016 at 11:21 PM, Cooper F. Nelson <cnelson at ucsd.edu> wrote:
> Hi Caesar,
>
> I've had circumstantial evidence for awhile that the 3.1 release isn't
> properly logging HTTP requests.  I've since confirmed it via this script:
>
>> $ for i in {1..100}; do curl -s http://www.berkeley.edu/$i > /dev/null; sleep 1; done
>
> Checking the logs shows a large number of missing http requests, as
> wells as bad data in the http return code field.
>
>> 06/26/2016-20:01:47.568056 - HTTP/1.1 GET www.berkeley.edu /1 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 21549 132.239.x.x:62124 -> 128.32.203.137:80
>> 06/26/2016-20:01:48.823252 - HTTP/1.1 GET www.berkeley.edu /2 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 25734 132.239.x.x:65175 -> 128.32.203.137:80
>> 06/26/2016-20:01:51.333115 - HTTP/1.1 GET www.berkeley.edu /4 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 12861 132.239.x.x:55097 -> 128.32.203.137:80
>> 06/26/2016-20:01:52.615975 - HTTP/1.1 GET www.berkeley.edu /5 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 - 0 132.239.x.x:64705 -> 128.32.203.137:80
>> 06/26/2016-20:01:53.865529 - HTTP/1.1 GET www.berkeley.edu /6 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 25734 132.239.x.x:52871 -> 128.32.203.137:80
>> 06/26/2016-20:01:55.096402 - HTTP/1.1 GET www.berkeley.edu /7 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 25734 132.239.x.x:61459 -> 128.32.203.137:80
>> 06/26/2016-20:02:01.504810 - HTTP/1.1 GET www.berkeley.edu /12 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 - 0 132.239.x.x:61103 -> 128.32.203.137:80
>
> What's odd is that shutting down the engine after the test causes some
> of the "missing" log entries to be spooled to disk, out of sequence.
>
>> 06/26/2016-20:15:57.862449 - HTTP/1.1 GET www.berkeley.edu /98 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 14308 132.239.x.x:60268 -> 128.32.203.137:80
>> 06/26/2016-20:15:58.463770 - HTTP/1.1 GET www.berkeley.edu /99 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 14308 132.239.x.x:61915 -> 128.32.203.137:80
>> 06/26/2016-20:15:58.687963 - HTTP/1.1 GET www.berkeley.edu /100 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 25734 132.239.x.x:65334 -> 128.32.203.137:80
>> 06/26/2016-20:16:20.060538 - HTTP/1.1 GET www.berkeley.edu /60 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 15756 132.239.x.x:55242 -> 128.32.203.137:80
>> 06/26/2016-20:16:20.061081 - HTTP/1.1 GET www.berkeley.edu /84 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 15756 132.239.x.x:64006 -> 128.32.203.137:80
>> 06/26/2016-20:16:20.063309 - HTTP/1.1 GET www.berkeley.edu /18 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 17204 132.239.x.x:60560 -> 128.32.203.137:80
>
> As a sanity check I ran suricata with a bpf filter to only monitor the
> single host making the http requests.  The behavior still happens, but
> the number of requests dropped isn't as high.
>
> I also turned on the stream and http events rules and logged the
> following, also only monitoring traffic for a single host:
>
>> 06/26/2016-20:43:28.626337  [**] [1:2013028:4] ET POLICY curl User-Agent Outbound [**] [Classification: Attempted Information Leak] [Priority: 2] {TCP} 132.239.x.x:61550 -> 128.32.203.137:80
>> 06/26/2016-20:43:28.903012  [**] [1:2210016:1] SURICATA STREAM CLOSEWAIT FIN out of window [**] [Classification: (null)] [Priority: 3] {TCP} 128.32.203.137:80 -> 132.239.x.x:51697
>> 06/26/2016-20:43:29.430583  [**] [1:2210000:1] SURICATA STREAM 3way handshake with ack in wrong dir [**] [Classification: (null)] [Priority: 3] {TCP} 128.32.203.137:80 -> 132.239.x.x:58191
>> 06/26/2016-20:43:29.635260  [**] [1:2210010:1] SURICATA STREAM 3way handshake wrong seq wrong ack [**] [Classification: (null)] [Priority: 3] {TCP} 132.239.x.x:58191 -> 128.32.203.137:80
>> 06/26/2016-20:43:28.640655  [**] [1:2210032:1] SURICATA STREAM FIN1 FIN with wrong seq [**] [Classification: (null)] [Priority: 3] {TCP} 132.239.x.x:61550 -> 128.32.203.137:80
>
> So something odd is happening.  We've recently changed our monitoring
> topology so it's possible that is introducing some problems, so I'll
> investigate that this week.
>
> I'm cc:'ing Victor and Peter to see if they have any idea what is
> happening.  It would also be a big help if other 3.1 users attempted to
> replicate this issue.  I've also verified that this behavior happens
> with both the hyperscan and ac algos.

@Cooper - If i am not wrong you are on kernel > 4.2 and using
af-packet. There is a bug in the kernel with regards to symmetric flow
hashing for afpacket/suricata. As a test it would be much appreciated
if you can please try kernel 4.2 or lower and see if it makes any
difference for you?

@Ceaser - do you have your NIC offloading disabled? (ethtool -k eth0)

Thank you

>
> -Coop
>
> On 6/22/2016 6:56 PM, Caesar Samsi wrote:
>> I still have my 3.0.1 build which works well.
>>
>> One sanity check I do when installing a new version (3.1 in this instance) is to do:
>> 1. curl -A “BlackSun” google.com <http://google.com/>
>> 2. host google.com <http://google.com/>
>>
>> First one is to supported to trigger possible trojan horse alert.
>> Second one is to capture dns traffic.
>>
>> Neither works in 3.1?
>>
>> I’m sure I’m just missing configuration.
>>
>> Help?
>>
>>
>>
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
>>
>
>
> --
> Cooper Nelson
> Network Security Analyst
> UCSD ITS Security Team
> cnelson at ucsd.edu x41042
>

-- 
Regards,
Peter Manev