[Oisf-users] HTTP and DNS alert and captures not working

Sun Jun 26 21:21:22 UTC 2016

Hi Caesar,

I've had circumstantial evidence for awhile that the 3.1 release isn't
properly logging HTTP requests.  I've since confirmed it via this script:

> $ for i in {1..100}; do curl -s http://www.berkeley.edu/$i > /dev/null; sleep 1; done

Checking the logs shows a large number of missing http requests, as
wells as bad data in the http return code field.

> 06/26/2016-20:01:47.568056 - HTTP/1.1 GET www.berkeley.edu /1 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 21549 132.239.x.x:62124 -> 128.32.203.137:80
> 06/26/2016-20:01:48.823252 - HTTP/1.1 GET www.berkeley.edu /2 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 25734 132.239.x.x:65175 -> 128.32.203.137:80
> 06/26/2016-20:01:51.333115 - HTTP/1.1 GET www.berkeley.edu /4 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 12861 132.239.x.x:55097 -> 128.32.203.137:80
> 06/26/2016-20:01:52.615975 - HTTP/1.1 GET www.berkeley.edu /5 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 - 0 132.239.x.x:64705 -> 128.32.203.137:80
> 06/26/2016-20:01:53.865529 - HTTP/1.1 GET www.berkeley.edu /6 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 25734 132.239.x.x:52871 -> 128.32.203.137:80
> 06/26/2016-20:01:55.096402 - HTTP/1.1 GET www.berkeley.edu /7 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 25734 132.239.x.x:61459 -> 128.32.203.137:80
> 06/26/2016-20:02:01.504810 - HTTP/1.1 GET www.berkeley.edu /12 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 - 0 132.239.x.x:61103 -> 128.32.203.137:80

What's odd is that shutting down the engine after the test causes some
of the "missing" log entries to be spooled to disk, out of sequence.

> 06/26/2016-20:15:57.862449 - HTTP/1.1 GET www.berkeley.edu /98 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 14308 132.239.x.x:60268 -> 128.32.203.137:80
> 06/26/2016-20:15:58.463770 - HTTP/1.1 GET www.berkeley.edu /99 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 14308 132.239.x.x:61915 -> 128.32.203.137:80
> 06/26/2016-20:15:58.687963 - HTTP/1.1 GET www.berkeley.edu /100 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 25734 132.239.x.x:65334 -> 128.32.203.137:80
> 06/26/2016-20:16:20.060538 - HTTP/1.1 GET www.berkeley.edu /60 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 15756 132.239.x.x:55242 -> 128.32.203.137:80
> 06/26/2016-20:16:20.061081 - HTTP/1.1 GET www.berkeley.edu /84 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 15756 132.239.x.x:64006 -> 128.32.203.137:80
> 06/26/2016-20:16:20.063309 - HTTP/1.1 GET www.berkeley.edu /18 curl/7.16.1 (i386-portbld-freebsd6.1) libcurl/7.16.3 OpenSSL/0.9.8e zlib/1.2.2 404 17204 132.239.x.x:60560 -> 128.32.203.137:80

As a sanity check I ran suricata with a bpf filter to only monitor the
single host making the http requests.  The behavior still happens, but
the number of requests dropped isn't as high.

I also turned on the stream and http events rules and logged the
following, also only monitoring traffic for a single host:

> 06/26/2016-20:43:28.626337  [**] [1:2013028:4] ET POLICY curl User-Agent Outbound [**] [Classification: Attempted Information Leak] [Priority: 2] {TCP} 132.239.x.x:61550 -> 128.32.203.137:80
> 06/26/2016-20:43:28.903012  [**] [1:2210016:1] SURICATA STREAM CLOSEWAIT FIN out of window [**] [Classification: (null)] [Priority: 3] {TCP} 128.32.203.137:80 -> 132.239.x.x:51697
> 06/26/2016-20:43:29.430583  [**] [1:2210000:1] SURICATA STREAM 3way handshake with ack in wrong dir [**] [Classification: (null)] [Priority: 3] {TCP} 128.32.203.137:80 -> 132.239.x.x:58191
> 06/26/2016-20:43:29.635260  [**] [1:2210010:1] SURICATA STREAM 3way handshake wrong seq wrong ack [**] [Classification: (null)] [Priority: 3] {TCP} 132.239.x.x:58191 -> 128.32.203.137:80
> 06/26/2016-20:43:28.640655  [**] [1:2210032:1] SURICATA STREAM FIN1 FIN with wrong seq [**] [Classification: (null)] [Priority: 3] {TCP} 132.239.x.x:61550 -> 128.32.203.137:80

So something odd is happening.  We've recently changed our monitoring
topology so it's possible that is introducing some problems, so I'll
investigate that this week.

I'm cc:'ing Victor and Peter to see if they have any idea what is
happening.  It would also be a big help if other 3.1 users attempted to
replicate this issue.  I've also verified that this behavior happens
with both the hyperscan and ac algos.

-Coop

On 6/22/2016 6:56 PM, Caesar Samsi wrote:
> I still have my 3.0.1 build which works well.
> 
> One sanity check I do when installing a new version (3.1 in this instance) is to do:
> 1. curl -A “BlackSun” google.com <http://google.com/>
> 2. host google.com <http://google.com/>
> 
> First one is to supported to trigger possible trojan horse alert.
> Second one is to capture dns traffic.
> 
> Neither works in 3.1?
> 
> I’m sure I’m just missing configuration.
> 
> Help?
> 
> 
> 
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC: http://oisfevents.net
> 

-- 
Cooper Nelson
Network Security Analyst
UCSD ITS Security Team
cnelson at ucsd.edu x41042

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20160626/906896b6/attachment-0002.sig>