[Oisf-users] Http/libhtp issue Suricata 2.0 on CentOS 6
(OISF) Martijn Schoemaker
oisf at ficture.nl
Thu May 1 15:12:32 UTC 2014
>
> On Thu, May 1, 2014 at 3:22 PM, (OISF) Martijn Schoemaker <oisf at ficture.nl <mailto:oisf at ficture.nl>> wrote:
>
>
> Some additional info:
>
> Working 1.4.7 release:
> --------------------------------
> # suricata-1.4.7/src/suricata --build-info
> This is Suricata version 1.4.7 RELEASE
> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW
> 64-bits, Little-endian architecture
> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
> compiled with libhtp 0.2.14, linked against 0.2.14
> Suricata Configuration:
> AF_PACKET support: yes
> PF_RING support: no
> NFQueue support: no
> IPFW support: no
> DAG enabled: no
> Napatech enabled: no
> Unix socket enabled: no
>
> libnss support: no
> libnspr support: no
> libjansson support: no
> Prelude support: no
> PCRE jit: no
> libluajit: no
> libgeoip: no
> Non-bundled htp: no
> Old barnyard2 support: no
> CUDA enabled: no
>
> Suricatasc install: yes
>
> Unit tests enabled: no
> Debug output enabled: no
> Debug validation enabled: no
> Profiling enabled: no
> Profiling locks enabled: no
>
> Generic build parameters:
> Installation prefix (--prefix): /usr
> Configuration directory (--sysconfdir): /etc/suricata/
> Log directory (--localstatedir) : /var/log/suricata/
>
> Host: x86_64-unknown-linux-gnu
> GCC binary: gcc
> GCC Protect enabled: no
> GCC march native enabled: yes
> GCC Profile enabled: no
>
> Git release (not working):
> -------------------------------------
> # suricata-git/oisf/src/suricata --build-info
> This is Suricata version 2.0dev (rev 6fbb955)
> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
> SIMD support: SSE_3
> Atomic intrisics: 1 2 4 8 16 byte(s)
> 64-bits, Little-endian architecture
> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
> L1 cache line size (CLS)=64
> compiled with LibHTP v0.5.11, linked against LibHTP v0.5.11
> Suricata Configuration:
> AF_PACKET support: yes
> PF_RING support: no
> NFQueue support: no
> IPFW support: no
> DAG enabled: no
> Napatech enabled: no
> Unix socket enabled: no
> Detection enabled: yes
>
> libnss support: no
> libnspr support: no
> libjansson support: no
> Prelude support: no
> PCRE jit: no
> libluajit: no
> libgeoip: no
> Non-bundled htp: no
> Old barnyard2 support: no
> CUDA enabled: no
>
> Suricatasc install: yes
>
> Unit tests enabled: no
> Debug output enabled: no
> Debug validation enabled: no
> Profiling enabled: no
> Profiling locks enabled: no
> Coccinelle / spatch: no
>
> Generic build parameters:
> Installation prefix (--prefix): /usr
> Configuration directory (--sysconfdir): /etc/suricata/
> Log directory (--localstatedir) : /var/log/suricata/
>
> Host: x86_64-unknown-linux-gnu
> GCC binary: gcc
> GCC Protect enabled: no
> GCC march native enabled: yes
> GCC Profile enabled: no
>
> 2.0 release (also not working):
> -------------------------------------------
> # suricata-2.0/src/suricata --build-info
> This is Suricata version 2.0 RELEASE
> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_NSS HAVE_LIBJANSSON
> SIMD support: SSE_3
> Atomic intrisics: 1 2 4 8 16 byte(s)
> 64-bits, Little-endian architecture
> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
> L1 cache line size (CLS)=64
> compiled with LibHTP v0.5.10, linked against LibHTP v0.5.10
> Suricata Configuration:
> AF_PACKET support: yes
> PF_RING support: no
> NFQueue support: no
> IPFW support: no
> DAG enabled: no
> Napatech enabled: no
> Unix socket enabled: yes
> Detection enabled: yes
>
> libnss support: yes
> libnspr support: yes
> libjansson support: yes
> Prelude support: no
> PCRE jit: no
> libluajit: no
> libgeoip: no
> Non-bundled htp: no
> Old barnyard2 support: no
> CUDA enabled: no
>
> Suricatasc install: yes
>
> Unit tests enabled: no
> Debug output enabled: no
> Debug validation enabled: no
> Profiling enabled: no
> Profiling locks enabled: no
> Coccinelle / spatch: yes
>
> Generic build parameters:
> Installation prefix (--prefix): /usr
> Configuration directory (--sysconfdir): /etc/suricata/
> Log directory (--localstatedir) : /var/log/suricata/
>
> Host: x86_64-unknown-linux-gnu
> GCC binary: gcc
> GCC Protect enabled: no
> GCC march native enabled: yes
> GCC Profile enabled: no
>
>
> On 05/01/2014 03:16 PM, (OISF) Martijn Schoemaker wrote:
>
> Hi,
>
> I have been running suricata 1.4.7 for quite some time and it's working like a charm. When I saw that suricata 2.0 supports the eve-json log format for integration with logstash I wanted to upgrade to 2.0.
>
> I downloaded the stable 2.0 release, built it and all seemed to run fine. However, I notices the http.log was no longer modified. Further investigation showed that all http event matching, http logging (http-log and eve http log) was no longer working. I started out with the exact same config as the working 1.4.7 release, then modified the 2.0 config accordingly but it just won't work.
>
> I also noticed it now includes libhtp 0.5.10 instead of 0.2 so I tried to build against 0.2 but that's not supported. I also built the git current release (libhtp 0.5.11), but still no go. Strange thing is that http events are also no longer matched. I run on a machine which is connected to a monitor port so it cannot be checksum offloading (I also manually disabled it on the interface and disabled checksum checking in the suricata config, but all to no avail).
>
> Whenever I revert to the 1.4.7 release everything works again.
>
> So I have a big suspicion that either I'm doing something terribly wrong, or the libhtp 0.5 release is not working correctly anymore.
>
> Is there anyone who observed the same issue ?
>
> Regards,
> Martijn Schoemaker
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org <mailto:oisf-users at openinfosecfoundation.org>
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org <mailto:oisf-users at openinfosecfoundation.org>
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/
>
>
>
> Hi,
>
> I think there is some sort of a (miss)configuration issue. For the JSON output to work you need libjansson4 and libjansson-dev present on the system.
> When you do (suricata --build-info) you should see -> " libjansson support: yes"
>
> What I would suggest -
>
> 1)
> Install 2.0 an a "new/clean" machine (virt if you want), and verify that everything is working. If this is the case - then there is some mixup on your current installation.
>
> 2)
> Suricata.yaml and yaml in general is very peculiar about spaces/tabs being at the right place and such. Please make sure some miss editing is not the issue. (try loading the default provided suricata.yaml from source)
>
> 3)
> Can you copy paste your suricata.log on pastebin and share it?
>
> 4)
> Can you provide the output of
> ldd /path/to/suricata_executable
> (example - ldd /usr/local/bin/suricata)
>
>
> Thanks
>
>
> --
> Regards,
> Peter Manev
Hi Peter,
Thanks for the quick reply, but JSON output works fine, but everything regarding HTTP does not work anymore (IDS rules, http-log, etc).
Also if use exactly the same config in the 1.4.7 and 2.0 I don't see any HTTP related matching (IDS rules, http-log). Even 2.0 with vanilla config from the build does not process any HTTP packets.
Unfortunately I have no quick possibility to install on a clean machine since I am dependent on the monitor port/switch configuration to give me the traffic I need to match against. So I will have to figure out a way to do this.
As for the suritcata log, it got overwritten unfortunately and I already reverted back to 1.4.7 due to the monitoring I need to do.
As for the ldd, see below.
Thanks again,
Martijn
Working 1.4.7:
# ldd suricata-1.4.7/src/.libs/suricata
linux-vdso.so.1 => (0x00007fffc9d38000)
libhtp-0.2.so.1 => /usr/lib/libhtp-0.2.so.1 (0x00007ff7806bf000)
libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007ff7804a1000)
libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007ff780260000)
libnet.so.1 => /lib64/libnet.so.1 (0x00007ff780047000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff77fe2a000)
libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007ff77fc0a000)
libpcre.so.0 => /lib64/libpcre.so.0 (0x00007ff77f9de000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff77f64a000)
libz.so.1 => /lib64/libz.so.1 (0x00007ff77f433000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff7808df000)
Not working 2.0:
ldd suricata-2.0/src/.libs/suricata
linux-vdso.so.1 => (0x00007fff7d1ff000)
libhtp-0.5.10.so.1 => /usr/lib/libhtp-0.5.10.so.1 (0x00007fa905b92000)
libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007fa905974000)
libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007fa90576e000)
libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007fa90552e000)
libnet.so.1 => /lib64/libnet.so.1 (0x00007fa905315000)
libjansson.so.4 => /usr/lib64/libjansson.so.4 (0x00007fa905109000)
libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007fa904eea000)
libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fa904cbe000)
libssl3.so => /usr/lib64/libssl3.so (0x00007fa904a7f000)
libsmime3.so => /usr/lib64/libsmime3.so (0x00007fa904853000)
libnss3.so => /usr/lib64/libnss3.so (0x00007fa904515000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007fa9042e8000)
libplds4.so => /lib64/libplds4.so (0x00007fa9040e4000)
libplc4.so => /lib64/libplc4.so (0x00007fa903edf000)
libnspr4.so => /lib64/libnspr4.so (0x00007fa903ca1000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa903a84000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fa903880000)
libc.so.6 => /lib64/libc.so.6 (0x00007fa9034eb000)
libz.so.1 => /lib64/libz.so.1 (0x00007fa9032d5000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa905dbb000)
librt.so.1 => /lib64/librt.so.1 (0x00007fa9030cc000)
Not working GIT:
# ldd suricata-git/oisf/src/.libs/suricata
linux-vdso.so.1 => (0x00007fff3fef7000)
libhtp-0.5.11.so.1 => /usr/lib/libhtp-0.5.11.so.1 (0x00007fdf65fa1000)
libz.so.1 => /lib64/libz.so.1 (0x00007fdf65d8b000)
libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007fdf65b6c000)
libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007fdf6592c000)
libnet.so.1 => /lib64/libnet.so.1 (0x00007fdf65713000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdf654f5000)
libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007fdf652d6000)
libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fdf650aa000)
libc.so.6 => /lib64/libc.so.6 (0x00007fdf64d15000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdf661cb000)
More information about the Oisf-users
mailing list