[Oisf-users] Http/libhtp issue Suricata 2.0 on CentOS 6
Victor Julien
lists at inliniac.net
Thu May 1 15:17:54 UTC 2014
On 05/01/2014 05:12 PM, (OISF) Martijn Schoemaker wrote:
>
>
>>
>> On Thu, May 1, 2014 at 3:22 PM, (OISF) Martijn Schoemaker
>> <oisf at ficture.nl <mailto:oisf at ficture.nl>> wrote:
>>
>>
>> Some additional info:
>>
>> Working 1.4.7 release:
>> --------------------------------
>> # suricata-1.4.7/src/suricata --build-info
>> This is Suricata version 1.4.7 RELEASE
>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>> HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>> HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW
>> 64-bits, Little-endian architecture
>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
>> compiled with libhtp 0.2.14, linked against 0.2.14
>> Suricata Configuration:
>> AF_PACKET support: yes
>> PF_RING support: no
>> NFQueue support: no
>> IPFW support: no
>> DAG enabled: no
>> Napatech enabled: no
>> Unix socket enabled: no
>>
>> libnss support: no
>> libnspr support: no
>> libjansson support: no
>> Prelude support: no
>> PCRE jit: no
>> libluajit: no
>> libgeoip: no
>> Non-bundled htp: no
>> Old barnyard2 support: no
>> CUDA enabled: no
>>
>> Suricatasc install: yes
>>
>> Unit tests enabled: no
>> Debug output enabled: no
>> Debug validation enabled: no
>> Profiling enabled: no
>> Profiling locks enabled: no
>>
>> Generic build parameters:
>> Installation prefix (--prefix): /usr
>> Configuration directory (--sysconfdir): /etc/suricata/
>> Log directory (--localstatedir) : /var/log/suricata/
>>
>> Host: x86_64-unknown-linux-gnu
>> GCC binary: gcc
>> GCC Protect enabled: no
>> GCC march native enabled: yes
>> GCC Profile enabled: no
>>
>> Git release (not working):
>> -------------------------------------
>> # suricata-git/oisf/src/suricata --build-info
>> This is Suricata version 2.0dev (rev 6fbb955)
>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>> HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>> SIMD support: SSE_3
>> Atomic intrisics: 1 2 4 8 16 byte(s)
>> 64-bits, Little-endian architecture
>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>> L1 cache line size (CLS)=64
>> compiled with LibHTP v0.5.11, linked against LibHTP v0.5.11
>> Suricata Configuration:
>> AF_PACKET support: yes
>> PF_RING support: no
>> NFQueue support: no
>> IPFW support: no
>> DAG enabled: no
>> Napatech enabled: no
>> Unix socket enabled: no
>> Detection enabled: yes
>>
>> libnss support: no
>> libnspr support: no
>> libjansson support: no
>> Prelude support: no
>> PCRE jit: no
>> libluajit: no
>> libgeoip: no
>> Non-bundled htp: no
>> Old barnyard2 support: no
>> CUDA enabled: no
>>
>> Suricatasc install: yes
>>
>> Unit tests enabled: no
>> Debug output enabled: no
>> Debug validation enabled: no
>> Profiling enabled: no
>> Profiling locks enabled: no
>> Coccinelle / spatch: no
>>
>> Generic build parameters:
>> Installation prefix (--prefix): /usr
>> Configuration directory (--sysconfdir): /etc/suricata/
>> Log directory (--localstatedir) : /var/log/suricata/
>>
>> Host: x86_64-unknown-linux-gnu
>> GCC binary: gcc
>> GCC Protect enabled: no
>> GCC march native enabled: yes
>> GCC Profile enabled: no
>>
>> 2.0 release (also not working):
>> -------------------------------------------
>> # suricata-2.0/src/suricata --build-info
>> This is Suricata version 2.0 RELEASE
>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>> HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>> HAVE_NSS HAVE_LIBJANSSON
>> SIMD support: SSE_3
>> Atomic intrisics: 1 2 4 8 16 byte(s)
>> 64-bits, Little-endian architecture
>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>> L1 cache line size (CLS)=64
>> compiled with LibHTP v0.5.10, linked against LibHTP v0.5.10
>> Suricata Configuration:
>> AF_PACKET support: yes
>> PF_RING support: no
>> NFQueue support: no
>> IPFW support: no
>> DAG enabled: no
>> Napatech enabled: no
>> Unix socket enabled: yes
>> Detection enabled: yes
>>
>> libnss support: yes
>> libnspr support: yes
>> libjansson support: yes
>> Prelude support: no
>> PCRE jit: no
>> libluajit: no
>> libgeoip: no
>> Non-bundled htp: no
>> Old barnyard2 support: no
>> CUDA enabled: no
>>
>> Suricatasc install: yes
>>
>> Unit tests enabled: no
>> Debug output enabled: no
>> Debug validation enabled: no
>> Profiling enabled: no
>> Profiling locks enabled: no
>> Coccinelle / spatch: yes
>>
>> Generic build parameters:
>> Installation prefix (--prefix): /usr
>> Configuration directory (--sysconfdir): /etc/suricata/
>> Log directory (--localstatedir) : /var/log/suricata/
>>
>> Host: x86_64-unknown-linux-gnu
>> GCC binary: gcc
>> GCC Protect enabled: no
>> GCC march native enabled: yes
>> GCC Profile enabled: no
>>
>>
>> On 05/01/2014 03:16 PM, (OISF) Martijn Schoemaker wrote:
>>
>> Hi,
>>
>> I have been running suricata 1.4.7 for quite some time and
>> it's working like a charm. When I saw that suricata 2.0 supports the
>> eve-json log format for integration with logstash I wanted to upgrade
>> to 2.0.
>>
>> I downloaded the stable 2.0 release, built it and all seemed
>> to run fine. However, I notices the http.log was no longer modified.
>> Further investigation showed that all http event matching, http
>> logging (http-log and eve http log) was no longer working. I started
>> out with the exact same config as the working 1.4.7 release, then
>> modified the 2.0 config accordingly but it just won't work.
>>
>> I also noticed it now includes libhtp 0.5.10 instead of 0.2 so
>> I tried to build against 0.2 but that's not supported. I also built
>> the git current release (libhtp 0.5.11), but still no go. Strange
>> thing is that http events are also no longer matched. I run on a
>> machine which is connected to a monitor port so it cannot be checksum
>> offloading (I also manually disabled it on the interface and disabled
>> checksum checking in the suricata config, but all to no avail).
>>
>> Whenever I revert to the 1.4.7 release everything works again.
>>
>> So I have a big suspicion that either I'm doing something
>> terribly wrong, or the libhtp 0.5 release is not working correctly
>> anymore.
>>
>> Is there anyone who observed the same issue ?
>>
>> Regards,
>> Martijn Schoemaker
>>
>> _______________________________________________
>> Suricata IDS Users mailing list:
>> oisf-users at openinfosecfoundation.org
>> <mailto:oisf-users at openinfosecfoundation.org>
>> Site: http://suricata-ids.org | Support:
>> http://suricata-ids.org/support/
>> List:
>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> OISF: http://www.openinfosecfoundation.org/
>>
>> _______________________________________________
>> Suricata IDS Users mailing list:
>> oisf-users at openinfosecfoundation.org
>> <mailto:oisf-users at openinfosecfoundation.org>
>> Site: http://suricata-ids.org | Support:
>> http://suricata-ids.org/support/
>> List:
>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> OISF: http://www.openinfosecfoundation.org/
>>
>>
>>
>> Hi,
>>
>> I think there is some sort of a (miss)configuration issue. For the
>> JSON output to work you need libjansson4 and libjansson-dev present on
>> the system.
>> When you do (suricata --build-info) you should see -> " libjansson
>> support: yes"
>>
>> What I would suggest -
>>
>> 1)
>> Install 2.0 an a "new/clean" machine (virt if you want), and verify
>> that everything is working. If this is the case - then there is some
>> mixup on your current installation.
>>
>> 2)
>> Suricata.yaml and yaml in general is very peculiar about spaces/tabs
>> being at the right place and such. Please make sure some miss editing
>> is not the issue. (try loading the default provided suricata.yaml from
>> source)
>>
>> 3)
>> Can you copy paste your suricata.log on pastebin and share it?
>>
>> 4)
>> Can you provide the output of
>> ldd /path/to/suricata_executable
>> (example - ldd /usr/local/bin/suricata)
>>
>>
>> Thanks
>>
>>
>> --
>> Regards,
>> Peter Manev
> Hi Peter,
>
> Thanks for the quick reply, but JSON output works fine, but everything
> regarding HTTP does not work anymore (IDS rules, http-log, etc).
>
> Also if use exactly the same config in the 1.4.7 and 2.0 I don't see any
> HTTP related matching (IDS rules, http-log). Even 2.0 with vanilla
> config from the build does not process any HTTP packets.
>
> Unfortunately I have no quick possibility to install on a clean machine
> since I am dependent on the monitor port/switch configuration to give me
> the traffic I need to match against. So I will have to figure out a way
> to do this.
>
> As for the suritcata log, it got overwritten unfortunately and I already
> reverted back to 1.4.7 due to the monitoring I need to do.
>
> As for the ldd, see below.
>
> Thanks again,
> Martijn
>
> Working 1.4.7:
> # ldd suricata-1.4.7/src/.libs/suricata
> linux-vdso.so.1 => (0x00007fffc9d38000)
> libhtp-0.2.so.1 => /usr/lib/libhtp-0.2.so.1 (0x00007ff7806bf000)
> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007ff7804a1000)
> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007ff780260000)
> libnet.so.1 => /lib64/libnet.so.1 (0x00007ff780047000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff77fe2a000)
> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007ff77fc0a000)
> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007ff77f9de000)
> libc.so.6 => /lib64/libc.so.6 (0x00007ff77f64a000)
> libz.so.1 => /lib64/libz.so.1 (0x00007ff77f433000)
> /lib64/ld-linux-x86-64.so.2 (0x00007ff7808df000)
>
> Not working 2.0:
> ldd suricata-2.0/src/.libs/suricata
> linux-vdso.so.1 => (0x00007fff7d1ff000)
> libhtp-0.5.10.so.1 => /usr/lib/libhtp-0.5.10.so.1 (0x00007fa905b92000)
> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007fa905974000)
> libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007fa90576e000)
> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007fa90552e000)
> libnet.so.1 => /lib64/libnet.so.1 (0x00007fa905315000)
> libjansson.so.4 => /usr/lib64/libjansson.so.4 (0x00007fa905109000)
> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007fa904eea000)
> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fa904cbe000)
> libssl3.so => /usr/lib64/libssl3.so (0x00007fa904a7f000)
> libsmime3.so => /usr/lib64/libsmime3.so (0x00007fa904853000)
> libnss3.so => /usr/lib64/libnss3.so (0x00007fa904515000)
> libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007fa9042e8000)
> libplds4.so => /lib64/libplds4.so (0x00007fa9040e4000)
> libplc4.so => /lib64/libplc4.so (0x00007fa903edf000)
> libnspr4.so => /lib64/libnspr4.so (0x00007fa903ca1000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa903a84000)
> libdl.so.2 => /lib64/libdl.so.2 (0x00007fa903880000)
> libc.so.6 => /lib64/libc.so.6 (0x00007fa9034eb000)
> libz.so.1 => /lib64/libz.so.1 (0x00007fa9032d5000)
> /lib64/ld-linux-x86-64.so.2 (0x00007fa905dbb000)
> librt.so.1 => /lib64/librt.so.1 (0x00007fa9030cc000)
>
> Not working GIT:
> # ldd suricata-git/oisf/src/.libs/suricata
> linux-vdso.so.1 => (0x00007fff3fef7000)
> libhtp-0.5.11.so.1 => /usr/lib/libhtp-0.5.11.so.1 (0x00007fdf65fa1000)
> libz.so.1 => /lib64/libz.so.1 (0x00007fdf65d8b000)
> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007fdf65b6c000)
> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007fdf6592c000)
> libnet.so.1 => /lib64/libnet.so.1 (0x00007fdf65713000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdf654f5000)
> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007fdf652d6000)
> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fdf650aa000)
> libc.so.6 => /lib64/libc.so.6 (0x00007fdf64d15000)
> /lib64/ld-linux-x86-64.so.2 (0x00007fdf661cb000)
>
Do you still have some stats.log records from the non-working setup?
One thing that may affect this is the updated vlan support. You may want
to try:
# This option controls the use of vlan ids in the flow (and defrag)
# hashing. Normally this should be enabled, but in some (broken)
# setups where both sides of a flow are not tagged with the same vlan
# tag, we can ignore the vlan id's in the flow hashing.
vlan:
use-for-tracking: false
Some weird equipment tags only one side of the conversation.
--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------
More information about the Oisf-users
mailing list