[Oisf-users] Http/libhtp issue Suricata 2.0 on CentOS 6
(OISF) Martijn Schoemaker
oisf at ficture.nl
Thu May 1 15:34:38 UTC 2014
On 05/01/2014 05:32 PM, Martijn Schoemaker (Ficture IT) wrote:
On 05/01/2014 05:17 PM, Victor Julien wrote:
> On 05/01/2014 05:12 PM, (OISF) Martijn Schoemaker wrote:
>>
>>> On Thu, May 1, 2014 at 3:22 PM, (OISF) Martijn Schoemaker
>>> <oisf at ficture.nl <mailto:oisf at ficture.nl>> wrote:
>>>
>>>
>>> Some additional info:
>>>
>>> Working 1.4.7 release:
>>> --------------------------------
>>> # suricata-1.4.7/src/suricata --build-info
>>> This is Suricata version 1.4.7 RELEASE
>>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>>> HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>>> HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW
>>> 64-bits, Little-endian architecture
>>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
>>> compiled with libhtp 0.2.14, linked against 0.2.14
>>> Suricata Configuration:
>>> AF_PACKET support: yes
>>> PF_RING support: no
>>> NFQueue support: no
>>> IPFW support: no
>>> DAG enabled: no
>>> Napatech enabled: no
>>> Unix socket enabled: no
>>>
>>> libnss support: no
>>> libnspr support: no
>>> libjansson support: no
>>> Prelude support: no
>>> PCRE jit: no
>>> libluajit: no
>>> libgeoip: no
>>> Non-bundled htp: no
>>> Old barnyard2 support: no
>>> CUDA enabled: no
>>>
>>> Suricatasc install: yes
>>>
>>> Unit tests enabled: no
>>> Debug output enabled: no
>>> Debug validation enabled: no
>>> Profiling enabled: no
>>> Profiling locks enabled: no
>>>
>>> Generic build parameters:
>>> Installation prefix (--prefix): /usr
>>> Configuration directory (--sysconfdir): /etc/suricata/
>>> Log directory (--localstatedir) : /var/log/suricata/
>>>
>>> Host: x86_64-unknown-linux-gnu
>>> GCC binary: gcc
>>> GCC Protect enabled: no
>>> GCC march native enabled: yes
>>> GCC Profile enabled: no
>>>
>>> Git release (not working):
>>> -------------------------------------
>>> # suricata-git/oisf/src/suricata --build-info
>>> This is Suricata version 2.0dev (rev 6fbb955)
>>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>>> HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>>> SIMD support: SSE_3
>>> Atomic intrisics: 1 2 4 8 16 byte(s)
>>> 64-bits, Little-endian architecture
>>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>>> L1 cache line size (CLS)=64
>>> compiled with LibHTP v0.5.11, linked against LibHTP v0.5.11
>>> Suricata Configuration:
>>> AF_PACKET support: yes
>>> PF_RING support: no
>>> NFQueue support: no
>>> IPFW support: no
>>> DAG enabled: no
>>> Napatech enabled: no
>>> Unix socket enabled: no
>>> Detection enabled: yes
>>>
>>> libnss support: no
>>> libnspr support: no
>>> libjansson support: no
>>> Prelude support: no
>>> PCRE jit: no
>>> libluajit: no
>>> libgeoip: no
>>> Non-bundled htp: no
>>> Old barnyard2 support: no
>>> CUDA enabled: no
>>>
>>> Suricatasc install: yes
>>>
>>> Unit tests enabled: no
>>> Debug output enabled: no
>>> Debug validation enabled: no
>>> Profiling enabled: no
>>> Profiling locks enabled: no
>>> Coccinelle / spatch: no
>>>
>>> Generic build parameters:
>>> Installation prefix (--prefix): /usr
>>> Configuration directory (--sysconfdir): /etc/suricata/
>>> Log directory (--localstatedir) : /var/log/suricata/
>>>
>>> Host: x86_64-unknown-linux-gnu
>>> GCC binary: gcc
>>> GCC Protect enabled: no
>>> GCC march native enabled: yes
>>> GCC Profile enabled: no
>>>
>>> 2.0 release (also not working):
>>> -------------------------------------------
>>> # suricata-2.0/src/suricata --build-info
>>> This is Suricata version 2.0 RELEASE
>>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>>> HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>>> HAVE_NSS HAVE_LIBJANSSON
>>> SIMD support: SSE_3
>>> Atomic intrisics: 1 2 4 8 16 byte(s)
>>> 64-bits, Little-endian architecture
>>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>>> L1 cache line size (CLS)=64
>>> compiled with LibHTP v0.5.10, linked against LibHTP v0.5.10
>>> Suricata Configuration:
>>> AF_PACKET support: yes
>>> PF_RING support: no
>>> NFQueue support: no
>>> IPFW support: no
>>> DAG enabled: no
>>> Napatech enabled: no
>>> Unix socket enabled: yes
>>> Detection enabled: yes
>>>
>>> libnss support: yes
>>> libnspr support: yes
>>> libjansson support: yes
>>> Prelude support: no
>>> PCRE jit: no
>>> libluajit: no
>>> libgeoip: no
>>> Non-bundled htp: no
>>> Old barnyard2 support: no
>>> CUDA enabled: no
>>>
>>> Suricatasc install: yes
>>>
>>> Unit tests enabled: no
>>> Debug output enabled: no
>>> Debug validation enabled: no
>>> Profiling enabled: no
>>> Profiling locks enabled: no
>>> Coccinelle / spatch: yes
>>>
>>> Generic build parameters:
>>> Installation prefix (--prefix): /usr
>>> Configuration directory (--sysconfdir): /etc/suricata/
>>> Log directory (--localstatedir) : /var/log/suricata/
>>>
>>> Host: x86_64-unknown-linux-gnu
>>> GCC binary: gcc
>>> GCC Protect enabled: no
>>> GCC march native enabled: yes
>>> GCC Profile enabled: no
>>>
>>>
>>> On 05/01/2014 03:16 PM, (OISF) Martijn Schoemaker wrote:
>>>
>>> Hi,
>>>
>>> I have been running suricata 1.4.7 for quite some time and
>>> it's working like a charm. When I saw that suricata 2.0 supports the
>>> eve-json log format for integration with logstash I wanted to upgrade
>>> to 2.0.
>>>
>>> I downloaded the stable 2.0 release, built it and all seemed
>>> to run fine. However, I notices the http.log was no longer modified.
>>> Further investigation showed that all http event matching, http
>>> logging (http-log and eve http log) was no longer working. I started
>>> out with the exact same config as the working 1.4.7 release, then
>>> modified the 2.0 config accordingly but it just won't work.
>>>
>>> I also noticed it now includes libhtp 0.5.10 instead of 0.2 so
>>> I tried to build against 0.2 but that's not supported. I also built
>>> the git current release (libhtp 0.5.11), but still no go. Strange
>>> thing is that http events are also no longer matched. I run on a
>>> machine which is connected to a monitor port so it cannot be checksum
>>> offloading (I also manually disabled it on the interface and disabled
>>> checksum checking in the suricata config, but all to no avail).
>>>
>>> Whenever I revert to the 1.4.7 release everything works again.
>>>
>>> So I have a big suspicion that either I'm doing something
>>> terribly wrong, or the libhtp 0.5 release is not working correctly
>>> anymore.
>>>
>>> Is there anyone who observed the same issue ?
>>>
>>> Regards,
>>> Martijn Schoemaker
>>>
>>> _______________________________________________
>>> Suricata IDS Users mailing list:
>>> oisf-users at openinfosecfoundation.org
>>> <mailto:oisf-users at openinfosecfoundation.org>
>>> Site: http://suricata-ids.org | Support:
>>> http://suricata-ids.org/support/
>>> List:
>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> OISF: http://www.openinfosecfoundation.org/
>>>
>>> _______________________________________________
>>> Suricata IDS Users mailing list:
>>> oisf-users at openinfosecfoundation.org
>>> <mailto:oisf-users at openinfosecfoundation.org>
>>> Site: http://suricata-ids.org | Support:
>>> http://suricata-ids.org/support/
>>> List:
>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> OISF: http://www.openinfosecfoundation.org/
>>>
>>>
>>>
>>> Hi,
>>>
>>> I think there is some sort of a (miss)configuration issue. For the
>>> JSON output to work you need libjansson4 and libjansson-dev present on
>>> the system.
>>> When you do (suricata --build-info) you should see -> " libjansson
>>> support: yes"
>>>
>>> What I would suggest -
>>>
>>> 1)
>>> Install 2.0 an a "new/clean" machine (virt if you want), and verify
>>> that everything is working. If this is the case - then there is some
>>> mixup on your current installation.
>>>
>>> 2)
>>> Suricata.yaml and yaml in general is very peculiar about spaces/tabs
>>> being at the right place and such. Please make sure some miss editing
>>> is not the issue. (try loading the default provided suricata.yaml from
>>> source)
>>>
>>> 3)
>>> Can you copy paste your suricata.log on pastebin and share it?
>>>
>>> 4)
>>> Can you provide the output of
>>> ldd /path/to/suricata_executable
>>> (example - ldd /usr/local/bin/suricata)
>>>
>>>
>>> Thanks
>>>
>>>
>>> --
>>> Regards,
>>> Peter Manev
>> Hi Peter,
>>
>> Thanks for the quick reply, but JSON output works fine, but everything
>> regarding HTTP does not work anymore (IDS rules, http-log, etc).
>>
>> Also if use exactly the same config in the 1.4.7 and 2.0 I don't see any
>> HTTP related matching (IDS rules, http-log). Even 2.0 with vanilla
>> config from the build does not process any HTTP packets.
>>
>> Unfortunately I have no quick possibility to install on a clean machine
>> since I am dependent on the monitor port/switch configuration to give me
>> the traffic I need to match against. So I will have to figure out a way
>> to do this.
>>
>> As for the suritcata log, it got overwritten unfortunately and I already
>> reverted back to 1.4.7 due to the monitoring I need to do.
>>
>> As for the ldd, see below.
>>
>> Thanks again,
>> Martijn
>>
>> Working 1.4.7:
>> # ldd suricata-1.4.7/src/.libs/suricata
>> linux-vdso.so.1 => (0x00007fffc9d38000)
>> libhtp-0.2.so.1 => /usr/lib/libhtp-0.2.so.1 (0x00007ff7806bf000)
>> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007ff7804a1000)
>> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007ff780260000)
>> libnet.so.1 => /lib64/libnet.so.1 (0x00007ff780047000)
>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff77fe2a000)
>> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007ff77fc0a000)
>> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007ff77f9de000)
>> libc.so.6 => /lib64/libc.so.6 (0x00007ff77f64a000)
>> libz.so.1 => /lib64/libz.so.1 (0x00007ff77f433000)
>> /lib64/ld-linux-x86-64.so.2 (0x00007ff7808df000)
>>
>> Not working 2.0:
>> ldd suricata-2.0/src/.libs/suricata
>> linux-vdso.so.1 => (0x00007fff7d1ff000)
>> libhtp-0.5.10.so.1 => /usr/lib/libhtp-0.5.10.so.1 (0x00007fa905b92000)
>> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007fa905974000)
>> libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007fa90576e000)
>> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007fa90552e000)
>> libnet.so.1 => /lib64/libnet.so.1 (0x00007fa905315000)
>> libjansson.so.4 => /usr/lib64/libjansson.so.4 (0x00007fa905109000)
>> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007fa904eea000)
>> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fa904cbe000)
>> libssl3.so => /usr/lib64/libssl3.so (0x00007fa904a7f000)
>> libsmime3.so => /usr/lib64/libsmime3.so (0x00007fa904853000)
>> libnss3.so => /usr/lib64/libnss3.so (0x00007fa904515000)
>> libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007fa9042e8000)
>> libplds4.so => /lib64/libplds4.so (0x00007fa9040e4000)
>> libplc4.so => /lib64/libplc4.so (0x00007fa903edf000)
>> libnspr4.so => /lib64/libnspr4.so (0x00007fa903ca1000)
>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa903a84000)
>> libdl.so.2 => /lib64/libdl.so.2 (0x00007fa903880000)
>> libc.so.6 => /lib64/libc.so.6 (0x00007fa9034eb000)
>> libz.so.1 => /lib64/libz.so.1 (0x00007fa9032d5000)
>> /lib64/ld-linux-x86-64.so.2 (0x00007fa905dbb000)
>> librt.so.1 => /lib64/librt.so.1 (0x00007fa9030cc000)
>>
>> Not working GIT:
>> # ldd suricata-git/oisf/src/.libs/suricata
>> linux-vdso.so.1 => (0x00007fff3fef7000)
>> libhtp-0.5.11.so.1 => /usr/lib/libhtp-0.5.11.so.1 (0x00007fdf65fa1000)
>> libz.so.1 => /lib64/libz.so.1 (0x00007fdf65d8b000)
>> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007fdf65b6c000)
>> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007fdf6592c000)
>> libnet.so.1 => /lib64/libnet.so.1 (0x00007fdf65713000)
>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdf654f5000)
>> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007fdf652d6000)
>> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fdf650aa000)
>> libc.so.6 => /lib64/libc.so.6 (0x00007fdf64d15000)
>> /lib64/ld-linux-x86-64.so.2 (0x00007fdf661cb000)
>>
> Do you still have some stats.log records from the non-working setup?
>
> One thing that may affect this is the updated vlan support. You may want
> to try:
>
> # This option controls the use of vlan ids in the flow (and defrag)
> # hashing. Normally this should be enabled, but in some (broken)
> # setups where both sides of a flow are not tagged with the same vlan
> # tag, we can ignore the vlan id's in the flow hashing.
> vlan:
> use-for-tracking: false
>
> Some weird equipment tags only one side of the conversation.
>
Victor, if I could remotely hug you I would ! :)
Bingo, vlan tagging probably gave me headaches. Could be the case since I indeed have a many-to-1 port monitor of ports in different VLAN's on the switch to enable me to see VPN traffic too before it's encrypted alongside the browser traffic from users. Disables it and all works fine!
Thanks a bundle! And sorry all for the hassle!
Regards,
Martijn
More information about the Oisf-users
mailing list