[Oisf-users] Http/libhtp issue Suricata 2.0 on CentOS 6
Peter Manev
petermanev at gmail.com
Thu May 1 15:41:55 UTC 2014
On Thu, May 1, 2014 at 5:34 PM, (OISF) Martijn Schoemaker
<oisf at ficture.nl> wrote:
>
> On 05/01/2014 05:32 PM, Martijn Schoemaker (Ficture IT) wrote:
>
>
> On 05/01/2014 05:17 PM, Victor Julien wrote:
>>
>> On 05/01/2014 05:12 PM, (OISF) Martijn Schoemaker wrote:
>>>
>>>
>>>> On Thu, May 1, 2014 at 3:22 PM, (OISF) Martijn Schoemaker
>>>> <oisf at ficture.nl <mailto:oisf at ficture.nl>> wrote:
>>>>
>>>>
>>>> Some additional info:
>>>>
>>>> Working 1.4.7 release:
>>>> --------------------------------
>>>> # suricata-1.4.7/src/suricata --build-info
>>>> This is Suricata version 1.4.7 RELEASE
>>>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>>>> HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>>>> HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW
>>>> 64-bits, Little-endian architecture
>>>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
>>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
>>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
>>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
>>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
>>>> compiled with libhtp 0.2.14, linked against 0.2.14
>>>> Suricata Configuration:
>>>> AF_PACKET support: yes
>>>> PF_RING support: no
>>>> NFQueue support: no
>>>> IPFW support: no
>>>> DAG enabled: no
>>>> Napatech enabled: no
>>>> Unix socket enabled: no
>>>>
>>>> libnss support: no
>>>> libnspr support: no
>>>> libjansson support: no
>>>> Prelude support: no
>>>> PCRE jit: no
>>>> libluajit: no
>>>> libgeoip: no
>>>> Non-bundled htp: no
>>>> Old barnyard2 support: no
>>>> CUDA enabled: no
>>>>
>>>> Suricatasc install: yes
>>>>
>>>> Unit tests enabled: no
>>>> Debug output enabled: no
>>>> Debug validation enabled: no
>>>> Profiling enabled: no
>>>> Profiling locks enabled: no
>>>>
>>>> Generic build parameters:
>>>> Installation prefix (--prefix): /usr
>>>> Configuration directory (--sysconfdir): /etc/suricata/
>>>> Log directory (--localstatedir) : /var/log/suricata/
>>>>
>>>> Host: x86_64-unknown-linux-gnu
>>>> GCC binary: gcc
>>>> GCC Protect enabled: no
>>>> GCC march native enabled: yes
>>>> GCC Profile enabled: no
>>>>
>>>> Git release (not working):
>>>> -------------------------------------
>>>> # suricata-git/oisf/src/suricata --build-info
>>>> This is Suricata version 2.0dev (rev 6fbb955)
>>>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>>>> HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>>>> SIMD support: SSE_3
>>>> Atomic intrisics: 1 2 4 8 16 byte(s)
>>>> 64-bits, Little-endian architecture
>>>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>>>> L1 cache line size (CLS)=64
>>>> compiled with LibHTP v0.5.11, linked against LibHTP v0.5.11
>>>> Suricata Configuration:
>>>> AF_PACKET support: yes
>>>> PF_RING support: no
>>>> NFQueue support: no
>>>> IPFW support: no
>>>> DAG enabled: no
>>>> Napatech enabled: no
>>>> Unix socket enabled: no
>>>> Detection enabled: yes
>>>>
>>>> libnss support: no
>>>> libnspr support: no
>>>> libjansson support: no
>>>> Prelude support: no
>>>> PCRE jit: no
>>>> libluajit: no
>>>> libgeoip: no
>>>> Non-bundled htp: no
>>>> Old barnyard2 support: no
>>>> CUDA enabled: no
>>>>
>>>> Suricatasc install: yes
>>>>
>>>> Unit tests enabled: no
>>>> Debug output enabled: no
>>>> Debug validation enabled: no
>>>> Profiling enabled: no
>>>> Profiling locks enabled: no
>>>> Coccinelle / spatch: no
>>>>
>>>> Generic build parameters:
>>>> Installation prefix (--prefix): /usr
>>>> Configuration directory (--sysconfdir): /etc/suricata/
>>>> Log directory (--localstatedir) : /var/log/suricata/
>>>>
>>>> Host: x86_64-unknown-linux-gnu
>>>> GCC binary: gcc
>>>> GCC Protect enabled: no
>>>> GCC march native enabled: yes
>>>> GCC Profile enabled: no
>>>>
>>>> 2.0 release (also not working):
>>>> -------------------------------------------
>>>> # suricata-2.0/src/suricata --build-info
>>>> This is Suricata version 2.0 RELEASE
>>>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>>>> HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>>>> HAVE_NSS HAVE_LIBJANSSON
>>>> SIMD support: SSE_3
>>>> Atomic intrisics: 1 2 4 8 16 byte(s)
>>>> 64-bits, Little-endian architecture
>>>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>>>> L1 cache line size (CLS)=64
>>>> compiled with LibHTP v0.5.10, linked against LibHTP v0.5.10
>>>> Suricata Configuration:
>>>> AF_PACKET support: yes
>>>> PF_RING support: no
>>>> NFQueue support: no
>>>> IPFW support: no
>>>> DAG enabled: no
>>>> Napatech enabled: no
>>>> Unix socket enabled: yes
>>>> Detection enabled: yes
>>>>
>>>> libnss support: yes
>>>> libnspr support: yes
>>>> libjansson support: yes
>>>> Prelude support: no
>>>> PCRE jit: no
>>>> libluajit: no
>>>> libgeoip: no
>>>> Non-bundled htp: no
>>>> Old barnyard2 support: no
>>>> CUDA enabled: no
>>>>
>>>> Suricatasc install: yes
>>>>
>>>> Unit tests enabled: no
>>>> Debug output enabled: no
>>>> Debug validation enabled: no
>>>> Profiling enabled: no
>>>> Profiling locks enabled: no
>>>> Coccinelle / spatch: yes
>>>>
>>>> Generic build parameters:
>>>> Installation prefix (--prefix): /usr
>>>> Configuration directory (--sysconfdir): /etc/suricata/
>>>> Log directory (--localstatedir) : /var/log/suricata/
>>>>
>>>> Host: x86_64-unknown-linux-gnu
>>>> GCC binary: gcc
>>>> GCC Protect enabled: no
>>>> GCC march native enabled: yes
>>>> GCC Profile enabled: no
>>>>
>>>>
>>>> On 05/01/2014 03:16 PM, (OISF) Martijn Schoemaker wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have been running suricata 1.4.7 for quite some time and
>>>> it's working like a charm. When I saw that suricata 2.0 supports the
>>>> eve-json log format for integration with logstash I wanted to upgrade
>>>> to 2.0.
>>>>
>>>> I downloaded the stable 2.0 release, built it and all seemed
>>>> to run fine. However, I notices the http.log was no longer modified.
>>>> Further investigation showed that all http event matching, http
>>>> logging (http-log and eve http log) was no longer working. I started
>>>> out with the exact same config as the working 1.4.7 release, then
>>>> modified the 2.0 config accordingly but it just won't work.
>>>>
>>>> I also noticed it now includes libhtp 0.5.10 instead of 0.2 so
>>>> I tried to build against 0.2 but that's not supported. I also built
>>>> the git current release (libhtp 0.5.11), but still no go. Strange
>>>> thing is that http events are also no longer matched. I run on a
>>>> machine which is connected to a monitor port so it cannot be checksum
>>>> offloading (I also manually disabled it on the interface and disabled
>>>> checksum checking in the suricata config, but all to no avail).
>>>>
>>>> Whenever I revert to the 1.4.7 release everything works again.
>>>>
>>>> So I have a big suspicion that either I'm doing something
>>>> terribly wrong, or the libhtp 0.5 release is not working correctly
>>>> anymore.
>>>>
>>>> Is there anyone who observed the same issue ?
>>>>
>>>> Regards,
>>>> Martijn Schoemaker
>>>>
>>>> _______________________________________________
>>>> Suricata IDS Users mailing list:
>>>> oisf-users at openinfosecfoundation.org
>>>> <mailto:oisf-users at openinfosecfoundation.org>
>>>> Site: http://suricata-ids.org | Support:
>>>> http://suricata-ids.org/support/
>>>> List:
>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>> OISF: http://www.openinfosecfoundation.org/
>>>>
>>>> _______________________________________________
>>>> Suricata IDS Users mailing list:
>>>> oisf-users at openinfosecfoundation.org
>>>> <mailto:oisf-users at openinfosecfoundation.org>
>>>> Site: http://suricata-ids.org | Support:
>>>> http://suricata-ids.org/support/
>>>> List:
>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>> OISF: http://www.openinfosecfoundation.org/
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>> I think there is some sort of a (miss)configuration issue. For the
>>>> JSON output to work you need libjansson4 and libjansson-dev present on
>>>> the system.
>>>> When you do (suricata --build-info) you should see -> " libjansson
>>>> support: yes"
>>>>
>>>> What I would suggest -
>>>>
>>>> 1)
>>>> Install 2.0 an a "new/clean" machine (virt if you want), and verify
>>>> that everything is working. If this is the case - then there is some
>>>> mixup on your current installation.
>>>>
>>>> 2)
>>>> Suricata.yaml and yaml in general is very peculiar about spaces/tabs
>>>> being at the right place and such. Please make sure some miss editing
>>>> is not the issue. (try loading the default provided suricata.yaml from
>>>> source)
>>>>
>>>> 3)
>>>> Can you copy paste your suricata.log on pastebin and share it?
>>>>
>>>> 4)
>>>> Can you provide the output of
>>>> ldd /path/to/suricata_executable
>>>> (example - ldd /usr/local/bin/suricata)
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Peter Manev
>>>
>>> Hi Peter,
>>>
>>> Thanks for the quick reply, but JSON output works fine, but everything
>>> regarding HTTP does not work anymore (IDS rules, http-log, etc).
>>>
>>> Also if use exactly the same config in the 1.4.7 and 2.0 I don't see any
>>> HTTP related matching (IDS rules, http-log). Even 2.0 with vanilla
>>> config from the build does not process any HTTP packets.
>>>
>>> Unfortunately I have no quick possibility to install on a clean machine
>>> since I am dependent on the monitor port/switch configuration to give me
>>> the traffic I need to match against. So I will have to figure out a way
>>> to do this.
>>>
>>> As for the suritcata log, it got overwritten unfortunately and I already
>>> reverted back to 1.4.7 due to the monitoring I need to do.
>>>
>>> As for the ldd, see below.
>>>
>>> Thanks again,
>>> Martijn
>>>
>>> Working 1.4.7:
>>> # ldd suricata-1.4.7/src/.libs/suricata
>>> linux-vdso.so.1 => (0x00007fffc9d38000)
>>> libhtp-0.2.so.1 => /usr/lib/libhtp-0.2.so.1 (0x00007ff7806bf000)
>>> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007ff7804a1000)
>>> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007ff780260000)
>>> libnet.so.1 => /lib64/libnet.so.1 (0x00007ff780047000)
>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff77fe2a000)
>>> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007ff77fc0a000)
>>> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007ff77f9de000)
>>> libc.so.6 => /lib64/libc.so.6 (0x00007ff77f64a000)
>>> libz.so.1 => /lib64/libz.so.1 (0x00007ff77f433000)
>>> /lib64/ld-linux-x86-64.so.2 (0x00007ff7808df000)
>>>
>>> Not working 2.0:
>>> ldd suricata-2.0/src/.libs/suricata
>>> linux-vdso.so.1 => (0x00007fff7d1ff000)
>>> libhtp-0.5.10.so.1 => /usr/lib/libhtp-0.5.10.so.1
>>> (0x00007fa905b92000)
>>> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007fa905974000)
>>> libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007fa90576e000)
>>> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007fa90552e000)
>>> libnet.so.1 => /lib64/libnet.so.1 (0x00007fa905315000)
>>> libjansson.so.4 => /usr/lib64/libjansson.so.4 (0x00007fa905109000)
>>> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007fa904eea000)
>>> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fa904cbe000)
>>> libssl3.so => /usr/lib64/libssl3.so (0x00007fa904a7f000)
>>> libsmime3.so => /usr/lib64/libsmime3.so (0x00007fa904853000)
>>> libnss3.so => /usr/lib64/libnss3.so (0x00007fa904515000)
>>> libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007fa9042e8000)
>>> libplds4.so => /lib64/libplds4.so (0x00007fa9040e4000)
>>> libplc4.so => /lib64/libplc4.so (0x00007fa903edf000)
>>> libnspr4.so => /lib64/libnspr4.so (0x00007fa903ca1000)
>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa903a84000)
>>> libdl.so.2 => /lib64/libdl.so.2 (0x00007fa903880000)
>>> libc.so.6 => /lib64/libc.so.6 (0x00007fa9034eb000)
>>> libz.so.1 => /lib64/libz.so.1 (0x00007fa9032d5000)
>>> /lib64/ld-linux-x86-64.so.2 (0x00007fa905dbb000)
>>> librt.so.1 => /lib64/librt.so.1 (0x00007fa9030cc000)
>>>
>>> Not working GIT:
>>> # ldd suricata-git/oisf/src/.libs/suricata
>>> linux-vdso.so.1 => (0x00007fff3fef7000)
>>> libhtp-0.5.11.so.1 => /usr/lib/libhtp-0.5.11.so.1
>>> (0x00007fdf65fa1000)
>>> libz.so.1 => /lib64/libz.so.1 (0x00007fdf65d8b000)
>>> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007fdf65b6c000)
>>> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007fdf6592c000)
>>> libnet.so.1 => /lib64/libnet.so.1 (0x00007fdf65713000)
>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdf654f5000)
>>> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007fdf652d6000)
>>> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fdf650aa000)
>>> libc.so.6 => /lib64/libc.so.6 (0x00007fdf64d15000)
>>> /lib64/ld-linux-x86-64.so.2 (0x00007fdf661cb000)
>>>
>> Do you still have some stats.log records from the non-working setup?
>>
>> One thing that may affect this is the updated vlan support. You may want
>> to try:
>>
>> # This option controls the use of vlan ids in the flow (and defrag)
>> # hashing. Normally this should be enabled, but in some (broken)
>> # setups where both sides of a flow are not tagged with the same vlan
>> # tag, we can ignore the vlan id's in the flow hashing.
>> vlan:
>> use-for-tracking: false
>>
>> Some weird equipment tags only one side of the conversation.
>>
> Victor, if I could remotely hug you I would ! :)
>
> Bingo, vlan tagging probably gave me headaches. Could be the case since I
> indeed have a many-to-1 port monitor of ports in different VLAN's on the
> switch to enable me to see VPN traffic too before it's encrypted alongside
> the browser traffic from users. Disables it and all works fine!
This is an excellent corner case example !
Thanks Martijn for sharing/confirming the info.
>
> Thanks a bundle! And sorry all for the hassle!
>
> Regards,
> Martijn
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/
--
Regards,
Peter Manev
More information about the Oisf-users
mailing list