[Oisf-users] Http/libhtp issue Suricata 2.0 on CentOS 6
(OISF) Martijn Schoemaker
oisf at ficture.nl
Thu May 1 16:51:09 UTC 2014
On 05/01/2014 05:41 PM, Peter Manev wrote:
> On Thu, May 1, 2014 at 5:34 PM, (OISF) Martijn Schoemaker
> <oisf at ficture.nl> wrote:
>> On 05/01/2014 05:32 PM, Martijn Schoemaker (Ficture IT) wrote:
>>
>>
>> On 05/01/2014 05:17 PM, Victor Julien wrote:
>>> On 05/01/2014 05:12 PM, (OISF) Martijn Schoemaker wrote:
>>>>
>>>>> On Thu, May 1, 2014 at 3:22 PM, (OISF) Martijn Schoemaker
>>>>> <oisf at ficture.nl <mailto:oisf at ficture.nl>> wrote:
>>>>>
>>>>>
>>>>> Some additional info:
>>>>>
>>>>> Working 1.4.7 release:
>>>>> --------------------------------
>>>>> # suricata-1.4.7/src/suricata --build-info
>>>>> This is Suricata version 1.4.7 RELEASE
>>>>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>>>>> HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>>>>> HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW
>>>>> 64-bits, Little-endian architecture
>>>>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>>>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
>>>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
>>>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
>>>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
>>>>> __GCC_HAVE_SYNC_COMPARE_AND_SWAP_16
>>>>> compiled with libhtp 0.2.14, linked against 0.2.14
>>>>> Suricata Configuration:
>>>>> AF_PACKET support: yes
>>>>> PF_RING support: no
>>>>> NFQueue support: no
>>>>> IPFW support: no
>>>>> DAG enabled: no
>>>>> Napatech enabled: no
>>>>> Unix socket enabled: no
>>>>>
>>>>> libnss support: no
>>>>> libnspr support: no
>>>>> libjansson support: no
>>>>> Prelude support: no
>>>>> PCRE jit: no
>>>>> libluajit: no
>>>>> libgeoip: no
>>>>> Non-bundled htp: no
>>>>> Old barnyard2 support: no
>>>>> CUDA enabled: no
>>>>>
>>>>> Suricatasc install: yes
>>>>>
>>>>> Unit tests enabled: no
>>>>> Debug output enabled: no
>>>>> Debug validation enabled: no
>>>>> Profiling enabled: no
>>>>> Profiling locks enabled: no
>>>>>
>>>>> Generic build parameters:
>>>>> Installation prefix (--prefix): /usr
>>>>> Configuration directory (--sysconfdir): /etc/suricata/
>>>>> Log directory (--localstatedir) : /var/log/suricata/
>>>>>
>>>>> Host: x86_64-unknown-linux-gnu
>>>>> GCC binary: gcc
>>>>> GCC Protect enabled: no
>>>>> GCC march native enabled: yes
>>>>> GCC Profile enabled: no
>>>>>
>>>>> Git release (not working):
>>>>> -------------------------------------
>>>>> # suricata-git/oisf/src/suricata --build-info
>>>>> This is Suricata version 2.0dev (rev 6fbb955)
>>>>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>>>>> HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>>>>> SIMD support: SSE_3
>>>>> Atomic intrisics: 1 2 4 8 16 byte(s)
>>>>> 64-bits, Little-endian architecture
>>>>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>>>>> L1 cache line size (CLS)=64
>>>>> compiled with LibHTP v0.5.11, linked against LibHTP v0.5.11
>>>>> Suricata Configuration:
>>>>> AF_PACKET support: yes
>>>>> PF_RING support: no
>>>>> NFQueue support: no
>>>>> IPFW support: no
>>>>> DAG enabled: no
>>>>> Napatech enabled: no
>>>>> Unix socket enabled: no
>>>>> Detection enabled: yes
>>>>>
>>>>> libnss support: no
>>>>> libnspr support: no
>>>>> libjansson support: no
>>>>> Prelude support: no
>>>>> PCRE jit: no
>>>>> libluajit: no
>>>>> libgeoip: no
>>>>> Non-bundled htp: no
>>>>> Old barnyard2 support: no
>>>>> CUDA enabled: no
>>>>>
>>>>> Suricatasc install: yes
>>>>>
>>>>> Unit tests enabled: no
>>>>> Debug output enabled: no
>>>>> Debug validation enabled: no
>>>>> Profiling enabled: no
>>>>> Profiling locks enabled: no
>>>>> Coccinelle / spatch: no
>>>>>
>>>>> Generic build parameters:
>>>>> Installation prefix (--prefix): /usr
>>>>> Configuration directory (--sysconfdir): /etc/suricata/
>>>>> Log directory (--localstatedir) : /var/log/suricata/
>>>>>
>>>>> Host: x86_64-unknown-linux-gnu
>>>>> GCC binary: gcc
>>>>> GCC Protect enabled: no
>>>>> GCC march native enabled: yes
>>>>> GCC Profile enabled: no
>>>>>
>>>>> 2.0 release (also not working):
>>>>> -------------------------------------------
>>>>> # suricata-2.0/src/suricata --build-info
>>>>> This is Suricata version 2.0 RELEASE
>>>>> Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET
>>>>> HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK
>>>>> HAVE_NSS HAVE_LIBJANSSON
>>>>> SIMD support: SSE_3
>>>>> Atomic intrisics: 1 2 4 8 16 byte(s)
>>>>> 64-bits, Little-endian architecture
>>>>> GCC version 4.4.7 20120313 (Red Hat 4.4.7-4), C version 199901
>>>>> L1 cache line size (CLS)=64
>>>>> compiled with LibHTP v0.5.10, linked against LibHTP v0.5.10
>>>>> Suricata Configuration:
>>>>> AF_PACKET support: yes
>>>>> PF_RING support: no
>>>>> NFQueue support: no
>>>>> IPFW support: no
>>>>> DAG enabled: no
>>>>> Napatech enabled: no
>>>>> Unix socket enabled: yes
>>>>> Detection enabled: yes
>>>>>
>>>>> libnss support: yes
>>>>> libnspr support: yes
>>>>> libjansson support: yes
>>>>> Prelude support: no
>>>>> PCRE jit: no
>>>>> libluajit: no
>>>>> libgeoip: no
>>>>> Non-bundled htp: no
>>>>> Old barnyard2 support: no
>>>>> CUDA enabled: no
>>>>>
>>>>> Suricatasc install: yes
>>>>>
>>>>> Unit tests enabled: no
>>>>> Debug output enabled: no
>>>>> Debug validation enabled: no
>>>>> Profiling enabled: no
>>>>> Profiling locks enabled: no
>>>>> Coccinelle / spatch: yes
>>>>>
>>>>> Generic build parameters:
>>>>> Installation prefix (--prefix): /usr
>>>>> Configuration directory (--sysconfdir): /etc/suricata/
>>>>> Log directory (--localstatedir) : /var/log/suricata/
>>>>>
>>>>> Host: x86_64-unknown-linux-gnu
>>>>> GCC binary: gcc
>>>>> GCC Protect enabled: no
>>>>> GCC march native enabled: yes
>>>>> GCC Profile enabled: no
>>>>>
>>>>>
>>>>> On 05/01/2014 03:16 PM, (OISF) Martijn Schoemaker wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I have been running suricata 1.4.7 for quite some time and
>>>>> it's working like a charm. When I saw that suricata 2.0 supports the
>>>>> eve-json log format for integration with logstash I wanted to upgrade
>>>>> to 2.0.
>>>>>
>>>>> I downloaded the stable 2.0 release, built it and all seemed
>>>>> to run fine. However, I notices the http.log was no longer modified.
>>>>> Further investigation showed that all http event matching, http
>>>>> logging (http-log and eve http log) was no longer working. I started
>>>>> out with the exact same config as the working 1.4.7 release, then
>>>>> modified the 2.0 config accordingly but it just won't work.
>>>>>
>>>>> I also noticed it now includes libhtp 0.5.10 instead of 0.2 so
>>>>> I tried to build against 0.2 but that's not supported. I also built
>>>>> the git current release (libhtp 0.5.11), but still no go. Strange
>>>>> thing is that http events are also no longer matched. I run on a
>>>>> machine which is connected to a monitor port so it cannot be checksum
>>>>> offloading (I also manually disabled it on the interface and disabled
>>>>> checksum checking in the suricata config, but all to no avail).
>>>>>
>>>>> Whenever I revert to the 1.4.7 release everything works again.
>>>>>
>>>>> So I have a big suspicion that either I'm doing something
>>>>> terribly wrong, or the libhtp 0.5 release is not working correctly
>>>>> anymore.
>>>>>
>>>>> Is there anyone who observed the same issue ?
>>>>>
>>>>> Regards,
>>>>> Martijn Schoemaker
>>>>>
>>>>> _______________________________________________
>>>>> Suricata IDS Users mailing list:
>>>>> oisf-users at openinfosecfoundation.org
>>>>> <mailto:oisf-users at openinfosecfoundation.org>
>>>>> Site: http://suricata-ids.org | Support:
>>>>> http://suricata-ids.org/support/
>>>>> List:
>>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>> OISF: http://www.openinfosecfoundation.org/
>>>>>
>>>>> _______________________________________________
>>>>> Suricata IDS Users mailing list:
>>>>> oisf-users at openinfosecfoundation.org
>>>>> <mailto:oisf-users at openinfosecfoundation.org>
>>>>> Site: http://suricata-ids.org | Support:
>>>>> http://suricata-ids.org/support/
>>>>> List:
>>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>> OISF: http://www.openinfosecfoundation.org/
>>>>>
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I think there is some sort of a (miss)configuration issue. For the
>>>>> JSON output to work you need libjansson4 and libjansson-dev present on
>>>>> the system.
>>>>> When you do (suricata --build-info) you should see -> " libjansson
>>>>> support: yes"
>>>>>
>>>>> What I would suggest -
>>>>>
>>>>> 1)
>>>>> Install 2.0 an a "new/clean" machine (virt if you want), and verify
>>>>> that everything is working. If this is the case - then there is some
>>>>> mixup on your current installation.
>>>>>
>>>>> 2)
>>>>> Suricata.yaml and yaml in general is very peculiar about spaces/tabs
>>>>> being at the right place and such. Please make sure some miss editing
>>>>> is not the issue. (try loading the default provided suricata.yaml from
>>>>> source)
>>>>>
>>>>> 3)
>>>>> Can you copy paste your suricata.log on pastebin and share it?
>>>>>
>>>>> 4)
>>>>> Can you provide the output of
>>>>> ldd /path/to/suricata_executable
>>>>> (example - ldd /usr/local/bin/suricata)
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Peter Manev
>>>> Hi Peter,
>>>>
>>>> Thanks for the quick reply, but JSON output works fine, but everything
>>>> regarding HTTP does not work anymore (IDS rules, http-log, etc).
>>>>
>>>> Also if use exactly the same config in the 1.4.7 and 2.0 I don't see any
>>>> HTTP related matching (IDS rules, http-log). Even 2.0 with vanilla
>>>> config from the build does not process any HTTP packets.
>>>>
>>>> Unfortunately I have no quick possibility to install on a clean machine
>>>> since I am dependent on the monitor port/switch configuration to give me
>>>> the traffic I need to match against. So I will have to figure out a way
>>>> to do this.
>>>>
>>>> As for the suritcata log, it got overwritten unfortunately and I already
>>>> reverted back to 1.4.7 due to the monitoring I need to do.
>>>>
>>>> As for the ldd, see below.
>>>>
>>>> Thanks again,
>>>> Martijn
>>>>
>>>> Working 1.4.7:
>>>> # ldd suricata-1.4.7/src/.libs/suricata
>>>> linux-vdso.so.1 => (0x00007fffc9d38000)
>>>> libhtp-0.2.so.1 => /usr/lib/libhtp-0.2.so.1 (0x00007ff7806bf000)
>>>> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007ff7804a1000)
>>>> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007ff780260000)
>>>> libnet.so.1 => /lib64/libnet.so.1 (0x00007ff780047000)
>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff77fe2a000)
>>>> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007ff77fc0a000)
>>>> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007ff77f9de000)
>>>> libc.so.6 => /lib64/libc.so.6 (0x00007ff77f64a000)
>>>> libz.so.1 => /lib64/libz.so.1 (0x00007ff77f433000)
>>>> /lib64/ld-linux-x86-64.so.2 (0x00007ff7808df000)
>>>>
>>>> Not working 2.0:
>>>> ldd suricata-2.0/src/.libs/suricata
>>>> linux-vdso.so.1 => (0x00007fff7d1ff000)
>>>> libhtp-0.5.10.so.1 => /usr/lib/libhtp-0.5.10.so.1
>>>> (0x00007fa905b92000)
>>>> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007fa905974000)
>>>> libcap-ng.so.0 => /lib64/libcap-ng.so.0 (0x00007fa90576e000)
>>>> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007fa90552e000)
>>>> libnet.so.1 => /lib64/libnet.so.1 (0x00007fa905315000)
>>>> libjansson.so.4 => /usr/lib64/libjansson.so.4 (0x00007fa905109000)
>>>> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007fa904eea000)
>>>> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fa904cbe000)
>>>> libssl3.so => /usr/lib64/libssl3.so (0x00007fa904a7f000)
>>>> libsmime3.so => /usr/lib64/libsmime3.so (0x00007fa904853000)
>>>> libnss3.so => /usr/lib64/libnss3.so (0x00007fa904515000)
>>>> libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007fa9042e8000)
>>>> libplds4.so => /lib64/libplds4.so (0x00007fa9040e4000)
>>>> libplc4.so => /lib64/libplc4.so (0x00007fa903edf000)
>>>> libnspr4.so => /lib64/libnspr4.so (0x00007fa903ca1000)
>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa903a84000)
>>>> libdl.so.2 => /lib64/libdl.so.2 (0x00007fa903880000)
>>>> libc.so.6 => /lib64/libc.so.6 (0x00007fa9034eb000)
>>>> libz.so.1 => /lib64/libz.so.1 (0x00007fa9032d5000)
>>>> /lib64/ld-linux-x86-64.so.2 (0x00007fa905dbb000)
>>>> librt.so.1 => /lib64/librt.so.1 (0x00007fa9030cc000)
>>>>
>>>> Not working GIT:
>>>> # ldd suricata-git/oisf/src/.libs/suricata
>>>> linux-vdso.so.1 => (0x00007fff3fef7000)
>>>> libhtp-0.5.11.so.1 => /usr/lib/libhtp-0.5.11.so.1
>>>> (0x00007fdf65fa1000)
>>>> libz.so.1 => /lib64/libz.so.1 (0x00007fdf65d8b000)
>>>> libmagic.so.1 => /usr/lib64/libmagic.so.1 (0x00007fdf65b6c000)
>>>> libpcap.so.1 => /usr/lib64/libpcap.so.1 (0x00007fdf6592c000)
>>>> libnet.so.1 => /lib64/libnet.so.1 (0x00007fdf65713000)
>>>> libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdf654f5000)
>>>> libyaml-0.so.2 => /usr/lib64/libyaml-0.so.2 (0x00007fdf652d6000)
>>>> libpcre.so.0 => /lib64/libpcre.so.0 (0x00007fdf650aa000)
>>>> libc.so.6 => /lib64/libc.so.6 (0x00007fdf64d15000)
>>>> /lib64/ld-linux-x86-64.so.2 (0x00007fdf661cb000)
>>>>
>>> Do you still have some stats.log records from the non-working setup?
>>>
>>> One thing that may affect this is the updated vlan support. You may want
>>> to try:
>>>
>>> # This option controls the use of vlan ids in the flow (and defrag)
>>> # hashing. Normally this should be enabled, but in some (broken)
>>> # setups where both sides of a flow are not tagged with the same vlan
>>> # tag, we can ignore the vlan id's in the flow hashing.
>>> vlan:
>>> use-for-tracking: false
>>>
>>> Some weird equipment tags only one side of the conversation.
>>>
>> Victor, if I could remotely hug you I would ! :)
>>
>> Bingo, vlan tagging probably gave me headaches. Could be the case since I
>> indeed have a many-to-1 port monitor of ports in different VLAN's on the
>> switch to enable me to see VPN traffic too before it's encrypted alongside
>> the browser traffic from users. Disables it and all works fine!
> This is an excellent corner case example !
> Thanks Martijn for sharing/confirming the info.
>
Np, and you thanks for the quick replies! Hope this will help someone else and save them a day of debugging, compiling, and headaches :)
Regards,
Martijn
>> Thanks a bundle! And sorry all for the hassle!
>>
>> Regards,
>> Martijn
>>
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> OISF: http://www.openinfosecfoundation.org/
>
>
More information about the Oisf-users
mailing list