<br><br><div class="gmail_quote">On Tue, May 1, 2012 at 8:55 AM, Victor Julien <span dir="ltr"><<a href="mailto:victor@inliniac.net" target="_blank">victor@inliniac.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">On 04/30/2012 06:44 PM, Mike Cox wrote:<br>
> Peter,<br>
><br>
> I do not have JSON logging enabled, just file-store with force-magic<br>
> and force-md5.  As you can see, MAGIC is included and it is all files<br>
> that do not have the MD5 sum included.<br>
><br>
> To answer Marcos' question about libnss, I believe it is installed:<br>
><br>
> [root@SURI2]# locate libnss<br>
> /lib/<a href="http://libnss_compat-2.5.so" target="_blank">libnss_compat-2.5.so</a><br>
> /lib/libnss_compat.so.2<br>
> /lib/<a href="http://libnss_db-2.2.so" target="_blank">libnss_db-2.2.so</a><br>
> /lib/libnss_db.so.2<br>
> /lib/<a href="http://libnss_dns-2.5.so" target="_blank">libnss_dns-2.5.so</a><br>
> /lib/libnss_dns.so.2<br>
> /lib/<a href="http://libnss_files-2.5.so" target="_blank">libnss_files-2.5.so</a><br>
> /lib/libnss_files.so.2<br>
> /lib/<a href="http://libnss_hesiod-2.5.so" target="_blank">libnss_hesiod-2.5.so</a><br>
> /lib/libnss_hesiod.so.2<br>
> /lib/<a href="http://libnss_ldap-2.5.so" target="_blank">libnss_ldap-2.5.so</a><br>
> /lib/libnss_ldap.so.2<br>
> /lib/<a href="http://libnss_nis-2.5.so" target="_blank">libnss_nis-2.5.so</a><br>
> /lib/libnss_nis.so.2<br>
> /lib/<a href="http://libnss_nisplus-2.5.so" target="_blank">libnss_nisplus-2.5.so</a><br>
> /lib/libnss_nisplus.so.2<br>
> /lib/libnss_winbind.so.2<br>
> /lib/libnss_wins.so.2<br>
> /usr/lib/libnss3.so<br>
> /usr/lib/libnss_compat.so<br>
> /usr/lib/libnss_db.so<br>
> /usr/lib/libnss_dns.so<br>
> /usr/lib/libnss_files.so<br>
> /usr/lib/libnss_hesiod.so<br>
> /usr/lib/libnss_ldap.so<br>
> /usr/lib/libnss_nis.so<br>
> /usr/lib/libnss_nisplus.so<br>
> /usr/lib/libnss_winbind.so<br>
> /usr/lib/libnss_wins.so<br>
> /usr/lib/libnssckbi.so<br>
> /usr/lib/libnssutil3.so<br>
> [root@SURI2 files]# which md5sum<br>
> /usr/bin/md5sum<br>
><br>
> Suricata was configured/installed with:<br>
><br>
> ./configure --enable-gccprotect --enable-profiling --enable-pfring<br>
> --with-libpfring-libraries=/usr/local/lib<br>
> --with-libpfring-includes=/usr/local/include<br>
> --with-libpcap-libraries=/usr/local/lib<br>
> --with-libpcap-includes=/usr/local/include<br>
> --with-libhtp-includes=/usr/local/include<br>
> --with-libhtp-libraries=/usr/local/lib --prefix=/usr/local/<br>
> --sysconfdir=/etc/ --localstatedir=/var/<br>
<br>
</div></div>Can you check if NSS was truly built in?<br>
<br>
$ suricata --build-info<br>
[6793] 1/5/2012 -- 06:50:32 - (suricata.c:502) <Info> (SCPrintBuildInfo)<br>
-- This is Suricata version 1.3dev (rev 5cc459f)<br>
[6793] 1/5/2012 -- 06:50:32 - (suricata.c:575) <Info> (SCPrintBuildInfo)<br>
-- Features: UNITTESTS NFQ PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1<br>
AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1<br>
HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW<br>
PCRE_JIT *HAVE_NSS* PROFILING PROFILE_LOCKING<br>
<br>
Another check would be looking at the output of "ldd" to see if suricata<br>
is linked to libnss.<br>
<br>
Cheers,<br>
Victor<br>
<div class="HOEnZb"><div class="h5"><br></div></div></blockquote><div><br>Hi,<br><br>Just to clarify - <br>I tested it from scratch, loading only that rule (-S option):<br><br>alert http any any -> any any (msg:"FILE store all"; filestore; sid:99; rev:99;)<br>
<br><br>you have to compile suri like this (in order to enable MD5s, for my Ubuntu at least):<br>./autogen.sh && ./configure --enable-debug --enable-profiling --enable-profiling-locks --with-libnss-libraries=/usr/lib --with-libnss-includes=/usr/include/nss/ --with-libnspr-libraries=/usr/lib --with-libnspr-includes=/usr/include/nspr && make clean && make install<br>
<br>"--enable-debug --enable-profiling --enable-profiling-locks" - are not mandatory<br><br>output of configure:<br>""<br>Suricata Configuration:<br>  AF_PACKET support:                       yes<br>  PF_RING support:                         no<br>
  NFQueue support:                         no<br>  IPFW support:                            no<br>  DAG enabled:                             no<br>  Napatech enabled:                        no<br><br><u><b>  libnss support:                          yes<br>
  libnspr support:                         yes</b></u><br>  Prelude support:                         no<br>  PCRE jit:                                no<br>........<br><br>""<br><br>/Downloads/oisf# suricata --build-info<br>
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:502) <Info> (SCPrintBuildInfo) -- This is Suricata version 1.3dev (rev e6dea5c)<br>[10010] 1/5/2012 -- 11:16:23 - (suricata.c:575) <Info> (SCPrintBuildInfo) -- Features: DEBUG UNITTESTS PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW <u><b>HAVE_NSS</b></u> PROFILING PROFILE_LOCKING <br>
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:589) <Info> (SCPrintBuildInfo) -- 32-bits, Little-endian architecture<br>[10010] 1/5/2012 -- 11:16:23 - (suricata.c:591) <Info> (SCPrintBuildInfo) -- GCC version 4.4.5, C version 199901<br>
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:597) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1<br>[10010] 1/5/2012 -- 11:16:23 - (suricata.c:600) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2<br>
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:603) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4<br>[10010] 1/5/2012 -- 11:16:23 - (suricata.c:606) <Info> (SCPrintBuildInfo) -- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8<br>
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:613) <Info> (SCPrintBuildInfo) -- compiled with -fstack-protector<br>[10010] 1/5/2012 -- 11:16:23 - (suricata.c:619) <Info> (SCPrintBuildInfo) -- compiled with _FORTIFY_SOURCE=2<br>
<br>in suricata yaml:<br> - file-store:<br>      enabled: yes       # set to yes to enable<br>      log-dir: files    # directory to store the files<br>      force-magic: yes   # force logging magic on all stored files<br>
      force-md5: yes     # force logging of md5 checksums<br>      #waldo: file.waldo # waldo file to store the file_id across runs<br>...<br><br><br>I tried that link (Cisco Prod Brochure PDF):<br><a href="http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q">http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q</a><br>
<br>and in file directory i got the meta data:<br><br>""<br>TIME:              05/01/2012-11:09:52.425751<br>SRC IP:            2.23.144.170<br>DST IP:            192.168.1.91<br>PROTO:             6<br>SRC PORT:          80<br>
DST PORT:          51598<br>HTTP URI:          /en/US/prod/collateral/routers/ps5855/prod_brochure0900aecd8019dc1f.pdf<br>HTTP HOST:         <a href="http://www.cisco.com">www.cisco.com</a><br>HTTP REFERER:      <a href="http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q">http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q</a><br>
FILENAME:          /en/US/prod/collateral/routers/ps5855/prod_brochure0900aecd8019dc1f.pdf<br>MAGIC:             PDF document, version 1.6<br>STATE:             CLOSED<br>MD5:               <u><b>59eba188e52467adc11bf2442ee5bf57</b></u><br>
SIZE:              9485123<br>""<br><br>and in files-json.log :<br><br>cat /var/log/suricata/files-json.log |grep <u><b>59eba188e52467adc11bf2442ee5bf57</b></u><br><br>{ "id": 1, "timestamp": "05\/01\/2012-11:10:27.693583", "ipver": 4, "srcip": "2.23.144.170", "dstip": "192.168.1.91", "protocol": 6, "sp": 80, "dp": 51598, "http_uri": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "http_host": "<a href="http://www.cisco.com">www.cisco.com</a>", "http_referer": "http:\/\/<a href="http://www.google.com">www.google.com</a>\/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%<a href="http://2Fwww.cisco.com">2Fwww.cisco.com</a>%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q", "filename": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "magic": "PDF document, version 1.6", "state": "CLOSED", "md5": "59eba188e52467adc11bf2442ee5bf57", "stored": true, "size": 9485123 }<br>
<br>{ "id": 12, "timestamp": "05\/01\/2012-11:12:57.421420", "ipver": 4, "srcip": "2.23.144.170", "dstip": "192.168.1.91", "protocol": 6, "sp": 80, "dp": 51598, "http_uri": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "http_host": "<a href="http://www.cisco.com">www.cisco.com</a>", "http_referer": "http:\/\/<a href="http://www.google.com">www.google.com</a>\/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%<a href="http://2Fwww.cisco.com">2Fwww.cisco.com</a>%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q", "filename": "\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf", "magic": "PDF document, version 1.6", "state": "CLOSED", "md5": "59eba188e52467adc11bf2442ee5bf57", "stored": true, "size": 9485123 }<br>
<br> <br><br>This is in short what is needed to have MD5s.<br><br><br>now that I look at it .....<br>why is the timing differnt in file.meta and files-json ?<br><br><br><br><br><br></div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="HOEnZb"><div class="h5">
<br>
> Thanks.<br>
><br>
>  -Mike Cox<br>
><br>
> On Mon, Apr 30, 2012 at 11:03 AM, Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>> wrote:<br>
>> Hi,<br>
>><br>
>> do you have the MD5s in your JSON log file?<br>
>><br>
>> and is it just this file that does not have MD5 or all files?<br>
>><br>
>> thanks<br>
>><br>
>> On Mon, Apr 30, 2012 at 4:38 PM, Mike Cox <<a href="mailto:mike.cox52@gmail.com">mike.cox52@gmail.com</a>> wrote:<br>
>>><br>
>>> I have grabbed the latest version of Suricata from GIT and enabled<br>
>>> file-store.  However, in the meta file, I do not see the md5 sum being<br>
>>> logged.  Of course, if the file is logged too, calculating the md5 on<br>
>>> the sensor machine (outside of Suricata) is trivial but I though it<br>
>>> would log the md5 if it was enabled.  From my config .yaml file:<br>
>>><br>
>>>  - file-store:<br>
>>>     enabled: yes       # set to yes to enable<br>
>>>     log-dir: files    # directory to store the files<br>
>>>     force-magic: yes   # force logging magic on all stored files<br>
>>>     force-md5: yes     # force logging of md5 checksums<br>
>>>     #waldo: file.waldo # waldo file to store the file-id across runs<br>
>>><br>
>>> I have the stream reassembly and HTTP request/response body sizes set<br>
>>> high enough that I am getting all of the file but I don't see the MD5<br>
>>> sum logged.  From the meta file:<br>
>>><br>
>>> TIME:              04/28/2012-03:31:01.457465<br>
>>> SRC IP:            97.67.101.89<br>
>>> DST IP:            192.168.5.21<br>
>>> PROTO:             6<br>
>>> SRC PORT:          80<br>
>>> DST PORT:          24593<br>
>>> HTTP URI:<br>
>>><br>
>>> /msdownload/update/software/defu/2012/04/am_delta_patch_1.125.561.0_07370866e162114165aa31f821c0ef655ef41117.exe<br>
>>> HTTP HOST:         <a href="http://download.windowsupdate.com" target="_blank">download.windowsupdate.com</a><br>
>>> HTTP REFERER:      <unknown><br>
>>> FILENAME:<br>
>>><br>
>>> /msdownload/update/software/defu/2012/04/am_delta_patch_1.125.561.0_07370866e162114165aa31f821c0ef655ef41117.exe<br>
>>> MAGIC:             PE32+ executable for MS Windows (GUI)<br>
>>> STATE:             CLOSED<br>
>>> SIZE:              5382<br>
>>><br>
>>> Also, does the filename normally include all the URL?<br>
>>><br>
>>> This is Suricata 1.3dev (rev e6dea5c).<br>
>>><br>
>>> Thanks.<br>
>>><br>
>>>  -Mike Cox<br>
>>> _______________________________________________<br>
>>> Oisf-devel mailing list<br>
>>> <a href="mailto:Oisf-devel@openinfosecfoundation.org">Oisf-devel@openinfosecfoundation.org</a><br>
>>> <a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel</a><br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Regards,<br>
>> Peter Manev<br>
>><br>
> _______________________________________________<br>
> Oisf-devel mailing list<br>
> <a href="mailto:Oisf-devel@openinfosecfoundation.org">Oisf-devel@openinfosecfoundation.org</a><br>
> <a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel</a><br>
><br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
---------------------------------------------<br>
Victor Julien<br>
<a href="http://www.inliniac.net/" target="_blank">http://www.inliniac.net/</a><br>
PGP: <a href="http://www.inliniac.net/victorjulien.asc" target="_blank">http://www.inliniac.net/victorjulien.asc</a><br>
---------------------------------------------<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
Oisf-devel mailing list<br>
<a href="mailto:Oisf-devel@openinfosecfoundation.org">Oisf-devel@openinfosecfoundation.org</a><br>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div>Regards,</div>
<div>Peter Manev</div><br>