[Oisf-devel] Suricata file-store not logging md5
Peter Manev
petermanev at gmail.com
Tue May 1 09:50:12 UTC 2012
On Tue, May 1, 2012 at 8:55 AM, Victor Julien <victor at inliniac.net> wrote:
> On 04/30/2012 06:44 PM, Mike Cox wrote:
> > Peter,
> >
> > I do not have JSON logging enabled, just file-store with force-magic
> > and force-md5. As you can see, MAGIC is included and it is all files
> > that do not have the MD5 sum included.
> >
> > To answer Marcos' question about libnss, I believe it is installed:
> >
> > [root at SURI2]# locate libnss
> > /lib/libnss_compat-2.5.so
> > /lib/libnss_compat.so.2
> > /lib/libnss_db-2.2.so
> > /lib/libnss_db.so.2
> > /lib/libnss_dns-2.5.so
> > /lib/libnss_dns.so.2
> > /lib/libnss_files-2.5.so
> > /lib/libnss_files.so.2
> > /lib/libnss_hesiod-2.5.so
> > /lib/libnss_hesiod.so.2
> > /lib/libnss_ldap-2.5.so
> > /lib/libnss_ldap.so.2
> > /lib/libnss_nis-2.5.so
> > /lib/libnss_nis.so.2
> > /lib/libnss_nisplus-2.5.so
> > /lib/libnss_nisplus.so.2
> > /lib/libnss_winbind.so.2
> > /lib/libnss_wins.so.2
> > /usr/lib/libnss3.so
> > /usr/lib/libnss_compat.so
> > /usr/lib/libnss_db.so
> > /usr/lib/libnss_dns.so
> > /usr/lib/libnss_files.so
> > /usr/lib/libnss_hesiod.so
> > /usr/lib/libnss_ldap.so
> > /usr/lib/libnss_nis.so
> > /usr/lib/libnss_nisplus.so
> > /usr/lib/libnss_winbind.so
> > /usr/lib/libnss_wins.so
> > /usr/lib/libnssckbi.so
> > /usr/lib/libnssutil3.so
> > [root at SURI2 files]# which md5sum
> > /usr/bin/md5sum
> >
> > Suricata was configured/installed with:
> >
> > ./configure --enable-gccprotect --enable-profiling --enable-pfring
> > --with-libpfring-libraries=/usr/local/lib
> > --with-libpfring-includes=/usr/local/include
> > --with-libpcap-libraries=/usr/local/lib
> > --with-libpcap-includes=/usr/local/include
> > --with-libhtp-includes=/usr/local/include
> > --with-libhtp-libraries=/usr/local/lib --prefix=/usr/local/
> > --sysconfdir=/etc/ --localstatedir=/var/
>
> Can you check if NSS was truly built in?
>
> $ suricata --build-info
> [6793] 1/5/2012 -- 06:50:32 - (suricata.c:502) <Info> (SCPrintBuildInfo)
> -- This is Suricata version 1.3dev (rev 5cc459f)
> [6793] 1/5/2012 -- 06:50:32 - (suricata.c:575) <Info> (SCPrintBuildInfo)
> -- Features: UNITTESTS NFQ PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1
> AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1
> HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW
> PCRE_JIT *HAVE_NSS* PROFILING PROFILE_LOCKING
>
> Another check would be looking at the output of "ldd" to see if suricata
> is linked to libnss.
>
> Cheers,
> Victor
>
>
Hi,
Just to clarify -
I tested it from scratch, loading only that rule (-S option):
alert http any any -> any any (msg:"FILE store all"; filestore; sid:99;
rev:99;)
you have to compile suri like this (in order to enable MD5s, for my Ubuntu
at least):
./autogen.sh && ./configure --enable-debug --enable-profiling
--enable-profiling-locks --with-libnss-libraries=/usr/lib
--with-libnss-includes=/usr/include/nss/ --with-libnspr-libraries=/usr/lib
--with-libnspr-includes=/usr/include/nspr && make clean && make install
"--enable-debug --enable-profiling --enable-profiling-locks" - are not
mandatory
output of configure:
""
Suricata Configuration:
AF_PACKET support: yes
PF_RING support: no
NFQueue support: no
IPFW support: no
DAG enabled: no
Napatech enabled: no
* libnss support: yes
libnspr support: yes*
Prelude support: no
PCRE jit: no
........
""
/Downloads/oisf# suricata --build-info
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:502) <Info> (SCPrintBuildInfo)
-- This is Suricata version 1.3dev (rev e6dea5c)
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:575) <Info> (SCPrintBuildInfo)
-- Features: DEBUG UNITTESTS PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1
AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1
HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW
*HAVE_NSS*PROFILING PROFILE_LOCKING
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:589) <Info> (SCPrintBuildInfo)
-- 32-bits, Little-endian architecture
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:591) <Info> (SCPrintBuildInfo)
-- GCC version 4.4.5, C version 199901
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:597) <Info> (SCPrintBuildInfo)
-- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:600) <Info> (SCPrintBuildInfo)
-- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:603) <Info> (SCPrintBuildInfo)
-- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:606) <Info> (SCPrintBuildInfo)
-- __GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:613) <Info> (SCPrintBuildInfo)
-- compiled with -fstack-protector
[10010] 1/5/2012 -- 11:16:23 - (suricata.c:619) <Info> (SCPrintBuildInfo)
-- compiled with _FORTIFY_SOURCE=2
in suricata yaml:
- file-store:
enabled: yes # set to yes to enable
log-dir: files # directory to store the files
force-magic: yes # force logging magic on all stored files
force-md5: yes # force logging of md5 checksums
#waldo: file.waldo # waldo file to store the file_id across runs
...
I tried that link (Cisco Prod Brochure PDF):
http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q
and in file directory i got the meta data:
""
TIME: 05/01/2012-11:09:52.425751
SRC IP: 2.23.144.170
DST IP: 192.168.1.91
PROTO: 6
SRC PORT: 80
DST PORT: 51598
HTTP URI:
/en/US/prod/collateral/routers/ps5855/prod_brochure0900aecd8019dc1f.pdf
HTTP HOST: www.cisco.com
HTTP REFERER:
http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q
FILENAME:
/en/US/prod/collateral/routers/ps5855/prod_brochure0900aecd8019dc1f.pdf
MAGIC: PDF document, version 1.6
STATE: CLOSED
MD5: *59eba188e52467adc11bf2442ee5bf57*
SIZE: 9485123
""
and in files-json.log :
cat /var/log/suricata/files-json.log |grep *59eba188e52467adc11bf2442ee5bf57
*
{ "id": 1, "timestamp": "05\/01\/2012-11:10:27.693583", "ipver": 4,
"srcip": "2.23.144.170", "dstip": "192.168.1.91", "protocol": 6, "sp": 80,
"dp": 51598, "http_uri":
"\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf",
"http_host": "www.cisco.com", "http_referer": "http:\/\/www.google.com
\/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%
2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q",
"filename":
"\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf",
"magic": "PDF document, version 1.6", "state": "CLOSED", "md5":
"59eba188e52467adc11bf2442ee5bf57", "stored": true, "size": 9485123 }
{ "id": 12, "timestamp": "05\/01\/2012-11:12:57.421420", "ipver": 4,
"srcip": "2.23.144.170", "dstip": "192.168.1.91", "protocol": 6, "sp": 80,
"dp": 51598, "http_uri":
"\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf",
"http_host": "www.cisco.com", "http_referer": "http:\/\/www.google.com
\/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CDAQFjAA&url=http%3A%2F%
2Fwww.cisco.com%2Fen%2FUS%2Fprod%2Fcollateral%2Frouters%2Fps5855%2Fprod_brochure0900aecd8019dc1f.pdf&ei=OqyfT9eoJubi4QTyiamhAw&usg=AFQjCNGdjDBpBDfQv2r3VogSH41V6T5x9Q",
"filename":
"\/en\/US\/prod\/collateral\/routers\/ps5855\/prod_brochure0900aecd8019dc1f.pdf",
"magic": "PDF document, version 1.6", "state": "CLOSED", "md5":
"59eba188e52467adc11bf2442ee5bf57", "stored": true, "size": 9485123 }
This is in short what is needed to have MD5s.
now that I look at it .....
why is the timing differnt in file.meta and files-json ?
> > Thanks.
> >
> > -Mike Cox
> >
> > On Mon, Apr 30, 2012 at 11:03 AM, Peter Manev <petermanev at gmail.com>
> wrote:
> >> Hi,
> >>
> >> do you have the MD5s in your JSON log file?
> >>
> >> and is it just this file that does not have MD5 or all files?
> >>
> >> thanks
> >>
> >> On Mon, Apr 30, 2012 at 4:38 PM, Mike Cox <mike.cox52 at gmail.com> wrote:
> >>>
> >>> I have grabbed the latest version of Suricata from GIT and enabled
> >>> file-store. However, in the meta file, I do not see the md5 sum being
> >>> logged. Of course, if the file is logged too, calculating the md5 on
> >>> the sensor machine (outside of Suricata) is trivial but I though it
> >>> would log the md5 if it was enabled. From my config .yaml file:
> >>>
> >>> - file-store:
> >>> enabled: yes # set to yes to enable
> >>> log-dir: files # directory to store the files
> >>> force-magic: yes # force logging magic on all stored files
> >>> force-md5: yes # force logging of md5 checksums
> >>> #waldo: file.waldo # waldo file to store the file-id across runs
> >>>
> >>> I have the stream reassembly and HTTP request/response body sizes set
> >>> high enough that I am getting all of the file but I don't see the MD5
> >>> sum logged. From the meta file:
> >>>
> >>> TIME: 04/28/2012-03:31:01.457465
> >>> SRC IP: 97.67.101.89
> >>> DST IP: 192.168.5.21
> >>> PROTO: 6
> >>> SRC PORT: 80
> >>> DST PORT: 24593
> >>> HTTP URI:
> >>>
> >>>
> /msdownload/update/software/defu/2012/04/am_delta_patch_1.125.561.0_07370866e162114165aa31f821c0ef655ef41117.exe
> >>> HTTP HOST: download.windowsupdate.com
> >>> HTTP REFERER: <unknown>
> >>> FILENAME:
> >>>
> >>>
> /msdownload/update/software/defu/2012/04/am_delta_patch_1.125.561.0_07370866e162114165aa31f821c0ef655ef41117.exe
> >>> MAGIC: PE32+ executable for MS Windows (GUI)
> >>> STATE: CLOSED
> >>> SIZE: 5382
> >>>
> >>> Also, does the filename normally include all the URL?
> >>>
> >>> This is Suricata 1.3dev (rev e6dea5c).
> >>>
> >>> Thanks.
> >>>
> >>> -Mike Cox
> >>> _______________________________________________
> >>> Oisf-devel mailing list
> >>> Oisf-devel at openinfosecfoundation.org
> >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
> >>
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Peter Manev
> >>
> > _______________________________________________
> > Oisf-devel mailing list
> > Oisf-devel at openinfosecfoundation.org
> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
> >
>
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Oisf-devel mailing list
> Oisf-devel at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>
--
Regards,
Peter Manev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-devel/attachments/20120501/a335c23b/attachment-0002.html>
More information about the Oisf-devel
mailing list