[Oisf-devel] Debug HashListTableLookup()

Wed Feb 10 07:47:31 UTC 2010

I have resolved this. Attached is the diff file. Instead of defining 
OS_SUNOS like I did, you probably want to use a new #define BIG_ENDIAN, 
so the patch is applicable to all big-endian processors.

Yaomin

On 02/04/10 11:12, Yao-Min Chen wrote:
> The issue is with the "any" ip address, when handled by 
> DetectAddressSetup().
>
>     /* if any, insert 0.0.0.0/0 and ::/0 as well */
>
> Not exactly sure the behavior for x86 but for sparc, the netmask 
> becomes 0xffffffff. Essentially the full range of IPv4 addresses are 
> added to the hash table.
>
> 1. IPOnlyAddSignature() calls IPOnlyAddSlash16() to add 0.0.0.0/0 in 
> 64K steps (2^32 divided by 2^16)
>
> 2. The hash function uses the lower 16 bits which are all 0's, the 
> same for each entry to be hashed.
>
>     #define IPONLY_EXTRACT_16(a) ((a)->ip[0] & 0x0000ffff)
>
> 3. This is the worst case for the hashed list table as all keys get 
> hashed to the same bucket. Towards the end of the 64K steps, each add 
> would traverse a linked list of size 64K.
>
> 4. I modified the hash function so the upper 16 bits are also used and 
> SigTestBidirec03 can run to completion now.
>
>     #define IPONLY_EXTRACT_16(a)\
>      ((((a)->ip[0] & 0x0000ffff) + (((a)->ip[0] & 0xffff0000)>>8)) & 
> 0x0000ffff)
>
> Question: I saw high = 0xffffffff and low = 0x0 in the following 
> invocation of IPOnlyAddSlash16(), for adding the address 0.0.0.0/0. 
> Are the high and low values correct?
>
> static void IPOnlyAddSlash16(DetectEngineCtx *de_ctx, 
> DetectEngineIPOnlyCtx *io_ctx,
> HashListTable *ht, DetectAddress *gr, char direction, Signature *s) {
>     uint32_t high = ntohl(gr->ip2[0]);
>     uint32_t low = ntohl(gr->ip[0]);
>
> Thanks,
> Yaomin
>
>
> On 01/31/10 10:00, Will Metcalf wrote:
>> I think this is a problem somewhere in the cidr masking.  I have seen 
>> this before on PPC so I'm guessing it is an endian issue, I have not 
>> had a chance to look into it any further.  Let me know if you figure 
>> it out ;-)....
>>
>> https://redmine.openinfosecfoundation.org/issues/show/63
>>
>> Regards,
>>
>> Will
>>
>> On Sun, Jan 31, 2010 at 11:23 AM, Yao-Min Chen <Yaomin.Chen at sun.com 
>> <mailto:Yaomin.Chen at sun.com>> wrote:
>>
>>     My unit test SigTestBidirec03 seems to loop forever while
>>     traversing the
>>     hashed list of signature patterns (I verified this by setting break
>>     point and single stepping the run). Any hint on how to check
>>     whether the
>>     hashed list is properly terminated, and without self-looping?
>>
>>     Test SigTestBidirec03                                             :
>>     ^Cdbx: warning: Interrupt ignored but forwarded to child.
>>     t at 1 (l at 1) signal INT (Interrupt) in DetectAddressCmp at line 1392 in
>>     file "detect-engine-address.c"
>>      1392       if (a->flags & ADDRESS_FLAG_ANY && b->flags &
>>     ADDRESS_FLAG_ANY)
>>     (dbx) where
>>     current thread: t at 1
>>     =>[1] DetectAddressCmp(a = <value not available>, b = 0xacc220), line
>>     1392 in "detect-engine-address.c"
>>      [2] IPOnlyCompareFunc(data1 = 0xa87ce8, len1 = <value not
>>     available>,
>>     data2 = 0xacc220, len2 = <value not available>), line 190 in
>>     "detect-engine-iponly.c"
>>      [3] HashListTableLookup(ht = <value not available>, data = 0xacc220,
>>     datalen = <value not available>), line 237 in "util-hashlist.c"
>>      [4] IPOnlyAddSlash16(de_ctx = 0x4c8240, io_ctx = 0x4ca2d0, ht =
>>     0x2b06f8, gr = 0x5e3898, direction = <value not available>, s =
>>     0x5e3780), line 81 in "detect-engine-iponly.c"
>>      [5] IPOnlyAddSignature(de_ctx = 0x4c8240, io_ctx = 0x4ca2d0, s =
>>     0x5e3780), line 475 in "detect-engine-iponly.c"
>>      [6] SigAddressPrepareStage2(de_ctx = 0x4c8240), line 1667 in
>>     "detect.c"
>>      [7] SigGroupBuild(de_ctx = 0x4c8240), line 2797 in "detect.c"
>>      [8] UTHMatchPackets(de_ctx = 0x4c8240, p = <value not available>,
>>     num_packets = -16948332), line 465 in "util-unittest-helper.c"
>>      [9] SigTestBidirec03(), line 1352 in "detect-parse.c"
>>      [10] UtRunTests(regex_arg = <value not available>), line 182 in
>>     "util-unittest.c"
>>      [11] main(argc = <value not available>, argv = <value not
>>     available>),
>>     line 596 in "suricata.c"
>>
>>     Thanks,
>>     Yaomin
>>
>>     _______________________________________________
>>     Oisf-devel mailing list
>>     Oisf-devel at openinfosecfoundation.org
>>     <mailto:Oisf-devel at openinfosecfoundation.org>
>>     http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Oisf-devel mailing list
>> Oisf-devel at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>   
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Oisf-devel mailing list
> Oisf-devel at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>   

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-devel/attachments/20100209/e76d221d/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: detect-engine-iponly.c.diff
Type: text/x-patch
Size: 2695 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-devel/attachments/20100209/e76d221d/attachment.bin>