The optimization introduced in commit
f13c2a8dff2329c6692a80176262ceaaf8a6f74e, causes regressions in
sorting for languages that have digraphs that change sort order, like
cs_CZ which sorts ch between h and i.
My analysis shows the fast-forwarding optimization in STRCOLL advances
through a digraph while possibly stopping in the middle which results
in a subsequent skipping of the digraph and incorrect sorting. The
optimization is incorrect as implemented and because of that I'm
removing it for 2.23, and I will also commit this fix for 2.22 where
it was originally introduced.
This patch reverts the optimization, introduces a new bug-strcoll2.c
regression test that tests both cs_CZ.UTF-8 and da_DK.ISO-8859-1 and
ensures they sort one digraph each correctly. The optimization can't be
applied without regressing this test.
Checked on x86_64, bug-strcoll2.c fails without this patch and passes
after. This will also get a fix on 2.22 which has the same bug.
this is a path that should solve bug 15884. It complains about the performance
of strcoll(). It was found out that the runtime of strcoll() is actually bound
to strlen which is needed for calculating the size of a cache that was
installed to improve the comparison performance.
The idea for this patch was that the cache is only useful in rare cases
(strings of same length and same first-level-chars) and that it would be
better to avoid memory allocation at all. To prove this I wrote a performance
test bench-strcoll.c with test data in benchtests-strcoll.tar.gz. Also
modifications in benchtests/Makefile and localedata/Makefile are necessary to
make it work.
After removing the cache the strcoll method showed the predicted behavior
(getting slightly faster) in all but the test case for hindi word sorting.
This was due the hindi text having much more equal words than the other ones.
For equal strings the performance was worse since all comparison levels were
run through and from the second level on the cache improved the comparison
performance of the original version.
Therefore I added a bytewise test via strcmp iff the first level comparison
found that both strings did match because in this case it is very likely that
equal strings are compared. This solved the problem with the hindi test case
and improved the performance of the others.
Performance comparison:
glibc files -33.77%
vi_VN.UTF-8 -34.12%
en_US.UTF-8 -42.42%
ar_SA.UTF-8 -27.49%
zh_CN.UTF-8 +07.90%
cs_CZ.UTF-8 -29.67%
en_GB.UTF-8 -28.50%
da_DK.UTF-8 -36.57%
pl_PL.UTF-8 -39.31%
fr_FR.UTF-8 -28.57%
pt_PT.UTF-8 -22.82%
el_GR.UTF-8 -26.77%
ru_RU.UTF-8 -35.81%
iw_IL.UTF-8 -35.34%
es_ES.UTF-8 -34.46%
hi_IN.UTF-8 -00.38%
sv_SE.UTF-8 -36.99%
hu_HU.UTF-8 -16.35%
tr_TR.UTF-8 -27.80%
is_IS.UTF-8 -33.24%
it_IT.UTF-8 -24.39%
sr_RS.UTF-8 -37.55%
ja_JP.UTF-8 +02.84%
strcoll is implemented using a cache for indices and weights of
collation sequences in the strings so that subsequent passes do not
have to search through collation data again. For very large string
inputs, the cache size computation could overflow. In such a case,
use the fallback function that does not cache indices and weights of
collation sequences.
Fixes CVE-2012-4412.
strcoll currently falls back to alloca if malloc fails, resulting in a
possible stack overflow. This patch implements sequence traversal and
comparison without caching indices and rules.
Fixes CVE-2012-4424.
2004-03-14 Ulrich Drepper <drepper@redhat.com>
Make the non-_l functions wrappers around the _l functions.
* include/monetary.h: Declare __vstrmon_l.
* include/string.h: Add libc_hidden_proto for __strcoll_l and
__strxfrm_l.
* include/time.h: Define ptime_locale_status. Declare
__strptime_internal.
* include/wchar.h: Add libc_hidden_proto for __wcscoll_l and
__wcsxfrm_l.
* stdlib/strfmon.c: Move the code to strfmon_l.c. Add little
wrapper around __vstrfmon_l.
* stdlib/strfmon_l.c: Add real implementation. Split into new
function __vstrfmon_l to allow calling it from strfmon.
* stdlib/strtod.c: Move real code to strtod_l.c and add wrapper.
* stdlib/strtod_l.c: Add real implementation.
* stdlib/strtof.c: Adjust to changed strtod.c.
* stdlib/strtof_l.c: Include strtod_l.c now.
* stdlib/strtold.c: New file.
* stdlib/strtold_l.c: Removed.
* string/strcoll.c: Move real code to strcoll_l.c: Add wrapper.
* string/strcoll_l.c: Add real implementation.
* string/strxfrm.c: Move real code to strxfrm_l.c: Add wrapper.
* string/strxfrm_l.c: Add real implementation.
* sysdeps/generic/strtol.c: Move real implementation to strtol_l.c.
Add wrappers.
* sysdeps/generic/strtol_l.c: Add real implementation.
* sysdeps/generic/strtold.c: Removed.
* sysdeps/generic/strtold_l.c: New file.
* sysdeps/generic/strtoll_l.c: Include strtol_l.c now. Adjust
#defines.
* sysdeps/generic/strtoul_l.c: Likewise.
* sysdeps/generic/strtoull_l.c: Likewise.
* sysdeps/generic/wcstol_l.c: Likewise.
* sysdeps/generic/wcstoll_l.c: Likewise.
* sysdeps/generic/wcstoul_l.c: Likewise.
* sysdeps/generic/wcstoull_l.c: Likewise.
* sysdeps/ieee754/ldbl-128/strtold.c: Removed.
* sysdeps/ieee754/ldbl-128/strtold_l.c: New file.
* sysdeps/ieee754/ldbl-96/strtold.c: Removed.
* sysdeps/ieee754/ldbl-96/strtold_l.c: New file.
* sysdeps/m68k/strtold.c: Removed.
* sysdeps/m68k/strtold_l.c: New file.
* time/strftime.c: Move real code to strftime_l.c. Add wrapper.
* time/strftime_l.c: Add real implementation.
* time/strptime.c: Move real code to strptime_l.c. Add wrapper.
* time/strptime_l.c: Add real implementation.
* time/wcsftime.c: Simplify since only wrappers are defined in
strftime.c.
* time/wcsftime_l.c: Include strftime_l.c.
* wcsmbs/wcscoll.c: Simplify since the file is not used by wcscoll_l.c
anymore.
* wcsmbs/wcscoll_l.c: Include strcoll_l.c.
* wcsmbs/wcsxfrm.c: Simplify since the file is not used by wcsxfrm_l.c
anymore.
* wcsmbs/wcsxfrm_l.c: Include strxfrm_l.c.
* wcsmbs/wcstod.c: Prepare to include new strtod.c.
* wcsmbs/wcstod_l.c: Include strtod_l.c.
* wcsmbs/wcstof.c: Prepare to include new strtof.c.
* wcsmbs/wcstof_l.c: Include strtof_l.c.
* wcsmbs/wcstold.c: Prepare to include new strtold.c.
* wcsmbs/wcstold_l.c: Include strtold_l.c.
* locale/uselocale.c: Use _NL_CURRENT_LOCALE instead of __libc_tsd_get.
* sysdeps/generic/strcasecmp.c: Optimize a bit. It's better to get
a reference to the current locale and then use the _l functions.
* sysdeps/generic/strncase.c: Likewise.
2001-07-06 Paul Eggert <eggert@twinsun.com>
* manual/argp.texi: Remove ignored LGPL copyright notice; it's
not appropriate for documentation anyway.
* manual/libc-texinfo.sh: "Library General Public License" ->
"Lesser General Public License".
2001-07-06 Andreas Jaeger <aj@suse.de>
* All files under GPL/LGPL version 2: Place under LGPL version
2.1.