diff --git a/packaging/debian/debianrclqt4/control b/packaging/debian/debianrclqt4/control index 9bcf0a0e..8df37ce7 100644 --- a/packaging/debian/debianrclqt4/control +++ b/packaging/debian/debianrclqt4/control @@ -9,15 +9,16 @@ Build-Depends: debhelper (>= 7), libxapian-dev (>= 1.0.15), libx11-dev, libz-dev, - python-dev, + python-all-dev (>= 2.6.6-3~), quilt -Standards-Version: 3.8.3 +Standards-Version: 3.9.3 Package: recoll Architecture: any Depends: ${shlibs:Depends}, ${misc:Depends} -Recommends: aspell, python, xsltproc -Suggests: antiword, catdoc, ghostscript, libimage-exiftool-perl, poppler-utils, unrtf, python-mutagen +Recommends: aspell, python, xdg-utils, xsltproc +Suggests: antiword, catdoc, ghostscript, libimage-exiftool-perl, poppler-utils, + pstotext, python-chm, python-mutagen, unrtf, untex Description: a personal full text search package with a QT GUI The Recoll personal full text search package is based on a very strong backend (Xapian), for which it provides an easy to use and feature-rich diff --git a/packaging/debian/debianrclqt4/copyright b/packaging/debian/debianrclqt4/copyright index 3cb35945..e934b0fd 100644 --- a/packaging/debian/debianrclqt4/copyright +++ b/packaging/debian/debianrclqt4/copyright @@ -1,113 +1,219 @@ -This package was debianized by Jean-Francois Dockes on -Wed, 10 Jan 2007 16:04:13 +0100. +Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ +Upstream-Name: recoll +Upstream-Contact: Jean-Francois Dockes +Source: http://www.lesbonscomptes.com/recoll/ -It was downloaded from http://www.recoll.org +Files: * +Copyright: 2005-2012, Jean-Francois Dockes +License: GPL-2+ -Upstream Author: Jean-Francois Dockes +Files: Binc IMAP project (bincimapmime/*) +Copyright: 2002-2005, Andreas Aardal Hanssen +License: GPL-2+ -Copyright: (C) 2005,2006, Jean-Francois Dockes +Files: filters/rcl* files index/csguess.cpp internfile/htmlparse.cpp +Copyright: 2000-2004, Mikio Hirabayashi +License: GPL-2+ +Files: filters/rclpython +Copyright: J\xfcrgen Hermann, Mike Brown, Christopher Arndt + +License: GPL-2+ + +Files: internfile/htmlparse.cpp mh_html.cpp +Copyright: 1999-2001, BrightStation PLC +Copyright: 2001, Ananova Ltd +Copyright: 2002-2004, Olly Betts +License: GPL-2+ + +Files: unac/* +Copyright: 2000-2002, Loic Dachary +License: GPL-2+ + +Files: common/* +Copyright: 2004-2005, J.F.Dockes +License: GPL-2+ + +Files: debian/* +Copyright: 2007-2012, Kartik Mistry +License: GPL-2+ + +License: GPL-2+ + This package is free software; you can redistribute it and/or modify it under + the terms of the GNU General Public License as published by the Free Software + Foundation; either version 2 of the License, or (at your option) any later + version. + . + This package is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS + FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. + . + You should have received a copy of the GNU General Public License along with + this package; if not, write to the Free Software Foundation, Inc., 51 Franklin + St, Fifth Floor, Boston, MA 02110-1301 USA + . + On Debian systems, the complete text of the GNU General Public License can be + found in `/usr/share/common-licenses/GPL-2' and + `/usr/share/common-licenses/GPL-3'. + +Files: aspell/* +Copyright: 2001-2002, by Kevin Atkinson +License: LGPL-2+ + +Files: images/text-x-python.png +Copyright: David Vignoni +License: LGPL-2+ + +License: LGPL-2+ + This package is free software; you can redistribute it and/or modify it under + the terms of the GNU Lesser General Public License as published by the Free + Software Foundation; either version 2 of the License, or (at your option) any + later version. + . + This package is distributed in the hope that it will be useful, but WITHOUT + ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS + FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more + details. + . + You should have received a copy of the GNU Lesser General Public License along + with this package; if not, write to the Free Software Foundation, Inc., 51 + Franklin St, Fifth Floor, Boston, MA 02110-1301 USA + . + On Debian systems, the complete text of the GNU Lesser General Public License + can be found in `/usr/share/common-licenses/LGPL-2' and + `/usr/share/common-licenses/LGPL-2.1' and `/usr/share/common-licenses/LGPL-3'. + +Files: qtgui/q3richtext_p.h +Copyright: 1992-2007, Trolltech ASA. All rights reserved License: + This file is part of the Qt3Support module of the Qt Toolkit. + . + This file may be used under the terms of the GNU General Public License + version 2.0 as published by the Free Software Foundation and appearing in the + file LICENSE.GPL included in the packaging of this file. Please review the + following information to ensure GNU General Public Licensing requirements will + be met: http://trolltech.com/products/qt/licenses/licensing/opensource/ + . + If you are unsure which license is appropriate for your use, please review the + following information: + http://trolltech.com/products/qt/licenses/licensing/licensingoverview + or contact the sales department at sales@trolltech.com. + . + In addition, as a special exception, Trolltech gives you certain additional + rights. These rights are described in the Trolltech GPL Exception version 1.0, + which can be found at http://www.trolltech.com/products/qt/gplexception/ and + in the file GPL_EXCEPTION.txt in this package. + . + In addition, as a special exception, Trolltech, as the sole copyright + holder for Qt Designer, grants users of the Qt/Eclipse Integration + plug-in the right for the Qt/Eclipse Integration to link to + functionality provided by Qt Designer and its related libraries. + . + Trolltech reserves all rights not expressly granted herein. + . + Trolltech ASA (c) 2007 + . + This file is provided AS IS with NO WARRANTY OF ANY KIND, INCLUDING THE + WARRANTY OF DESIGN, MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. - This package is free software; you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation; either version 2 of the License, or - (at your option) any later version. +Files: common/uproplist.h +Copyright: 1991-2006, Unicode, Inc. +License: + All rights reserved. Distributed under the Terms of Use in + http://www.unicode.org/copyright.html + . + Permission is hereby granted, free of charge, to any person obtaining a copy + of the Unicode data files and any associated documentation (the "Data Files") + or Unicode software and any associated documentation (the "Software") to deal + in the Data Files or Software without restriction, including without + limitation the rights to use, copy, modify, merge, publish, distribute, and/or + sell copies of the Data Files or Software, and to permit persons to whom the + Data Files or Software are furnished to do so, provided that (a) the above + copyright notice(s) and this permission notice appear with all copies of the + Data Files or Software, (b) both the above copyright notice(s) and this + permission notice appear in associated documentation, and (c) there is clear + notice in each modified Data File or in the Software as well as in the + documentation associated with the Data File(s) or Software that the data or + software has been modified. + . + THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY + KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD + PARTY RIGHTS. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN + THIS NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL + DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR + PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS + ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE + DATA FILES OR SOFTWARE. + . + Except as contained in this notice, the name of a copyright holder shall not + be used in advertising or otherwise to promote the sale, use or other dealings + in these Data Files or Software without prior written authorization of the + copyright holder. - This package is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. +Files: utils/md5.* +Copyright: 1991-1992, RSA Data Security, Inc. All rights reserved. +License: + MD5C.C - RSA Data Security, Inc., MD5 message-digest algorithm + . + License to copy and use this software is granted provided that it is + identified as the "RSA Data Security, Inc. MD5 Message-Digest Algorithm" in + all material mentioning or referencing this software or this function. + . + License is also granted to make and use derivative works provided that such + works are identified as "derived from the RSA Data Security, Inc. MD5 + Message-Digest Algorithm" in all material mentioning or referencing the + derived work. + . + RSA Data Security, Inc. makes no representations concerning either the + merchantability of this software or the suitability of this software for any + particular purpose. It is provided "as is" without express or implied warranty + of any kind. + . + These notices must be retained in any copies of any part of this documentation + and/or software. - You should have received a copy of the GNU General Public License - along with this package; if not, write to the Free Software - Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA +Files: desktop/xdg-utils-1.0.1/* +Copyright: 2006, Kevin Krammer , + Jeremy White +License: + Permission is hereby granted, free of charge, to any person obtaining a copy + of this software and associated documentation files (the "Software"), to deal + in the Software without restriction, including without limitation the rights + to use, copy, modify, merge, publish, distribute, sublicense, and/or sell + copies of the Software, and to permit persons to whom the Software is + furnished to do so, subject to the following conditions: + . + The above copyright notice and this permission notice shall be included in all + copies or substantial portions of the Software. + . + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE + AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER + LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, + OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + SOFTWARE. -On Debian systems, the complete text of the GNU General -Public License can be found in `/usr/share/common-licenses/GPL'. - -The Debian packaging is (C) 2007, Jean-Francois Dockes and -is licensed under the GPL, see above. - -Portions of the software are: - -Copyright (C) 2001-2002 by Kevin Atkinson under the GNU LGPL: -/* This file is part of The New Aspell - * Copyright (C) 2001-2002 by Kevin Atkinson under the GNU LGPL - * license version 2.0 or 2.1. You should have received a copy of the - * LGPL license along with this library if you did not you can find it - * at http://www.gnu.org/. - -On Debian systems, the complete text of the GNU LGPL -can be found in `/usr/share/common-licenses/LGPL-2'. - - -Copyright 2002-2005 Andreas Aardal Hanssen -Copyright (C) 2000-2004 Mikio Hirabayashi -Copyright 1999,2000,2001 BrightStation PLC -Copyright 2001 Ananova Ltd -Copyright 2002 Olly Betts -Copyright (C) 2000, 2001, 2002 Loic Dachary - - GPL V2 or later, same license text as above - -Copyright (c) 1991-2004 Unicode, Inc. - - COPYRIGHT AND PERMISSION NOTICE - - Copyright © 1991-2006 Unicode, Inc. All rights reserved. Distributed under - the Terms of Use in http://www.unicode.org/copyright.html. - - Permission is hereby granted, free of charge, to any person obtaining a - copy of the Unicode data files and any associated documentation (the "Data - Files") or Unicode software and any associated documentation (the - "Software") to deal in the Data Files or Software without restriction, - including without limitation the rights to use, copy, modify, merge, - publish, distribute, and/or sell copies of the Data Files or Software, and - to permit persons to whom the Data Files or Software are furnished to do - so, provided that (a) the above copyright notice(s) and this permission - notice appear with all copies of the Data Files or Software, (b) both the - above copyright notice(s) and this permission notice appear in associated - documentation, and (c) there is clear notice in each modified Data File or - in the Software as well as in the documentation associated with the Data - File(s) or Software that the data or software has been modified. - - THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY - KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF - MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF - THIRD PARTY RIGHTS. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS - INCLUDED IN THIS NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR - CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF - USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER - TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR - PERFORMANCE OF THE DATA FILES OR SOFTWARE. - - Except as contained in this notice, the name of a copyright holder shall - not be used in advertising or otherwise to promote the sale, use or other - dealings in these Data Files or Software without prior written - authorization of the copyright holder. - -Copyright (C) 1991-2, RSA Data Security, Inc. Created 1991. All -/* - * MD5C.C - RSA Data Security, Inc., MD5 message-digest algorithm - * - * Copyright (C) 1991-2, RSA Data Security, Inc. Created 1991. All - * rights reserved. - * - * License to copy and use this software is granted provided that it - * is identified as the "RSA Data Security, Inc. MD5 Message-Digest - * Algorithm" in all material mentioning or referencing this software - * or this function. - * - * License is also granted to make and use derivative works provided - * that such works are identified as "derived from the RSA Data - * Security, Inc. MD5 Message-Digest Algorithm" in all material - * mentioning or referencing the derived work. - * - * RSA Data Security, Inc. makes no representations concerning either - * the merchantability of this software or the suitability of this - * software for any particular purpose. It is provided "as is" - * without express or implied warranty of any kind. - * - * These notices must be retained in any copies of any part of this - * documentation and/or software. - */ +Files: index/csguess.cpp +Copyright: 2000-2004, Mikio Hirabayashi +License: LGPL-2.1+ + This file is part of QDBM, Quick Database Manager. + . + QDBM is free software; you can redistribute it and/or modify it under + the terms of the GNU Lesser General Public License as published by the + Free Software Foundation; either version 2.1 of the License or any later + version. + . + QDBM is distributed in the hope that it will be useful, but WITHOUT ANY + WARRANTY; without even the implied warranty of MERCHANTABILITY or + FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public + License for more details. + . + You should have received a copy of the GNU Lesser General Public License + along with QDBM; if not, write to the Free Software Foundation, Inc., + 51 Franklin St, Fifth Floor, Boston, MA 02110-1301, USA. + . + On Debian systems, the complete text of the GNU Lesser General Public License + can be found in `/usr/share/common-licenses/LGPL-2.1' and + `/usr/share/common-licenses/LGPL-3'. diff --git a/packaging/debian/debianrclqt4/docs b/packaging/debian/debianrclqt4/docs deleted file mode 100644 index e845566c..00000000 --- a/packaging/debian/debianrclqt4/docs +++ /dev/null @@ -1 +0,0 @@ -README diff --git a/packaging/debian/debianrclqt4/patches/fix-python-install.patch b/packaging/debian/debianrclqt4/patches/fix-python-install.patch index 49ce3727..4c57ca55 100644 --- a/packaging/debian/debianrclqt4/patches/fix-python-install.patch +++ b/packaging/debian/debianrclqt4/patches/fix-python-install.patch @@ -1,5 +1,8 @@ -setup.py --root and --user options interfer with debian wanting a dist-packages, -not site-packages installation for python modules +Description: Fix python module installation + setup.py --root and --user options interfer with debian wanting a + dist-packages, not site-packages installation for python modules +Author: Jean-Francois Dockes +Last-Update: 2012-03-26 --- a/recollinstall.in +++ b/recollinstall.in @@ -139,5 +139,4 @@ diff --git a/packaging/debian/debianrclqt4/rules b/packaging/debian/debianrclqt4/rules index 3af18c6a..5c8224ed 100755 --- a/packaging/debian/debianrclqt4/rules +++ b/packaging/debian/debianrclqt4/rules @@ -1,7 +1,5 @@ #!/usr/bin/make -f -include /usr/share/quilt/quilt.make - # Uncomment this to turn on verbose mode. #export DH_VERBOSE=1 @@ -28,13 +26,15 @@ config.status: configure --mandir=\$${prefix}/share/man \ --prefix=/usr -build: build-stamp -build-stamp: $(QUILT_STAMPFN) config.status +build: build-arch build-indep +build-arch: build-stamp +build-indep: build-stamp +build-stamp: config.status dh_testdir $(MAKE) touch $@ -clean: unpatch +clean: dh_testdir dh_testroot rm -f build-stamp config.log @@ -46,8 +46,11 @@ install: dh_testroot dh_prep dh_installdirs - $(MAKE) prefix=$(CURDIR)/debian/recoll/usr install - (cd python/recoll;python setup.py install --install-layout=deb --root=$(CURDIR)/debian/recoll/ ) + + $(MAKE) STRIP=ls prefix=$(CURDIR)/debian/tmp/usr install + (cd python/recoll; python setup.py install \ + --install-layout=deb \ + --prefix=$(CURDIR)/debian/tmp/usr ) binary-arch: build install dh_testdir @@ -56,6 +59,7 @@ binary-arch: build install dh_installdocs README dh_installmenu dh_installman + dh_install --sourcedir=debian/tmp dh_link dh_strip dh_compress @@ -69,4 +73,4 @@ binary-arch: build install binary-indep: build install binary: binary-indep binary-arch -.PHONY: build clean binary-indep binary-arch binary install +.PHONY: build build-arch build-indep clean binary-indep binary-arch binary install diff --git a/packaging/debian/debianrclqt4/source/format b/packaging/debian/debianrclqt4/source/format new file mode 100644 index 00000000..163aaf8d --- /dev/null +++ b/packaging/debian/debianrclqt4/source/format @@ -0,0 +1 @@ +3.0 (quilt) diff --git a/packaging/debian/debianrclqt4/watch b/packaging/debian/debianrclqt4/watch index 8b5064a9..e707c144 100644 --- a/packaging/debian/debianrclqt4/watch +++ b/packaging/debian/debianrclqt4/watch @@ -1,12 +1,2 @@ -# Example watch control file for uscan -# Rename this file to "watch" and then you can run the "uscan" command -# to check for upstream updates and more. -# See uscan(1) for format - -# Compulsory line, this is a version 3 file version=3 - -# Uncomment to examine a Webpage -# http://www.recoll.org/download.html recoll-(.*)\.tar\.gz - diff --git a/src/README b/src/README index 59fb4665..8da24bfb 100644 --- a/src/README +++ b/src/README @@ -235,24 +235,37 @@ More documentation can be found in the doc/ directory or at http://www.recoll.or files with different character sets, encodings, and languages into the same index. It has input filters for many document types. - Stemming depends on the document language. Recoll stores the unstemmed - versions of terms and uses auxiliary databases for term expansion. It can - switch stemming languages, or add a language, without re-indexing. Storing - documents in different languages in the same index is possible, and useful - in practice, but does introduce possibilities of confusion. Recoll - currently makes no attempt at automatic language recognition. + Stemming is the process by which Recoll reduces words to their radicals so + that searching does not depend, for example, on a word being singular or + plural (floor, floors), or on a verb tense (flooring, floored). Because + the mechanisms used for stemming depend on the specific grammatical rules + for each language, there is a separate stemmer module for most common + languages where stemming makes sense. Storing documents written in + different languages in the same index is possible, and commonly done. In + this situation, you can specify several stemming languages for the index. + Recoll stores the unstemmed versions of terms in the main index and uses + auxiliary databases for term expansion (one for each stemming language), + which means that you can switch stemming languages between searches, or + add a language without needing a full reindex. Recoll currently makes no + attempt at automatic language recognition, which means that the stemmer + will sometimes be applied to terms from other languages with potentially + strange results. In practise, even if this introduces possibilities of + confusion, this approach has been proven quite useful, and, awaiting the + addition of an automatic language recognition module to Recoll, it is much + less cumbersome than separating your documents according to what language + they are written in. Recoll has many parameters which define exactly what to index, and how to classify and decode the source documents. These are kept in configuration files. A default configuration is copied into a standard location (usually something like /usr/[local/]share/recoll/examples) during installation. - The default parameters from this file may be overridden by values that you - set inside your personal configuration, found by default in the .recoll - sub-directory of your home directory. The default configuration will index - your home directory with default parameters and should be sufficient for - giving Recoll a try, but you may want to adjust it later, which can be - done either by editing the text files or by using configuration menus in - the recoll GUI + The default values set by the configuration files in this directory may be + overridden by values that you set inside your personal configuration, + found by default in the .recoll sub-directory of your home directory. The + default configuration will index your home directory with default + parameters and should be sufficient for giving Recoll a try, but you may + want to adjust it later, which can be done either by editing the text + files or by using configuration menus in the recoll GUI Indexing is started automatically the first time you execute the recoll search graphical user interface, or by executing the recollindex command. diff --git a/src/VERSION b/src/VERSION index 511a76e6..06fb41b6 100644 --- a/src/VERSION +++ b/src/VERSION @@ -1 +1 @@ -1.17.1 +1.17.2 diff --git a/src/common/autoconfig.h.in b/src/common/autoconfig.h.in index 5e99c482..41a7691e 100644 --- a/src/common/autoconfig.h.in +++ b/src/common/autoconfig.h.in @@ -109,7 +109,9 @@ #undef _FILE_OFFSET_BITS #undef _LARGE_FILES -#if _FILE_OFFSET_BITS == 64 || defined(__APPLE__) || defined(__OpenBSD__) +/* The __FreeBSD_kernel__ thing is for debian/kfreebsd. */ +#if _FILE_OFFSET_BITS == 64 || defined(__APPLE__) || defined(__OpenBSD__) ||\ + defined(__FreeBSD_kernel__) #define OFFTPC "%lld" #else #define OFFTPC "%ld" diff --git a/src/common/rclinit.cpp b/src/common/rclinit.cpp index 37391888..1578a9e5 100644 --- a/src/common/rclinit.cpp +++ b/src/common/rclinit.cpp @@ -107,9 +107,15 @@ RclConfig *recollinit(RclInitFlags flags, // Make sure the locale charset is initialized (so that multiple // threads don't try to do it at once). config->getDefCharset(); + // Init unac locking unac_init_mt(); + // Init Unac translation exceptions + string unacex; + if (config->getConfParam("unac_except_trans", unacex) && !unacex.empty()) + unac_set_except_translations(unacex.c_str()); + int flushmb; if (config->getConfParam("idxflushmb", &flushmb) && flushmb > 0) { LOGDEB1(("rclinit: idxflushmb=%d, set XAPIAN_FLUSH_THRESHOLD to 10E6\n", diff --git a/src/common/unacpp.cpp b/src/common/unacpp.cpp index 280e8ef5..11d6775d 100644 --- a/src/common/unacpp.cpp +++ b/src/common/unacpp.cpp @@ -88,6 +88,7 @@ using namespace std; #include "unacpp.h" #include "readfile.h" +#include "rclinit.h" int main(int argc, char **argv) { @@ -98,8 +99,13 @@ int main(int argc, char **argv) } const char *encoding = argv[1]; string ifn = argv[2]; + if (!ifn.compare("stdin")) + ifn.clear(); const char *ofn = argv[3]; + string reason; + (void)recollinit(RCLINIT_NONE, 0, 0, reason, 0); + string odata; if (!file_to_string(ifn, odata)) { cerr << "file_to_string: " << odata << endl; @@ -111,7 +117,12 @@ int main(int argc, char **argv) exit(1); } - int fd = open(ofn, O_CREAT|O_EXCL|O_WRONLY, 0666); + int fd; + if (strcmp(ofn, "stdout")) { + fd = open(ofn, O_CREAT|O_EXCL|O_WRONLY, 0666); + } else { + fd = 1; + } if (fd < 0) { cerr << "Open/Create " << ofn << " failed: " << strerror(errno) << endl; diff --git a/src/configure b/src/configure index e461ab05..959edd29 100755 --- a/src/configure +++ b/src/configure @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.68 for Recoll 1.17.0. +# Generated by GNU Autoconf 2.68 for Recoll 1.17.1. # # # Copyright (C) 1992, 1993, 1994, 1995, 1996, 1998, 1999, 2000, 2001, @@ -557,8 +557,8 @@ MAKEFLAGS= # Identity of this package. PACKAGE_NAME='Recoll' PACKAGE_TARNAME='recoll' -PACKAGE_VERSION='1.17.0' -PACKAGE_STRING='Recoll 1.17.0' +PACKAGE_VERSION='1.17.1' +PACKAGE_STRING='Recoll 1.17.1' PACKAGE_BUGREPORT='' PACKAGE_URL='' @@ -618,6 +618,7 @@ QTGUI QMAKE LIBFAM LIBXAPIANSTATICEXTRA +LIBXAPIANDIR LIBXAPIAN LIBICONV INCICONV @@ -1257,7 +1258,7 @@ if test "$ac_init_help" = "long"; then # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures Recoll 1.17.0 to adapt to many kinds of systems. +\`configure' configures Recoll 1.17.1 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1322,7 +1323,7 @@ fi if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of Recoll 1.17.0:";; + short | recursive ) echo "Configuration of Recoll 1.17.1:";; esac cat <<\_ACEOF @@ -1449,7 +1450,7 @@ fi test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -Recoll configure 1.17.0 +Recoll configure 1.17.1 generated by GNU Autoconf 2.68 Copyright (C) 2010 Free Software Foundation, Inc. @@ -2002,7 +2003,7 @@ cat >config.log <<_ACEOF This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by Recoll $as_me 1.17.0, which was +It was created by Recoll $as_me 1.17.1, which was generated by GNU Autoconf 2.68. Invocation command line was $ $0 $@ @@ -4598,10 +4599,16 @@ done LIBXAPIAN=$tmpxaplib # Also recent xapian libs need lz even when they think they don't... LIBXAPIAN="$LIBXAPIAN -lz" +LIBXAPIANDIR=`$XAPIAN_CONFIG --libs | awk '{print $1}'` +case A"$LIBXAPIANDIR" in + A-L*) LIBXAPIANDIR=`echo $LIBXAPIANDIR | sed -e 's/-L//'`;; + *) LIBXAPIANDIR="";; +esac XAPIANCXXFLAGS=`$XAPIAN_CONFIG --cxxflags` #echo XAPIAN_CONFIG: $XAPIAN_CONFIG #echo LIBXAPIAN: $LIBXAPIAN +#echo LIBXAPIANDIR: $LIBXAPIANDIR #echo LIBXAPIANSTATICEXTRA: $LIBXAPIANSTATICEXTRA #echo XAPIANCXXFLAGS: $XAPIANCXXFLAGS @@ -5644,7 +5651,8 @@ test "X$m_prefix" = "XNONE" && m_prefix=/usr/local m_datadir=${m_prefix}/share QTRECOLL_DATADIR=${m_datadir}/recoll -RCLVERSION='1.17.0' +RCLVERSION='1.17.1' + @@ -6213,7 +6221,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by Recoll $as_me 1.17.0, which was +This file was extended by Recoll $as_me 1.17.1, which was generated by GNU Autoconf 2.68. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -6275,7 +6283,7 @@ _ACEOF cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -Recoll config.status 1.17.0 +Recoll config.status 1.17.1 configured by $0, generated by GNU Autoconf 2.68, with options \\"\$ac_cs_config\\" diff --git a/src/configure.ac b/src/configure.ac index 7fb79821..8bad8920 100644 --- a/src/configure.ac +++ b/src/configure.ac @@ -321,10 +321,16 @@ done LIBXAPIAN=$tmpxaplib # Also recent xapian libs need lz even when they think they don't... LIBXAPIAN="$LIBXAPIAN -lz" +LIBXAPIANDIR=`$XAPIAN_CONFIG --libs | awk '{print $1}'` +case A"$LIBXAPIANDIR" in + A-L*) LIBXAPIANDIR=`echo $LIBXAPIANDIR | sed -e 's/-L//'`;; + *) LIBXAPIANDIR="";; +esac XAPIANCXXFLAGS=`$XAPIAN_CONFIG --cxxflags` #echo XAPIAN_CONFIG: $XAPIAN_CONFIG #echo LIBXAPIAN: $LIBXAPIAN +#echo LIBXAPIANDIR: $LIBXAPIANDIR #echo LIBXAPIANSTATICEXTRA: $LIBXAPIANSTATICEXTRA #echo XAPIANCXXFLAGS: $XAPIANCXXFLAGS @@ -504,6 +510,7 @@ AC_SUBST(X_EXTRA_LIBS) AC_SUBST(INCICONV) AC_SUBST(LIBICONV) AC_SUBST(LIBXAPIAN) +AC_SUBST(LIBXAPIANDIR) AC_SUBST(LIBXAPIANSTATICEXTRA) AC_SUBST(LIBFAM) AC_SUBST(QMAKE) diff --git a/src/doc/user/docbook-xsl.css b/src/doc/user/docbook-xsl.css new file mode 100644 index 00000000..8cc0a8b2 --- /dev/null +++ b/src/doc/user/docbook-xsl.css @@ -0,0 +1,212 @@ +/* + * Copyright (c) 2001, 2003, 2010 The FreeBSD Documentation Project + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD: doc/share/misc/docbook.css,v 1.15 2010/03/20 04:15:01 hrs Exp $ + */ + +body address { + line-height: 1.3; + margin: .6em 0; +} + +body blockquote { + margin-top: .75em; + line-height: 1.5; + margin-bottom: .75em; +} + +html body { + margin: 1em 8% 1em 10%; + line-height: 1.2; +} + +.legalnotice { + font-size: small; + font-variant: small-caps; +} + +body div { + margin: 0; +} + +dl { + margin: .8em 0; + line-height: 1.2; +} + +body form { + margin: .6em 0; +} + +h1, h2, h3, h4, h5, h6, +div.example p b, +.question, +div.table p b, +div.procedure p b { + color: #990000; +} + +body h1, body h2, body h3, body h4, body h5, body h6 { + line-height: 1.3; + margin-left: 0; +} + +body h1, body h2 { + margin: .8em 0 0 -4%; +} + +body h3, body h4 { + margin: .8em 0 0 -3%; +} + +body h5 { + margin: .8em 0 0 -2%; +} + +body h6 { + margin: .8em 0 0 -1%; +} + +body hr { + margin: .6em; + border-width: 0 0 1px 0; + border-style: solid; + border-color: #cecece; +} + +body img.navheader { + margin: 0 0 0 -4%; +} + +ol { + margin: 0 0 0 5%; + line-height: 1.2; +} + +body pre { + margin: .75em 0; + line-height: 1.0; + font-family: monospace; +} + +body td, body th { + line-height: 1.2; +} + +ul, body dir, body menu { + margin: 0 0 0 5%; + line-height: 1.2; +} + +html { + margin: 0; + padding: 0; +} + +body p b.application { + color: #000000; +} +body p span.application { + font-weight: bold; + color: #000000; +} + +.filename { + color: #007a00; +} + +.guimenu, .guimenuitem, .guisubmenu, +.guilabel, .interface, +.shortcut, .shortcut .keycap { + font-weight: bold; +} + +.guibutton { + background-color: #cfcfcf; + padding: 2px; +} + +.accel { + background-color: #f0f0f0; + text-decoration: underline; +} + +.screen { + padding: 1ex; +} + +.programlisting { + padding: 1ex; + background-color: #eee; + border: 1px solid #ccc; +} + +@media screen { /* hide from ie3 */ + a[href]:hover { background: #ffa } +} + +blockquote.note { + color: #222; + background: #eee; + border: 1px solid #ccc; + padding: 0.4em 0.4em; + width: 85%; +} + +blockquote.tip { + color: #004f00; + background: #d8ecd6; + border: 1px solid green; + padding: 0.2em 2em; + width: 85%; +} + +blockquote.important { + font-style:italic; + border: 1px solid #a00; + border-left: 12px solid #c00; + padding: 0.1em 1em; +} + +blockquote.warning { + color: #9f1313; + background: #f8e8e8; + border: 1px solid #e59595; + padding: 0.2em 2em; + width: 85%; +} + +.example { + background: #fefde6; + border: 1px solid #f1bb16; + margin: 1em 0; + padding: 0.2em 2em; + width: 90%; +} + +.informaltable table.calstable tr td { + padding-left: 1em; + padding-right: 1em; +} diff --git a/src/doc/user/usermanual.sgml b/src/doc/user/usermanual.sgml index d5432611..499006b4 100644 --- a/src/doc/user/usermanual.sgml +++ b/src/doc/user/usermanual.sgml @@ -34,6 +34,9 @@ This document introduces full text search notions and describes the installation and use of the &RCL; application. It currently describes &RCL; &RCLVERSION;. + @@ -45,12 +48,12 @@ Giving it a try - If you do not like reading manuals (who does?) and would - like to give &RCL; a try, just perform installation and start the - recoll user interface, which will index your - home directory by default, allowing you to search immediately after - indexing completes. + If you do not like reading manuals (who does?) and would like + to give &RCL; a try, just install the application and + start the recoll graphical user interface (GUI), + which will ask to index your home directory by default, allowing + you to search immediately after indexing completes. Do not do this if your home directory contains a huge number of documents and you do not want to wait or are very @@ -135,45 +138,65 @@ different character sets, encodings, and languages into the same index. It has input filters for many document types. - Stemming depends on the document language. &RCL; stores - the unstemmed versions of terms and uses auxiliary databases for - term expansion. It can switch stemming languages, or add a - language, without re-indexing. Storing documents in different - languages in the same index is possible, and useful in - practice, but does introduce possibilities of confusion. &RCL; - currently makes no attempt at automatic language recognition. + Stemming is the process by which &RCL; reduces words to + their radicals so that searching does not depend, for example, + on a word being singular or plural (floor, floors), or on a verb + tense (flooring, floored). Because the mechanisms used for + stemming depend on the specific grammatical rules for each + language, there is a separate stemmer module for most common + languages where stemming makes sense. Storing documents written + in different languages in the same index is possible, and + commonly done. In this situation, you can specify several + stemming languages for the index. &RCL; stores the unstemmed + versions of terms in the main index and uses auxiliary databases + for term expansion (one for each stemming language), which means + that you can switch stemming languages between searches, or add + a language without needing a full reindex. &RCL; currently + makes no attempt at automatic language recognition, which means + that the stemmer will sometimes be applied to terms from other + languages with potentially strange results. In practise, even if + this introduces possibilities of confusion, this approach has + been proven quite useful, and, awaiting the addition of an + automatic language recognition module to &RCL;, it is much less + cumbersome than separating your documents according to what + language they are written in. &RCL; has many parameters which define exactly what to - index, and how to classify and decode the source documents. These - are kept in configuration - files. A default configuration is copied into a standard - location (usually something like - /usr/[local/]share/recoll/examples) during - installation. The default parameters from this file may be - overridden by values that you set inside your personal - configuration, found by default in the .recoll - sub-directory of your home directory. The default configuration - will index your home directory with default parameters and should - be sufficient for giving &RCL; a try, but you may want to adjust it - later, which can be done either by editing the text files or by - using configuration menus in the recoll - GUI + index, and how to classify and decode the source + documents. These are kept in configuration files. A + default configuration is copied into a standard location + (usually something like + /usr/[local/]share/recoll/examples) + during installation. The default values set by the + configuration files in this directory may be overridden by + values that you set inside your personal configuration, found + by default in the .recoll sub-directory + of your home directory. The default configuration will index + your home directory with default parameters and should be + sufficient for giving &RCL; a try, but you may want to adjust + it later, which can be done either by editing the text files + or by using configuration menus in the + recoll GUI - Indexing - is started automatically the first time you execute the - recoll search graphical user interface, or by - executing the recollindex command. + The indexing + process is started automatically the first time you + execute the recoll GUI. Indexing can also be + performed by executing the recollindex + command. Searches are usually - performed inside the recoll graphical user - interface (GUI) program, which has many options to help you find - what you are looking for. However, there are other ways to perform - &RCL; searches: mostly a - command line tool, a + performed inside the recoll GUI, which has many + options to help you find what you are looking for. However, there + are other ways to perform &RCL; searches: mostly a + command line interface, a Python - programming interface, and a - KDE KIO slave module. + programming interface, a + KDE KIO slave module, and + a Ubuntu Unity Lens module. + @@ -234,25 +257,27 @@ configuration files. Most file types, like HTML or word processing files, only hold - one document. Some file types, like mail folder files or zip + one document. Some file types, like email folders or zip archives, can hold many individually indexed documents, which may in turn be themselves compound ones. Such hierarchies can go quite - deep, and &RCL; has no problem processing, for example, an ms-word - document which would be an attachment to an email message part of - a folder file archived inside a zip file... + deep, and &RCL; can process, for example, an + ms-word + document stored as an attachment to an email message inside an + email folder archived in a zip file... - &RCL; indexing processes plain text, HTML, openoffice - and e-mail files, and a few others internally. + &RCL; indexing processes plain text, HTML, OpenDocument + (Open/LibreOffice), email formats, and a few others internally. Other file types (ie: postscript, pdf, ms-word, rtf ...) need external applications for preprocessing. The list is in the installation section. After every indexing operation, &RCL; updates a list of commands that would be needed for indexing existing files - types. This list can be displayed from the - recoll File menu. It is - stored in the missing text file - inside the configuration directory. + types. This list can be displayed by selecting the menu option + File->Show Missing Helpers + in the recoll GUI. It is stored in the + missing text file inside the configuration + directory. Without further configuration, &RCL; will index all appropriate files from your home directory, with a reasonable @@ -336,9 +361,9 @@ recoll indexed). Of course, images, sound and video do not increase the - index size, which means that it will be quite typical nowadays - (2006), that even a big index will be negligible against the - total amount of data on the computer. + index size, which means that nowadays (2012), typically, even a big + index will be negligible against the total amount of data on the + computer. The index data directory (xapiandb) only contains data that can be completely rebuilt by an index run @@ -439,14 +464,20 @@ recoll option.) The interface is started from the - Preferences menu. It has two main - panels. The first panel allows setting global variables, like - the list of top directories or the list of skipped paths. The - second panel allows setting variables that can be redefined - for subdirectories. This second panel has an initially empty list of - customisation directories, to which you can add. The variables - are then set for the currently selected directory (or at the top - level if the empty line is selected). + Preferences->Indexing + Configuration menu entry. It is divided in three tabs, + Global parameters, Local + parameters, and Beagle web history, + which is explained in the next section. + + The first tab allows setting global variables, like the lists + of top directories, skipped paths, or stemming languages. + + The second tab allows setting variables that can be redefined + for subdirectories. This second tab has an initially empty list of + customisation directories, to which you can add. The variables are + then set for the currently selected directory (or at the top level + if the empty line is selected). The meaning for most entries in the interface is self-evident and documented by a ToolTip @@ -521,15 +552,17 @@ recoll if canceled). The recollindex indexing process can be - interrupted by sending an interrupt (^C, SIGINT) or terminate + interrupted by sending an interrupt (Ctrl-C, SIGINT) or terminate (SIGTERM) signal. Some time may elapse before the process exits, - because it needs to properly flush and close the index. The - indexing thread can be equivalently stopped from the menu. + because it needs to properly flush and close the index. This can + also be done from the recoll GUI + File->Stop Indexing + menu entry. After such an interruption, the index will be somewhat inconsistent because some operations which are normally performed at the end of the indexing pass will have been skipped (for - exemple, the stemming and spelling databases will be inexistant + example, the stemming and spelling databases will be inexistant or out of date). You just need to restart indexing at a later time to restore consistency. The indexing will restart at the interruption point (the full file tree will be traversed, @@ -576,7 +609,8 @@ recoll As of version 1.17 the &RCL; GUI has dialogs to manage crontab entries for recollindex. You can reach them from the - Preferences->Indexing Schedule menu. They only + Preferences->Indexing + Schedule menu. They only work with the good old cron, and do not give access to all features of cron scheduling. @@ -652,12 +686,13 @@ fvwm on the log level. When building &RCL;, the real time indexing support can be - customised during package - configuration - with the --with[out]-fam or + customised during package configuration with the + --with[out]-fam or --with[out]-inotify options. The default is - currently to include inotify monitoring on systems that support - it, and, as of recoll 1.17, gamin support on FreeBSD. + currently to include inotify monitoring + on systems that support it, and, as of recoll 1.17, + gamin support on FreeBSD. While it is convenient that data is indexed in real time, repeated indexing can generate a significant load on the @@ -712,7 +747,7 @@ fvwm In most cases, you can enter the terms as you think them, even if they contain embedded punctuation or other non-textual characters. For - exemple, &RCL; can handle things like e-mail addresses, or + example, &RCL; can handle things like email addresses, or arbitrary cut and paste from another text window, punctation and all. @@ -950,7 +985,7 @@ fvwm that you can't actually visualize the folder (there will be an error dialog if you try). &RCL; is unfortunately not yet smart enough to disable the entry in this case. In other cases, the - Open option makes sense, for exemple to + Open option makes sense, for example to start a chm viewer on the parent document for a help page. @@ -1006,7 +1041,7 @@ fvwm create a new preview window. The old one stays open until you close it. - You can close a preview tab by typing ^W + You can close a preview tab by typing Ctrl-W (Ctrl + W) in the window. Closing the last tab for a window will also close the window. @@ -1030,7 +1065,7 @@ fvwm F3 inside the text area to get to the next occurrence. - If you have a search string entered and you use ^Up/^Down + If you have a search string entered and you use Ctrl-Up/Ctrl-Down to browse the results, the search is initiated for each successive document. If the string is found, the cursor will be positioned at the first occurrence of the search string. @@ -1042,8 +1077,8 @@ fvwm the main text but in one of the fields. You can print the current preview window contents by typing - ^P (Ctrl + P) in - the window text. + Ctrl-P (Ctrl + + P) in the window text. @@ -1539,19 +1574,19 @@ fvwm Closing previews - Entering ^W in a tab will + Entering Ctrl-W in a tab will close it (and, for the last tab, close the preview window). Entering Esc will close the preview window and all its tabs. Printing previews - Entering ^P in a preview window will print + Entering Ctrl-P in a preview window will print the currently displayed text. Quitting - Entering ^Q almost anywhere will + Entering Ctrl-Q almost anywhere will close the application. @@ -1588,9 +1623,10 @@ fvwm on startup. The default value is empty, but there is a skeleton style sheet (recoll.qss) inside the /usr/share/recoll/examples - directory. Using a style sheet, you can change most Recoll - graphical parameters: colors, fonts, etc. See the sample - file for a few simple examples. + directory. Using a style sheet, you can change most + recoll graphical parameters: colors, + fonts, etc. See the sample file for a few simple + examples. Maximum text size highlighted for @@ -1830,7 +1866,7 @@ fvwm No more detail will be given about the header part (only useful with the WebKit build), if there are restrictions to what you can do, they are beyond this author's HTML/CSS/Javascript - abilities... There are a few exemples on the + abilities... There are a few examples on the page about customising the result list on the &RCL; web site. @@ -2126,7 +2162,7 @@ text/html [file:///Users/uncrypted-dockes/projets/bateaux/ilur/factEtCie/r potatoes (in any part of the document). An element is composed of an optional field specification, - and a value, separated by a colon. Exemple: + and a value, separated by a colon. Example: Beatles, author:balzac, dc:title:grandet @@ -2163,7 +2199,7 @@ text/html [file:///Users/uncrypted-dockes/projets/bateaux/ilur/factEtCie/r title:prejudice title:pride, and is unlikely to find a result. - Modifiers can be set on a phrase clause, for exemple to specify + Modifiers can be set on a phrase clause, for example to specify a proximity search (unordered). See the modifier section. @@ -2209,7 +2245,7 @@ text/html [file:///Users/uncrypted-dockes/projets/bateaux/ilur/factEtCie/r size for filtering the - results on file size. Exemple: + results on file size. Example: size<10000. You can use <, > or = as operators. You can specify a range like the @@ -2233,7 +2269,7 @@ text/html [file:///Users/uncrypted-dockes/projets/bateaux/ilur/factEtCie/r The days and months parts may be missing. If the / is present but an element is missing, the missing element is interpreted as the lowest or highest date in the - index. Exemples: + index. Examples: 2001-03-01/2002-05-01 the basic syntax for an interval of dates. @@ -2555,7 +2591,7 @@ text/html [file:///Users/uncrypted-dockes/projets/bateaux/ilur/factEtCie/r Subject: for email) when indexing. This is not essential. - You should look to one of the simple filters, for exemple + You should look to one of the simple filters, for example rclps for a starting point. Don't forget to make your filter executable before @@ -3087,7 +3123,7 @@ while query.next >= 0 and query.next < nres: You will only have to check or install supporting applications for the file types that you want to index beyond those that are - natively processed by &RCL; (text, HTML, mail files, and a few + natively processed by &RCL; (text, HTML, email files, and a few others). You should also maybe have a look at the @@ -3259,13 +3295,13 @@ while query.next >= 0 and query.next < nres: Konqueror webarchive format with Python (uses the Tarfile module). - mimehtml web archive format (support based on the mail + mimehtml web archive format (support based on the email filter, which introduces some mild weirdness, but still usable). - Text, HTML, mail folders, and Scribus files are + Text, HTML, email folders, and Scribus files are processed internally. Lyx is used to index Lyx files. Many filters need iconv and the standard sed and awk. @@ -3557,6 +3593,28 @@ while query.next >= 0 and query.next < nres: List elements with embedded spaces can be quoted using double-quotes. + Encoding issues + Most of the configuration parameters are plain ASCII. Two + particular sets of values may cause encoding issues: + + + File path parameters may contain non-ascii + characters and should use the exact same byte values as found in + the file system directory. Usually, this means that the + configuration file should use the system default locale + encoding. + + The unac_except_trans parameter + should be encoded in UTF-8. If your system locale is not UTF-8, and + you need to also specify non-ascii file paths, this poses a + difficulty because common text editors cannot handle multiple + encodings in a single file. In this relatively unlikely case, you + can edit the configuration file as two separate text files with + appropriate encodings, and concatenate them to create the complete + configuration. + + + Main configuration file @@ -3611,7 +3669,7 @@ skippedNames = #* bin CVS Cache cache* caughtspam tmp .thumbnails .svn \ The list in the default configuration does not exclude hidden directories (names beginning with a dot), which means that it may index quite a few things - that you do not want. On the other hand, mail user + that you do not want. On the other hand, email user agents like thunderbird usually store messages in hidden directories, and you probably want this indexed. One possible solution is to @@ -3816,9 +3874,36 @@ skippedPaths = ~/somedir/∗.txt + unac_except_trans + This is a list of characters, encoded in UTF-8, + which should be handled specially when converting text to + unaccented lowercase. For example, in Swedish, the letter + a with diaeresis has full alphabet + citizenship and should not be turned into an + a. Each element in the space-separated list + has the special character as first element and the translation + following. The handling of both the lowercase and upper-case + versions of a character should be specified, as appartenance to + the list will turn-off both standard accent and case + processing. Example for Swedish: + +unac_except_trans = åå Åå ää Ää öö Öö + + + Note that the translation is not limited to a single + character, you could very well have something like + üue in the list. + + This parameter can't be defined for subdirectories, it + is global, because there is no way to do otherwise when + querying. If you have document sets which would need different + values, you will have to index and query them separately. + + + maildefcharset This can be used to define the default - character set specifically for mail messages which don't + character set specifically for email messages which don't specify it. This is mainly useful for readpst (libpst) dumps, which are utf-8 but do not say so. @@ -4081,9 +4166,9 @@ mondelaypatterns = *.log:20 "this one has spaces*:10" filter-specific sections Some filters may need specific - configuration for handling fields. Only the mail message filter + configuration for handling fields. Only the email message filter currently has such a section (named - [mail]). It allows indexing arbitrary mail + [mail]). It allows indexing arbitrary email headers in addition to the ones indexed by default. Other such sections may appear in the future. @@ -4093,9 +4178,9 @@ mondelaypatterns = *.log:20 "this one has spaces*:10" Here follows a small example of a personal fields - file. This would extract a specific mail header and + file. This would extract a specific email header and use it as a searchable field, with data displayable inside result - lists. (Side note: as the mail filter does no decoding on the values, + lists. (Side note: as the email filter does no decoding on the values, only plain ascii headers can be indexed, and only the first occurrence will be used for headers that occur several times). diff --git a/src/doc/user/xmlmake.sh b/src/doc/user/xmlmake.sh new file mode 100644 index 00000000..0cc228b6 --- /dev/null +++ b/src/doc/user/xmlmake.sh @@ -0,0 +1,49 @@ +#!/bin/sh + +# A script to produce the Recoll manual with an xml toolchain. +# Limitations: +# - Does not produce the links to the whole/chunked versions at the top +# of the document +# - The anchor names from the source text are converted to uppercase by +# the sgml toolchain. This does not happen with the xml toolchain, +# which means that external links like +# usermanual.html#RCL.CONFIG.INDEXING won't work because fragments are +# case-sensitive. This could be solved by converting all ids inside the +# source file to upper-case. + +# Wherever docbook.xsl and chunk.xsl live +XSLDIR="/usr/local/share/xsl/docbook/" + +dochunky=1 +test $# -eq 1 && dochunky=0 + +# Remove the SGML header and uncomment the XML one +sed -e '\!//FreeBSD//DTD!d' \ + -e '\!DTD DocBook XML!s///' \ + < usermanual.sgml > usermanual.xml + +# Options common to the single-file and chunked versions +commonoptions="--stringparam section.autolabel 1 \ + --stringparam section.autolabel.max.depth 3 \ + --stringparam section.label.includes.component.label 1 \ + --stringparam autotoc.label.in.hyperlink 0 \ + --stringparam abstract.notitle.enabled 1 \ + --stringparam html.stylesheet docbook-xsl.css \ + --stringparam generate.toc \"book toc,title,figure,table,example,equation\" \ +" + +# Do the chunky thing +test $dochunky -ne 0 && eval xsltproc $commonoptions \ + --stringparam use.id.as.filename 1 \ + "$XSLDIR/html/chunk.xsl" \ + usermanual.xml + +# Produce the single file version +eval xsltproc $commonoptions \ + -o usermanual-xml.html \ + "$XSLDIR/html/docbook.xsl" \ + usermanual.xml + +tidy -indent usermanual-xml.html > tmpfile +mv -f tmpfile usermanual-xml.html diff --git a/src/filters/rclchm b/src/filters/rclchm index 6be74113..b78a1cd4 100755 --- a/src/filters/rclchm +++ b/src/filters/rclchm @@ -2,11 +2,20 @@ """Extract Html files from a Microsoft Compiled Html Help file (.chm) Needs at least python 2.2 for HTMLParser (chmlib needs 2.2 too)""" +# Do we return individual chapters as html pages or concatenate everything? +rclchm_catenate = 0 +# Use special html type to allow for mimeconf/mimeview Open magic, +# Or go the regular html way with text/html +#rclchm_html_mtype = "text/x-chm-html" +rclchm_html_mtype = "text/html" + import sys import os import posixpath import urlparse import urllib +if rclchm_catenate: + import subprocess import rclexecm @@ -171,7 +180,11 @@ class rclCHM: self.tp = ChmTopicsParser(em) self.currentindex = 0 self.em = em - + if rclchm_catenate: + self.em.setmimetype("text/plain") + else: + self.em.setmimetype(rclchm_html_mtype) + def extractone(self, path): """Extract one path-named internal file from the chm file""" @@ -188,10 +201,27 @@ class rclCHM: res, doc = self.chm.RetrieveObject(ui) #self.em.rclog("extract: RetrieveObject: %d [%s]" % (res, doc)) if res > 0: - self.em.setmimetype("text/html") + self.em.setmimetype(rclchm_html_mtype) return (True, doc, path, iseof) return (False, "", path, iseof) - + + def dumpall(self): + alltxt="" + for pth in self.tp.contents: + ret,doc,path,iseof = self.extractone(pth) + if not ret: + continue + # Feed doc to lynx + process = subprocess.Popen(["lynx", "-stdin", "-dump", "-nolist", + "-display_charset=utf8", + "-force_html"], + stdin=subprocess.PIPE, + stdout=subprocess.PIPE + ) + txt,err = process.communicate(doc) + alltxt += txt + return alltxt + def openfile(self, params): """Open the chm file and build the contents list by extracting and parsing the Topics object""" @@ -239,6 +269,13 @@ class rclCHM: return self.extractone(params["ipath:"]) def getnext(self, params): + if rclchm_catenate: + alltxt = self.dumpall() + if alltxt: + return (True, alltxt, "", rclexecm.RclExecM.eofnext) + else: + return (False, "", "", rclexecm.RclExecM.eofnow) + if self.currentindex >= len(self.tp.contents): return (False, "", "", rclexecm.RclExecM.eofnow) else: diff --git a/src/filters/rcldia b/src/filters/rcldia new file mode 100755 index 00000000..185a4dc3 --- /dev/null +++ b/src/filters/rcldia @@ -0,0 +1,124 @@ +#!/usr/bin/env python +# -*- coding: utf-8 -*- +# dia (http://live.gnome.org/Dia) file filter for recoll +# stefan.friedel@iwr.uni-heidelberg.de 2012 +# +# add the following to ~/.recoll/mimeconf into the [index] section: +# application/x-dia-diagram = execm rcldia;mimetype=text/html;charset=utf-8 +# and into the [icons] section: +# application/x-dia-diagram = drawing +# and finally under [categories]: +# other = ...\ +# application/x-dia-diagram +# +# in ~/.recoll/mimemap: +# .dia = application/x-dia-diagram + +# Small fixes from jfd: dia files are sometimes not compressed. +# And a note: this file actually has no reason to return HTML as there is +# no metadata. We could just as well and more simply return text/plain +import rclexecm +import re +from gzip import GzipFile +import xml.parsers.expat + +# some regexps to parse/format the xml data: delete #/spaces at the b/eol and +# ignore empty lines +rhs = re.compile(r'^[#|\s+](.*)') +rhe = re.compile(r'(.*)[#|\s+]$') +rempty = re.compile(r'^#?\s*#?$') + +htmltemplate = ''' + + + + +{0} + + +''' + +# xml parser for dia xml file +class Parser: + def __init__(self,rclem): + self._parser = xml.parsers.expat.ParserCreate(encoding='UTF-8') + self._parser.StartElementHandler = self.startelement + self._parser.EndElementHandler = self.endelement + self._parser.CharacterDataHandler = self.chardata + self.string = [] + self.handlethis = False + self.rclem = rclem + + def startelement(self, name, attrs): + if name == 'dia:string': + self.handlethis = True + else: + self.handlethis = False + + def chardata(self,data): + if self.handlethis: + # check if line is not empty and replace hashes/spaces + # tricky: after htmlescape check also for umlauts + if not rempty.search(data): + self.string.append(self.rclem.htmlescape( + rhe.sub(r'\1',rhs.sub(r'\1',data))).encode('ascii', 'xmlcharrefreplace')) + + def endelement(self,name): + self.handlethis = False + + def feed(self, fh): + self._parser.ParseFile(fh) + del self._parser + +class DiaExtractor: + def __init__(self, em): + self.em = em + + def extractdia(self): + docdata = "" + ipath = "" + try: + docdata = self.ExtractDiaText() + ok = True + except Exception, err: + ok = False + iseof = rclexecm.RclExecM.eofnext + self.em.setmimetype("text/html") + return (ok, docdata, ipath, iseof) + + ###### File type handler api, used by rclexecm ----------> + def openfile(self, params): + try: + self.dia = GzipFile(params["filename:"], 'r') + # Dial files are sometimes not compressed. Quite weirdly, + # GzipFile does not complain until we try to read. Have to do it + # here to be able to retry an uncompressed open. + data = self.dia.readline() + self.dia.seek(0) + return True + except: + # File not compressed ? + try: + self.dia = open(params["filename:"], 'r') + except: + return False + return True + + def getipath(self, params): + ok, data, ipath, eof = self.extractdia() + return (ok, data, ipath, eof) + + def getnext(self, params): + ok, data, ipath, eof = self.extractdia() + return (ok, data, ipath, eof) + + ###### read data + def ExtractDiaText(self): + diap = Parser(self.em) + diap.feed(self.dia) + return htmltemplate.format('\n'.join(diap.string)) + +# Main program: create protocol handler and extractor and run them +proto = rclexecm.RclExecM() +extract = DiaExtractor(proto) +rclexecm.main(proto, extract) diff --git a/src/filters/rclexecm.py b/src/filters/rclexecm.py index 2e8fc7df..821dcff0 100644 --- a/src/filters/rclexecm.py +++ b/src/filters/rclexecm.py @@ -24,8 +24,11 @@ class RclExecM: sys.exit(exitvalue) def htmlescape(self, txt): - txt = txt.replace("<", "<") + # This must stay first (it somehow had managed to skip after + # the next line, with rather interesting results) txt = txt.replace("&", "&") + + txt = txt.replace("<", "<") txt = txt.replace('"', "&dquot;") return txt diff --git a/src/index/Makefile b/src/index/Makefile index 7c24c36b..04c983d9 100644 --- a/src/index/Makefile +++ b/src/index/Makefile @@ -28,14 +28,6 @@ subtreelist : $(SUBTREELIST_OBJS) subtreelist.o : subtreelist.cpp $(CXX) $(ALL_CXXFLAGS) -DTEST_SUBTREELIST -c subtreelist.cpp -CSGUESS_OBJS= trcsguess.o $(BIGLIB) -csguess : $(CSGUESS_OBJS) - $(CXX) $(ALL_CXXFLAGS) -o csguess $(CSGUESS_OBJS) \ - $(LIBICONV) -trcsguess.o : csguess.cpp - $(CXX) $(ALL_CXXFLAGS) -DTEST_CSGUESS -c -o trcsguess.o \ - csguess.cpp - MIMETYPE_OBJS= trmimetype.o $(BIGLIB) mimetype : $(MIMETYPE_OBJS) $(CXX) $(ALL_CXXFLAGS) -o mimetype $(MIMETYPE_OBJS) \ diff --git a/src/index/csguess.cpp b/src/index/csguess.cpp deleted file mode 100644 index 28da1ed1..00000000 --- a/src/index/csguess.cpp +++ /dev/null @@ -1,215 +0,0 @@ -/* Copyright (C) 2004 J.F.Dockes - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the - * Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. - */ - -#ifndef TEST_CSGUESS - -// This code was converted from estraier / qdbm / myconf.c: - -/************************************************************************** - * Copyright (C) 2000-2004 Mikio Hirabayashi - * - * This file is part of QDBM, Quick Database Manager. - * - * QDBM is free software; you can redistribute it and/or modify it under the - * terms of the GNU Lesser General Public License as published by the Free - * Software Foundation; either version 2.1 of the License or any later - * version. QDBM is distributed in the hope that it will be useful, but - * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY - * or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public - * License for more details. You should have received a copy of the GNU - * Lesser General Public License along with QDBM; if not, write to the Free - * Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA - * 02111-1307 USA. - * *********************************************************/ - -#include -#include -#include - -#ifndef NO_NAMESPACES -using std::string; -#endif /* NO_NAMESPACES */ - -#include - -#include "csguess.h" -#include "autoconfig.h" -#ifdef RCL_ICONV_INBUF_CONST -#define ICV_P2_TYPE const char** -#else -#define ICV_P2_TYPE char** -#endif - -// The values from estraier were 32768, 256, 0.001 -const int ICONVCHECKSIZ = 32768; -const int ICONVMISSMAX = 256; -const double ICONVALLWRAT = 0.001; - -// Try to transcode and count errors (for charset guessing) -static int transcodeErrCnt(const char *ptr, int size, - const char *icode, const char *ocode) -{ - iconv_t ic; - char obuf[2*ICONVCHECKSIZ], *wp, *rp; - size_t isiz, osiz; - int miss; - isiz = size; - if((ic = iconv_open(ocode, icode)) == (iconv_t)-1) - return size; - miss = 0; - rp = (char *)ptr; - while(isiz > 0){ - osiz = 2*ICONVCHECKSIZ; - wp = obuf; - if(iconv(ic, (ICV_P2_TYPE)&rp, &isiz, &wp, &osiz) == (size_t)-1){ - if(errno == EILSEQ || errno == EINVAL){ - rp++; - isiz--; - miss++; - if(miss >= ICONVMISSMAX) - break; - } else { - miss = size; - break; - } - } - } - if(iconv_close(ic) == -1) - return size; - return miss; -} - -// Try to guess character encoding. This could be optimized quite a -// lot by avoiding the multiple passes on the document, to be done -// after usefulness is demonstrated... -string csguess(const string &in, const string &dflt) -{ - const char *hypo; - int i, miss; - const char *text = in.c_str(); - bool cr = false; - - int size = in.length(); - if (size > ICONVCHECKSIZ) - size = ICONVCHECKSIZ; - - // UTF-16 with normal prefix ? - if (size >= 2 && (!memcmp(text, "\xfe\xff", 2) || - !memcmp(text, "\xff\xfe", 2))) - return "UTF-16"; - - // If we find a zero at an appropriate position, guess it's UTF-16 - // anyway. This is a quite expensive test for other texts as we'll - // have to scan the whole thing. - for (i = 0; i < size - 1; i += 2) { - if (text[i] == 0 && text[i + 1] != 0) - return "UTF-16BE"; - if (text[i + 1] == 0 && text[i] != 0) - return "UTF-16LE"; - } - - // Look for iso-2022 (rfc1468) specific escape sequences. As - // iso-2022 begins in ascii, and typically soon escapes, these - // succeed fast for a japanese text, but are quite expensive for - // any other - for (i = 0; i < size - 3; i++) { - if (text[i] == 0x1b) { - i++; - if (text[i] == '(' && strchr("BJHI", text[i + 1])) - return "ISO-2022-JP"; - if (text[i] == '$' && strchr("@B(", text[i + 1])) - return "ISO-2022-JP"; - } - } - - // Try conversions from ascii and utf-8. These are unlikely to succeed - // by mistake. - if (transcodeErrCnt(text, size, "US-ASCII", "UTF-16BE") < 1) - return "US-ASCII"; - - if (transcodeErrCnt(text, size, "UTF-8", "UTF-16BE") < 1) - return "UTF-8"; - - hypo = 0; - for (i = 0; i < size; i++) { - if (text[i] == 0xd) { - cr = true; - break; - } - } - - if (cr) { - if ((miss = transcodeErrCnt(text, size, "Shift_JIS", "EUC-JP")) < 1) - return "Shift_JIS"; - if (!hypo && miss / (double)size <= ICONVALLWRAT) - hypo = "Shift_JIS"; - if ((miss = transcodeErrCnt(text, size, "EUC-JP", "UTF-16BE")) < 1) - return "EUC-JP"; - if (!hypo && miss / (double)size <= ICONVALLWRAT) - hypo = "EUC-JP"; - } else { - if ((miss = transcodeErrCnt(text, size, "EUC-JP", "UTF-16BE")) < 1) - return "EUC-JP"; - if (!hypo && miss / (double)size <= ICONVALLWRAT) - hypo = "EUC-JP"; - if ((miss = transcodeErrCnt(text, size, "Shift_JIS", "EUC-JP")) < 1) - return "Shift_JIS"; - if (!hypo && miss / (double)size <= ICONVALLWRAT) - hypo = "Shift_JIS"; - } - if ((miss = transcodeErrCnt(text, size, "UTF-8", "UTF-16BE")) < 1) - return "UTF-8"; - if (!hypo && miss / (double)size <= ICONVALLWRAT) - hypo = "UTF-8"; - if ((miss = transcodeErrCnt(text, size, "CP932", "UTF-16BE")) < 1) - return "CP932"; - if (!hypo && miss / (double)size <= ICONVALLWRAT) - hypo = "CP932"; - - return hypo ? hypo : dflt; -} - -#else - -#include - -#include -#include -#include - -using namespace std; - -#include "readfile.h" -#include "csguess.h" - -int main(int argc, char **argv) -{ - if (argc != 2) { - cerr << "Usage: trcsguess " << endl; - exit(1); - } - const string filename = argv[1]; - const string dflt = argv[2]; - string text; - if (!file_to_string(filename, text)) { - cerr << "Couldnt read file, errno " << errno << endl; - exit(1); - } - cout << csguess(text, dflt) << endl; - exit(0); -} -#endif diff --git a/src/index/csguess.h b/src/index/csguess.h deleted file mode 100644 index 34176b98..00000000 --- a/src/index/csguess.h +++ /dev/null @@ -1,28 +0,0 @@ -/* Copyright (C) 2004 J.F.Dockes - * This program is free software; you can redistribute it and/or modify - * it under the terms of the GNU General Public License as published by - * the Free Software Foundation; either version 2 of the License, or - * (at your option) any later version. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * GNU General Public License for more details. - * - * You should have received a copy of the GNU General Public License - * along with this program; if not, write to the - * Free Software Foundation, Inc., - * 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. - */ -#ifndef _CSGUESS_H_INCLUDED_ -#define _CSGUESS_H_INCLUDED_ - -#include - - -// Try to guess the character set. This might guess unicode encodings, and -// some asian charsets, but has no chance, for example, of discriminating -// betweeen the different iso8859-xx charsets. -extern std::string csguess(const std::string &in, const std::string &dflt); - -#endif /* _CSGUESS_H_INCLUDED_ */ diff --git a/src/index/fsindexer.cpp b/src/index/fsindexer.cpp index 9cd4715f..3c4e7519 100644 --- a/src/index/fsindexer.cpp +++ b/src/index/fsindexer.cpp @@ -39,7 +39,6 @@ #include "readfile.h" #include "indexer.h" #include "fsindexer.h" -#include "csguess.h" #include "transcode.h" #include "debuglog.h" #include "internfile.h" diff --git a/src/internfile/mh_html.cpp b/src/internfile/mh_html.cpp index e8e07d05..31615397 100644 --- a/src/internfile/mh_html.cpp +++ b/src/internfile/mh_html.cpp @@ -18,7 +18,6 @@ #include "cstr.h" #include "mimehandler.h" #include "debuglog.h" -#include "csguess.h" #include "readfile.h" #include "transcode.h" #include "mimeparse.h" diff --git a/src/internfile/mh_text.cpp b/src/internfile/mh_text.cpp index a9b7a203..03a2461b 100644 --- a/src/internfile/mh_text.cpp +++ b/src/internfile/mh_text.cpp @@ -29,7 +29,6 @@ using namespace std; #include "cstr.h" #include "mh_text.h" -#include "csguess.h" #include "debuglog.h" #include "readfile.h" #include "md5.h" diff --git a/src/lib/Makefile b/src/lib/Makefile index efb07466..6d831a67 100644 --- a/src/lib/Makefile +++ b/src/lib/Makefile @@ -6,15 +6,15 @@ LIBS = librcl.a all: $(LIBS) -OBJS = rclaspell.o beaglequeuecache.o cstr.o rclconfig.o rclinit.o textsplit.o unacpp.o beaglequeue.o csguess.o fsindexer.o indexer.o mimetype.o subtreelist.o htmlparse.o myhtmlparse.o mimehandler.o internfile.o mh_exec.o mh_execm.o mh_html.o mh_mail.o mh_mbox.o mh_text.o txtdcode.o docseq.o docseqdb.o docseqhist.o filtseq.o dynconf.o plaintorich.o recollq.o reslistpager.o sortseq.o wasastringtoquery.o wasatorcl.o rcldb.o rcldoc.o rclquery.o searchdata.o stemdb.o stoplist.o base64.o circache.o closefrom.o conftree.o copyfile.o debuglog.o ecrontab.o execmd.o fstreewalk.o idfile.o fileudi.o md5.o mimeparse.o netcon.o pathut.o pxattr.o rclionice.o readfile.o smallut.o transcode.o wipedir.o x11mon.o mime-getpart.o mime-parsefull.o mime-parseonlyheader.o mime-printbody.o mime-printdoc.o mime-printheader.o mime.o convert.o iodevice.o iofactory.o -DEPS = rclaspell.dep.stamp beaglequeuecache.dep.stamp cstr.dep.stamp rclconfig.dep.stamp rclinit.dep.stamp textsplit.dep.stamp unacpp.dep.stamp beaglequeue.dep.stamp csguess.dep.stamp fsindexer.dep.stamp indexer.dep.stamp mimetype.dep.stamp subtreelist.dep.stamp htmlparse.dep.stamp myhtmlparse.dep.stamp mimehandler.dep.stamp internfile.dep.stamp mh_exec.dep.stamp mh_execm.dep.stamp mh_html.dep.stamp mh_mail.dep.stamp mh_mbox.dep.stamp mh_text.dep.stamp txtdcode.dep.stamp docseq.dep.stamp docseqdb.dep.stamp docseqhist.dep.stamp filtseq.dep.stamp dynconf.dep.stamp plaintorich.dep.stamp recollq.dep.stamp reslistpager.dep.stamp sortseq.dep.stamp wasastringtoquery.dep.stamp wasatorcl.dep.stamp rcldb.dep.stamp rcldoc.dep.stamp rclquery.dep.stamp searchdata.dep.stamp stemdb.dep.stamp stoplist.dep.stamp base64.dep.stamp circache.dep.stamp closefrom.dep.stamp conftree.dep.stamp copyfile.dep.stamp debuglog.dep.stamp ecrontab.dep.stamp execmd.dep.stamp fstreewalk.dep.stamp idfile.dep.stamp fileudi.dep.stamp md5.dep.stamp mimeparse.dep.stamp netcon.dep.stamp pathut.dep.stamp pxattr.dep.stamp rclionice.dep.stamp readfile.dep.stamp smallut.dep.stamp transcode.dep.stamp wipedir.dep.stamp x11mon.dep.stamp mime-getpart.dep.stamp mime-parsefull.dep.stamp mime-parseonlyheader.dep.stamp mime-printbody.dep.stamp mime-printdoc.dep.stamp mime-printheader.dep.stamp mime.dep.stamp convert.dep.stamp iodevice.dep.stamp iofactory.dep.stamp +OBJS = rclaspell.o beaglequeuecache.o cstr.o rclconfig.o rclinit.o textsplit.o unacpp.o beaglequeue.o fsindexer.o indexer.o mimetype.o subtreelist.o htmlparse.o myhtmlparse.o mimehandler.o internfile.o mh_exec.o mh_execm.o mh_html.o mh_mail.o mh_mbox.o mh_text.o txtdcode.o docseq.o docseqdb.o docseqhist.o filtseq.o dynconf.o plaintorich.o recollq.o reslistpager.o sortseq.o wasastringtoquery.o wasatorcl.o rcldb.o rcldoc.o rclquery.o searchdata.o stemdb.o stoplist.o base64.o circache.o closefrom.o conftree.o copyfile.o debuglog.o ecrontab.o execmd.o fstreewalk.o idfile.o fileudi.o md5.o mimeparse.o netcon.o pathut.o pxattr.o rclionice.o readfile.o smallut.o transcode.o wipedir.o x11mon.o mime-getpart.o mime-parsefull.o mime-parseonlyheader.o mime-printbody.o mime-printdoc.o mime-printheader.o mime.o convert.o iodevice.o iofactory.o +DEPS = rclaspell.dep.stamp beaglequeuecache.dep.stamp cstr.dep.stamp rclconfig.dep.stamp rclinit.dep.stamp textsplit.dep.stamp unacpp.dep.stamp beaglequeue.dep.stamp fsindexer.dep.stamp indexer.dep.stamp mimetype.dep.stamp subtreelist.dep.stamp htmlparse.dep.stamp myhtmlparse.dep.stamp mimehandler.dep.stamp internfile.dep.stamp mh_exec.dep.stamp mh_execm.dep.stamp mh_html.dep.stamp mh_mail.dep.stamp mh_mbox.dep.stamp mh_text.dep.stamp txtdcode.dep.stamp docseq.dep.stamp docseqdb.dep.stamp docseqhist.dep.stamp filtseq.dep.stamp dynconf.dep.stamp plaintorich.dep.stamp recollq.dep.stamp reslistpager.dep.stamp sortseq.dep.stamp wasastringtoquery.dep.stamp wasatorcl.dep.stamp rcldb.dep.stamp rcldoc.dep.stamp rclquery.dep.stamp searchdata.dep.stamp stemdb.dep.stamp stoplist.dep.stamp base64.dep.stamp circache.dep.stamp closefrom.dep.stamp conftree.dep.stamp copyfile.dep.stamp debuglog.dep.stamp ecrontab.dep.stamp execmd.dep.stamp fstreewalk.dep.stamp idfile.dep.stamp fileudi.dep.stamp md5.dep.stamp mimeparse.dep.stamp netcon.dep.stamp pathut.dep.stamp pxattr.dep.stamp rclionice.dep.stamp readfile.dep.stamp smallut.dep.stamp transcode.dep.stamp wipedir.dep.stamp x11mon.dep.stamp mime-getpart.dep.stamp mime-parsefull.dep.stamp mime-parseonlyheader.dep.stamp mime-printbody.dep.stamp mime-printdoc.dep.stamp mime-printheader.dep.stamp mime.dep.stamp convert.dep.stamp iodevice.dep.stamp iofactory.dep.stamp librcl.a : $(DEPS) $(OBJS) unac.o ar ru librcl.a $(OBJS) unac.o $(RANLIB) librcl.a unac.o : $(depth)/unac/unac.c $(depth)/unac/unac.h $(depth)/mk/localdefs - $(CC) $(ALL_CXXFLAGS) -c $(depth)/unac/unac.c + $(CXX) $(ALL_CXXFLAGS) -c $(depth)/unac/unac.c rclaspell.o : ../aspell/rclaspell.cpp $(depth)/mk/localdefs $(CXX) $(ALL_CXXFLAGS) -c ../aspell/rclaspell.cpp beaglequeuecache.o : ../common/beaglequeuecache.cpp $(depth)/mk/localdefs @@ -31,8 +31,6 @@ unacpp.o : ../common/unacpp.cpp $(depth)/mk/localdefs $(CXX) $(ALL_CXXFLAGS) -c ../common/unacpp.cpp beaglequeue.o : ../index/beaglequeue.cpp $(depth)/mk/localdefs $(CXX) $(ALL_CXXFLAGS) -c ../index/beaglequeue.cpp -csguess.o : ../index/csguess.cpp $(depth)/mk/localdefs - $(CXX) $(ALL_CXXFLAGS) -c ../index/csguess.cpp fsindexer.o : ../index/fsindexer.cpp $(depth)/mk/localdefs $(CXX) $(ALL_CXXFLAGS) -c ../index/fsindexer.cpp indexer.o : ../index/indexer.cpp $(depth)/mk/localdefs @@ -191,9 +189,6 @@ unacpp.dep.stamp : ../common/unacpp.cpp $(depth)/mk/localdefs beaglequeue.dep.stamp : ../index/beaglequeue.cpp $(depth)/mk/localdefs $(CXX) -M $(ALL_CXXFLAGS) ../index/beaglequeue.cpp > beaglequeue.dep touch beaglequeue.dep.stamp -csguess.dep.stamp : ../index/csguess.cpp $(depth)/mk/localdefs - $(CXX) -M $(ALL_CXXFLAGS) ../index/csguess.cpp > csguess.dep - touch csguess.dep.stamp fsindexer.dep.stamp : ../index/fsindexer.cpp $(depth)/mk/localdefs $(CXX) -M $(ALL_CXXFLAGS) ../index/fsindexer.cpp > fsindexer.dep touch fsindexer.dep.stamp @@ -364,7 +359,6 @@ include rclinit.dep include textsplit.dep include unacpp.dep include beaglequeue.dep -include csguess.dep include fsindexer.dep include indexer.dep include mimetype.dep diff --git a/src/lib/mkMake b/src/lib/mkMake index 510a435d..98c2079a 100755 --- a/src/lib/mkMake +++ b/src/lib/mkMake @@ -12,7 +12,6 @@ ${depth}/common/rclinit.cpp \ ${depth}/common/textsplit.cpp \ ${depth}/common/unacpp.cpp \ ${depth}/index/beaglequeue.cpp \ -${depth}/index/csguess.cpp \ ${depth}/index/fsindexer.cpp \ ${depth}/index/indexer.cpp \ ${depth}/index/mimetype.cpp \ @@ -122,7 +121,7 @@ librcl.a : \$(DEPS) \$(OBJS) unac.o \$(RANLIB) librcl.a unac.o : \$(depth)/unac/unac.c \$(depth)/unac/unac.h $defs - \$(CC) \$(ALL_CXXFLAGS) -c \$(depth)/unac/unac.c + \$(CXX) \$(ALL_CXXFLAGS) -c \$(depth)/unac/unac.c EOF for c in $SRC_CPP;do diff --git a/src/mk/manifest.txt b/src/mk/manifest.txt index 03452b52..d97b0ff3 100644 --- a/src/mk/manifest.txt +++ b/src/mk/manifest.txt @@ -142,6 +142,7 @@ filters/rclabw filters/rclaptosidman filters/rclaudio filters/rclchm +filters/rcldia filters/rcldjvu filters/rcldoc filters/rcldvi @@ -185,8 +186,6 @@ index/ index/Makefile index/beaglequeue.cpp index/beaglequeue.h -index/csguess.cpp -index/csguess.h index/fsindexer.cpp index/fsindexer.h index/indexer.cpp diff --git a/src/python/recoll/setup.py.in b/src/python/recoll/setup.py.in index 2f35d33c..fd73f8ef 100644 --- a/src/python/recoll/setup.py.in +++ b/src/python/recoll/setup.py.in @@ -3,14 +3,21 @@ import os import sys sysname = os.uname()[0] -if sysname == 'Linux': - libs = ['rcl', 'xapian'] -else: - libs = ['rcl', 'xapian', 'iconv'] - - top = os.path.join('..', '..') +libiconv = "@LIBICONV@" + +if libiconv.find("-liconv") != -1: + libs = ['rcl', 'xapian', 'iconv'] +else: + libs = ['rcl', 'xapian'] + +libxapiandir="@LIBXAPIANDIR@" +if libxapiandir != "": + libdirs = [os.path.join(top, 'lib'), libxapiandir, '/usr/local/lib'] +else: + libdirs = [os.path.join(top, 'lib'), '/usr/local/lib'] + # Verify that the Recoll library was compiled with the PIC options localdefs = os.path.join(top, 'mk', 'localdefs') try: @@ -42,7 +49,7 @@ module1 = Extension('recoll', os.path.join(top, 'unac') ], libraries = libs, - library_dirs = [os.path.join(top, 'lib'), '/usr/local/lib'], + library_dirs = libdirs, sources = ['pyrecoll.cpp', ]) diff --git a/src/qtgui/i18n/recoll_es.ts b/src/qtgui/i18n/recoll_es.ts index e391f564..e26f7154 100644 --- a/src/qtgui/i18n/recoll_es.ts +++ b/src/qtgui/i18n/recoll_es.ts @@ -176,7 +176,7 @@ Filter sizes - Tamaños de filtros + Filtro de tamaños diff --git a/src/qtgui/recoll.pro.in b/src/qtgui/recoll.pro.in index 21d09e98..b5a4b259 100644 --- a/src/qtgui/recoll.pro.in +++ b/src/qtgui/recoll.pro.in @@ -88,12 +88,6 @@ unix { UNAME = $$system(uname -s) contains( UNAME, [lL]inux ) { - LIBS -= -liconv - LIBS += -ldl -lX11 -} - -contains( UNAME, GNU/kFreeBSD ) { - LIBS -= -liconv LIBS += -ldl -lX11 } diff --git a/src/qtgui/reslist.cpp b/src/qtgui/reslist.cpp index bc06cf2f..18215e71 100644 --- a/src/qtgui/reslist.cpp +++ b/src/qtgui/reslist.cpp @@ -231,9 +231,16 @@ string QtGuiResListPager::iconUrl(RclConfig *config, Rcl::Doc& doc) ConfIndexer::docsToPaths(docs, paths); if (!paths.empty()) { string path; + LOGDEB2(("ResList::iconUrl: source path [%s]\n", paths[0].c_str())); if (thumbPathForUrl(cstr_fileu + paths[0], 128, path)) { + LOGDEB2(("ResList::iconUrl: icon path [%s]\n", path.c_str())); return cstr_fileu + path; + } else { + LOGDEB2(("ResList::iconUrl: no icon: path [%s]\n", + path.c_str())); } + } else { + LOGDEB(("ResList::iconUrl: docsToPaths failed\n")); } } return ResListPager::iconUrl(config, doc); diff --git a/src/sampleconf/mimeconf b/src/sampleconf/mimeconf index 210039e4..37774f47 100644 --- a/src/sampleconf/mimeconf +++ b/src/sampleconf/mimeconf @@ -71,6 +71,7 @@ application/vnd.wordperfect = exec wpd2html;mimetype=text/html application/x-abiword = exec rclabw application/x-awk = internal text/plain application/x-chm = execm rclchm +application/x-dia-diagram = execm rcldia;mimetype=text/html;charset=utf-8 application/x-dvi = exec rcldvi application/x-flac = execm rclaudio application/x-gnuinfo = execm rclinfo @@ -107,6 +108,7 @@ text/x-fictionbook = exec rclfb2 text/x-gaim-log = exec rclgaim text/x-html-sidux-man = exec rclsiduxman text/x-html-aptosid-man = exec rclaptosidman +text/x-chm-html = internal text/html text/x-ini = internal text/plain text/x-mail = internal text/x-man = exec rclman @@ -153,6 +155,7 @@ application/vnd.sun.xml.writer.global = wordprocessing application/vnd.sun.xml.writer.template = wordprocessing application/vnd.wordperfect = wordprocessing application/x-abiword = wordprocessing +application/x-dia-diagram = drawing application/x-dvi = document application/x-flac = sownd application/x-fsdirectory = folder @@ -283,6 +286,7 @@ message = message/rfc822 \ other = application/vnd.sun.xml.draw \ application/vnd.sun.xml.draw.template \ application/vnd.sun.xml.math \ + application/x-dia-diagram \ application/x-fsdirectory \ application/x-mimehtml \ application/x-rar \ diff --git a/src/sampleconf/mimemap b/src/sampleconf/mimemap index 7b532828..3d184aec 100644 --- a/src/sampleconf/mimemap +++ b/src/sampleconf/mimemap @@ -41,6 +41,7 @@ .djvu = image/vnd.djvu .svg = image/svg+xml +.dia = application/x-dia-diagram .gz = application/x-gzip .Z = application/x-gzip diff --git a/src/sampleconf/mimeview b/src/sampleconf/mimeview index fdff86b3..d421e06b 100644 --- a/src/sampleconf/mimeview +++ b/src/sampleconf/mimeview @@ -52,6 +52,7 @@ application/vnd.sun.xml.writer.template = libreoffice %f application/vnd.wordperfect = libreoffice %f application/x-chm = kchmviewer %f +application/x-dia-diagram = dia %f application/x-fsdirectory = dolphin %f application/x-gnuinfo = xterm -e "info -f %f" application/x-gnumeric = gnumeric %f @@ -82,6 +83,7 @@ text/x-c+ = emacsclient %f text/x-c++ = emacsclient %f text/x-html-sidux-man = konqueror %f text/x-html-aptosid-man = iceweasel %f +text/x-chm-html = openchm %f %i text/x-ini = emacsclient %f text/x-man = xterm -u8 -e "groff -T ascii -man %f | more" text/x-python = idle %f diff --git a/src/sampleconf/recoll.conf.in b/src/sampleconf/recoll.conf.in index d36fefd7..bab728b3 100644 --- a/src/sampleconf/recoll.conf.in +++ b/src/sampleconf/recoll.conf.in @@ -65,6 +65,17 @@ indexstemminglanguages = english # match your LANG and is not 8859-1, set it here. # defaultcharset = iso-8859-1 +# A list of characters, encoded in UTF-8, which should be handled specially +# when converting text to unaccented lowercase. For example, in Swedish, +# the letter a with diaeresis has full alphabet citizenship and should not +# be turned into an a. +# Each element in the space-separated list has the special character as +# first element and the translation following. The handling of both the +# lowercase and upper-case versions of a character should be specified, as +# appartenance to the list will turn-off both standard accent and case +# processing. Example for Swedish: +# unac_except_trans = åå Ã…Ã¥ ää Ää öö Öö + # Where to store the database (directory). This may be an absolute path, # else it is taken as relative to the configuration directory (-c argument # or $RECOLL_CONFDIR). diff --git a/src/unac/unac.c b/src/unac/unac.c index ff02e00a..bc39ddfc 100644 --- a/src/unac/unac.c +++ b/src/unac/unac.c @@ -17,15 +17,57 @@ */ #ifdef HAVE_CONFIG_H +#ifdef RECOLL_DATADIR +#include "autoconfig.h" +#else #include "config.h" +#endif /* RECOLL */ #endif /* HAVE_CONFIG_H */ +#ifdef RECOLL_DATADIR +/* Yes, recoll unac is actually c++, lets face modernity, I will not be + caught writing another binary search */ +#include +#include +#include +#include +using std::string; +using std::vector; +using std::map; +#include "smallut.h" + +/* + Storage for the exception translations. These are chars which + should not be translated according to what UnicodeData says, but + instead according to some local rule. There will usually be very + few of them, but they must be looked up for every translated char. + + We use a sorted vector for fastest elimination by binary search and + a vector to store the translations + */ +static vector except_chars; +static vector except_trans; +static inline size_t is_except_char(unsigned short c) +{ + vector::iterator it = + std::lower_bound(except_chars.begin(), except_chars.end(), c); + if (it == except_chars.end() || *it != c) { + return (size_t(-1)); + } + return std::distance(except_chars.begin(), it); +} +#endif /* RECOLL_DATADIR */ + /* * If configure.in has not defined this symbol, assume const. It * does not harm much: a warning will be issued during compilation. */ #ifndef ICONV_CONST +#ifdef RCL_ICONV_INBUF_CONST +#define ICONV_CONST const +#else #define ICONV_CONST +#endif #endif /* ICONV_CONST */ #include @@ -12622,12 +12664,12 @@ int unacmaybefold_string_utf16(const char* in, size_t in_length, char* out; int out_size; int out_length; - int i; + unsigned int i; out_size = in_length > 0 ? in_length : 1024; out = *outp; - out = realloc(out, out_size + 1); + out = (char*)realloc(out, out_size + 1); if(out == 0) { if(debug_level >= UNAC_DEBUG_LOW) DEBUG("realloc %d bytes failed\n", out_size+1); @@ -12646,11 +12688,25 @@ int unacmaybefold_string_utf16(const char* in, size_t in_length, /* * Lookup the tables for decomposition information */ - if (dofold) { - unacfold_char_utf16(c, p, l); +#ifdef RECOLL_DATADIR + size_t idx; + if (except_chars.size() != 0 && (idx=is_except_char(c)) != (size_t)-1) { + p = (unsigned short *)(except_trans[idx].c_str() + 2); + l = (except_trans[idx].size() - 2) / 2; + /* unsigned char *cp = (unsigned char *)p; + fprintf(stderr, "l %d cp[0] %x cp[1] %x\n", l, (unsigned int)cp[0], + (unsigned int)cp[1]);*/ } else { - unac_char_utf16(c, p, l); +#endif /* RECOLL_DATADIR */ + if (dofold) { + unacfold_char_utf16(c, p, l); + } else { + unac_char_utf16(c, p, l); + } +#ifdef RECOLL_DATADIR } +#endif /* RECOLL_DATADIR */ + /* * Explain what's done in great detail */ @@ -12678,7 +12734,7 @@ int unacmaybefold_string_utf16(const char* in, size_t in_length, char *saved; out_size += ((l + 1) * 2) + 1024; saved = out; - out = realloc(out, out_size); + out = (char *)realloc(out, out_size); if(out == 0) { if(debug_level >= UNAC_DEBUG_LOW) DEBUG("realloc %d bytes failed\n", out_size); @@ -12798,7 +12854,7 @@ static int convert(const char* from, const char* to, out_size = in_length > 0 ? in_length : 1024; out = *outp; - out = realloc(out, out_size + 1); + out = (char *)realloc(out, out_size + 1); if(out == 0) { /* *outp still valid, no freeing */ if(debug_level >= UNAC_DEBUG_LOW) @@ -12884,7 +12940,7 @@ static int convert(const char* from, const char* to, { char *saved = out_base; /* +1 for null */ - out_base = realloc(out_base, out_size + 1); + out_base = (char *)realloc(out_base, out_size + 1); if (out_base == 0) { /* *outp potentially not valid any more. Free here, * and zero out */ @@ -12929,7 +12985,7 @@ int unacmaybefold_string(const char* charset, */ if (in_length <= 0) { if(!*outp) { - if ((*outp = malloc(32)) == 0) + if ((*outp = (char*)malloc(32)) == 0) return -1; } (*outp)[0] = '\0'; @@ -12975,3 +13031,64 @@ const char* unac_version(void) return UNAC_VERSION; } +#ifdef RECOLL_DATADIR +void unac_set_except_translations(const char *spectrans) +{ + except_chars.clear(); + except_trans.clear(); + if (!spectrans || !spectrans[0]) + return; + + // The translation tables out of Unicode are in machine byte order (we + // just let the compiler read the values). + // For the translation part, we need to choose our encoding in accordance ) + // (16BE or 16LE depending on processor) + // On the contrary, the source char is always to be compared to + // the input text, which is encoded in UTF-16BE ... What a mess. + static const char *machinecoding = 0; + bool littleendian = true; + if (machinecoding == 0) { + const char* charshort = "\001\002"; + short *ip = (short *)charshort; + if (*ip == 0x0102) { + littleendian = false; + machinecoding = "UTF-16BE"; + } else { + littleendian = true; + machinecoding = "UTF-16LE"; + } + } + + vector vtrans; + stringToStrings(spectrans, vtrans); + + for (vector::iterator it = vtrans.begin(); + it != vtrans.end(); it++) { + + /* Convert the whole thing to utf-16be/le according to endianness */ + char *out = 0; + size_t outsize; + if (convert("UTF-8", machinecoding, + it->c_str(), it->size(), + &out, &outsize) != 0 || outsize < 2) + continue; + + /* The source char must be utf-16be as this is what we convert the + input text to for internal processing */ + unsigned short ch; + if (littleendian) + ch = (out[1] << 8) | (out[0] & 0xff); + else + ch = (out[0] << 8) | (out[1] & 0xff); + + /* fprintf(stderr, "outsize %d Ch is 0x%hx\n", int(outsize), ch);*/ + except_chars.push_back(ch); + // We keep ch as the first 2 bytes in the translation so that + // both vectors sort identically + except_trans.push_back(string((const char *)out, outsize)); + free(out); + } + std::sort(except_chars.begin(), except_chars.end()); + std::sort(except_trans.begin(), except_trans.end()); +} +#endif /* RECOLL_DATADIR */ diff --git a/src/unac/unac.h b/src/unac/unac.h index 74b6e602..13c52098 100644 --- a/src/unac/unac.h +++ b/src/unac/unac.h @@ -116,6 +116,22 @@ int unacfold_string(const char* charset, /* To be called before starting threads in mt programs */ void unac_init_mt(); +#ifdef RECOLL_DATADIR +#include +/** + * Set exceptions for unaccenting, for characters which should not be + * handled according to what the Unicode tables say. For example "a + * with circle above" should not be stripped to a in swedish, etc. + * + * @param spectrans defines the translations as a blank separated list of + * UTF-8 strings. Inside each string, the first character is the exception + * the rest is the translation (which may be empty). You can use double + * quotes for translations which should include white space. The double-quote + * can't be an exception character, deal with it... + */ +void unac_set_except_translations(const char *spectrans); +#endif /* RECOLL_DATADIR */ + /* * Return unac version number. */ diff --git a/src/utils/pathut.cpp b/src/utils/pathut.cpp index 8b4c59fd..65f406a0 100644 --- a/src/utils/pathut.cpp +++ b/src/utils/pathut.cpp @@ -389,6 +389,52 @@ bool path_isdir(const string& path) return false; } +// Allowed punctuation in the path part of an URI according to RFC2396 +// -_.!~*'():@&=+$, +/* +21 ! + 22 " + 23 # +24 $ + 25 % +26 & +27 ' +28 ( +29 ) +2A * +2B + +2C , +2D - +2E . +2F / +30 0 +... +39 9 +3A : + 3B ; + 3C < +3D = + 3E > + 3F ? +40 @ +41 A +... +5A Z + 5B [ + 5C \ + 5D ] + 5E ^ +5F _ + 60 ` +61 a +... +7A z + 7B { + 7C | + 7D } +7E ~ + 7F DEL +*/ string url_encode(const string& url, string::size_type offs) { string out = url.substr(0, offs); @@ -397,24 +443,23 @@ string url_encode(const string& url, string::size_type offs) int c; const char *h = "0123456789ABCDEF"; c = cp[i]; - if(c <= 0x1f || + if (c <= 0x20 || c >= 0x7f || - c == '<' || - c == '>' || - c == ' ' || - c == '\t'|| c == '"' || c == '#' || c == '%' || - c == '{' || - c == '}' || - c == '|' || - c == '\\' || - c == '^' || - c == '~'|| + c == ';' || + c == '<' || + c == '>' || + c == '?' || c == '[' || + c == '\\' || c == ']' || - c == '`') { + c == '^' || + c == '`' || + c == '{' || + c == '|' || + c == '}' ) { out += '%'; out += h[(c >> 4) & 0xf]; out += h[c & 0xf]; @@ -445,7 +490,9 @@ string url_gpath(const string& url) return path_canon(url.substr(colon+1)); } -// Convert to file path if url is like file:// +// Convert to file path if url is like file: +// Note: this only works with our internal pseudo-urls which are not +// encoded/escaped string fileurltolocalpath(string url) { if (url.find("file://") == 0) @@ -570,7 +617,8 @@ static const string thmbdirnormal = ".thumbnails/normal"; static void thumbname(const string& url, string& name) { string digest; - MD5String(url, digest); + string l_url = url_encode(url); + MD5String(l_url, digest); MD5HexPrint(digest, name); name += ".png"; } diff --git a/src/utils/rclionice.cpp b/src/utils/rclionice.cpp index 31072cb2..c5a01bb2 100644 --- a/src/utils/rclionice.cpp +++ b/src/utils/rclionice.cpp @@ -15,6 +15,7 @@ * 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA. */ #include +#include #include "rclionice.h" #include "execmd.h" diff --git a/tests/config/recoll.conf b/tests/config/recoll.conf index 59d3e389..df13a81b 100644 --- a/tests/config/recoll.conf +++ b/tests/config/recoll.conf @@ -15,6 +15,8 @@ daemSkippedPaths = \ defaultcharset = iso-8859-1 +unac_except_trans = åå Ã…Ã¥ ää Ää öö Öö + [/home/dockes/projets/fulltext/testrecoll/special] defaultcharset = iso-8859-1 [/home/dockes/projets/fulltext/testrecoll/txt] diff --git a/tests/dia/dia.sh b/tests/dia/dia.sh new file mode 100755 index 00000000..313c5d96 --- /dev/null +++ b/tests/dia/dia.sh @@ -0,0 +1,14 @@ +#!/bin/sh + +topdir=`dirname $0`/.. +. $topdir/shared.sh + +initvariables $0 +( +recollq DEPARTMENT NumberOfEmployees +recollq Within5Min WaterLevel KeptBelow +) 2> $mystderr | egrep -v '^Recoll query: ' > $mystdout + +diff -w ${myname}.txt $mystdout > $mydiffs 2>&1 + +checkresult diff --git a/tests/dia/dia.txt b/tests/dia/dia.txt new file mode 100644 index 00000000..5bb5b1bb --- /dev/null +++ b/tests/dia/dia.txt @@ -0,0 +1,4 @@ +1 results +application/x-dia-diagram [file:///home/dockes/projets/fulltext/testrecoll/dia/ER-demo.dia] [ER-demo.dia] 60803 bytes +1 results +application/x-dia-diagram [file:///home/dockes/projets/fulltext/testrecoll/dia/Istar.dia] [Istar.dia] 3537 bytes diff --git a/tests/lyx/lyx.sh b/tests/lyx/lyx.sh index 7610142c..0aad06c7 100755 --- a/tests/lyx/lyx.sh +++ b/tests/lyx/lyx.sh @@ -8,7 +8,9 @@ initvariables $0 ( recollq 'Bienvenue Dans Univers De Lyx' recollq 'Welcome To Lyx' -recollq 'Udvozli Ont A LyX' +recollq 'LyX rendkivul jol dokumentalt' +# cant use this one because o-diaeresis is in the swedish unacex +# recollq 'Udvozli Ont A LyX' ) 2> $mystderr | egrep -v '^Recoll query: ' > $mystdout diff -w ${myname}.txt $mystdout > $mydiffs 2>&1 diff --git a/tests/unacex/unacex.sh b/tests/unacex/unacex.sh new file mode 100755 index 00000000..582a5206 --- /dev/null +++ b/tests/unacex/unacex.sh @@ -0,0 +1,20 @@ +#!/bin/sh + +topdir=`dirname $0`/.. +. $topdir/shared.sh + +initvariables $0 + +# We need an utf-8 locale for the 1st command to properly read its argument +export LANG=fr_FR.UTF-8 + +( +# Should succeed +recollq '"strippes: UNACEXååääöö"' +# Should fail +recollq '"strippes: UNACEXaaaaoo"' +) 2> $mystderr | egrep -v '^Recoll query: ' > $mystdout + +diff -w ${myname}.txt $mystdout > $mydiffs 2>&1 + +checkresult diff --git a/tests/unacex/unacex.txt b/tests/unacex/unacex.txt new file mode 100644 index 00000000..242edba6 --- /dev/null +++ b/tests/unacex/unacex.txt @@ -0,0 +1,3 @@ +1 results +text/html [file:///home/dockes/projets/fulltext/testrecoll/unacex/swedishchars.html] [Some chars] 293 bytes +0 results diff --git a/unac/unac.c b/unac/unac.c index ff02e00a..bc39ddfc 100644 --- a/unac/unac.c +++ b/unac/unac.c @@ -17,15 +17,57 @@ */ #ifdef HAVE_CONFIG_H +#ifdef RECOLL_DATADIR +#include "autoconfig.h" +#else #include "config.h" +#endif /* RECOLL */ #endif /* HAVE_CONFIG_H */ +#ifdef RECOLL_DATADIR +/* Yes, recoll unac is actually c++, lets face modernity, I will not be + caught writing another binary search */ +#include +#include +#include +#include +using std::string; +using std::vector; +using std::map; +#include "smallut.h" + +/* + Storage for the exception translations. These are chars which + should not be translated according to what UnicodeData says, but + instead according to some local rule. There will usually be very + few of them, but they must be looked up for every translated char. + + We use a sorted vector for fastest elimination by binary search and + a vector to store the translations + */ +static vector except_chars; +static vector except_trans; +static inline size_t is_except_char(unsigned short c) +{ + vector::iterator it = + std::lower_bound(except_chars.begin(), except_chars.end(), c); + if (it == except_chars.end() || *it != c) { + return (size_t(-1)); + } + return std::distance(except_chars.begin(), it); +} +#endif /* RECOLL_DATADIR */ + /* * If configure.in has not defined this symbol, assume const. It * does not harm much: a warning will be issued during compilation. */ #ifndef ICONV_CONST +#ifdef RCL_ICONV_INBUF_CONST +#define ICONV_CONST const +#else #define ICONV_CONST +#endif #endif /* ICONV_CONST */ #include @@ -12622,12 +12664,12 @@ int unacmaybefold_string_utf16(const char* in, size_t in_length, char* out; int out_size; int out_length; - int i; + unsigned int i; out_size = in_length > 0 ? in_length : 1024; out = *outp; - out = realloc(out, out_size + 1); + out = (char*)realloc(out, out_size + 1); if(out == 0) { if(debug_level >= UNAC_DEBUG_LOW) DEBUG("realloc %d bytes failed\n", out_size+1); @@ -12646,11 +12688,25 @@ int unacmaybefold_string_utf16(const char* in, size_t in_length, /* * Lookup the tables for decomposition information */ - if (dofold) { - unacfold_char_utf16(c, p, l); +#ifdef RECOLL_DATADIR + size_t idx; + if (except_chars.size() != 0 && (idx=is_except_char(c)) != (size_t)-1) { + p = (unsigned short *)(except_trans[idx].c_str() + 2); + l = (except_trans[idx].size() - 2) / 2; + /* unsigned char *cp = (unsigned char *)p; + fprintf(stderr, "l %d cp[0] %x cp[1] %x\n", l, (unsigned int)cp[0], + (unsigned int)cp[1]);*/ } else { - unac_char_utf16(c, p, l); +#endif /* RECOLL_DATADIR */ + if (dofold) { + unacfold_char_utf16(c, p, l); + } else { + unac_char_utf16(c, p, l); + } +#ifdef RECOLL_DATADIR } +#endif /* RECOLL_DATADIR */ + /* * Explain what's done in great detail */ @@ -12678,7 +12734,7 @@ int unacmaybefold_string_utf16(const char* in, size_t in_length, char *saved; out_size += ((l + 1) * 2) + 1024; saved = out; - out = realloc(out, out_size); + out = (char *)realloc(out, out_size); if(out == 0) { if(debug_level >= UNAC_DEBUG_LOW) DEBUG("realloc %d bytes failed\n", out_size); @@ -12798,7 +12854,7 @@ static int convert(const char* from, const char* to, out_size = in_length > 0 ? in_length : 1024; out = *outp; - out = realloc(out, out_size + 1); + out = (char *)realloc(out, out_size + 1); if(out == 0) { /* *outp still valid, no freeing */ if(debug_level >= UNAC_DEBUG_LOW) @@ -12884,7 +12940,7 @@ static int convert(const char* from, const char* to, { char *saved = out_base; /* +1 for null */ - out_base = realloc(out_base, out_size + 1); + out_base = (char *)realloc(out_base, out_size + 1); if (out_base == 0) { /* *outp potentially not valid any more. Free here, * and zero out */ @@ -12929,7 +12985,7 @@ int unacmaybefold_string(const char* charset, */ if (in_length <= 0) { if(!*outp) { - if ((*outp = malloc(32)) == 0) + if ((*outp = (char*)malloc(32)) == 0) return -1; } (*outp)[0] = '\0'; @@ -12975,3 +13031,64 @@ const char* unac_version(void) return UNAC_VERSION; } +#ifdef RECOLL_DATADIR +void unac_set_except_translations(const char *spectrans) +{ + except_chars.clear(); + except_trans.clear(); + if (!spectrans || !spectrans[0]) + return; + + // The translation tables out of Unicode are in machine byte order (we + // just let the compiler read the values). + // For the translation part, we need to choose our encoding in accordance ) + // (16BE or 16LE depending on processor) + // On the contrary, the source char is always to be compared to + // the input text, which is encoded in UTF-16BE ... What a mess. + static const char *machinecoding = 0; + bool littleendian = true; + if (machinecoding == 0) { + const char* charshort = "\001\002"; + short *ip = (short *)charshort; + if (*ip == 0x0102) { + littleendian = false; + machinecoding = "UTF-16BE"; + } else { + littleendian = true; + machinecoding = "UTF-16LE"; + } + } + + vector vtrans; + stringToStrings(spectrans, vtrans); + + for (vector::iterator it = vtrans.begin(); + it != vtrans.end(); it++) { + + /* Convert the whole thing to utf-16be/le according to endianness */ + char *out = 0; + size_t outsize; + if (convert("UTF-8", machinecoding, + it->c_str(), it->size(), + &out, &outsize) != 0 || outsize < 2) + continue; + + /* The source char must be utf-16be as this is what we convert the + input text to for internal processing */ + unsigned short ch; + if (littleendian) + ch = (out[1] << 8) | (out[0] & 0xff); + else + ch = (out[0] << 8) | (out[1] & 0xff); + + /* fprintf(stderr, "outsize %d Ch is 0x%hx\n", int(outsize), ch);*/ + except_chars.push_back(ch); + // We keep ch as the first 2 bytes in the translation so that + // both vectors sort identically + except_trans.push_back(string((const char *)out, outsize)); + free(out); + } + std::sort(except_chars.begin(), except_chars.end()); + std::sort(except_trans.begin(), except_trans.end()); +} +#endif /* RECOLL_DATADIR */ diff --git a/unac/unac.h b/unac/unac.h index 74b6e602..13c52098 100644 --- a/unac/unac.h +++ b/unac/unac.h @@ -116,6 +116,22 @@ int unacfold_string(const char* charset, /* To be called before starting threads in mt programs */ void unac_init_mt(); +#ifdef RECOLL_DATADIR +#include +/** + * Set exceptions for unaccenting, for characters which should not be + * handled according to what the Unicode tables say. For example "a + * with circle above" should not be stripped to a in swedish, etc. + * + * @param spectrans defines the translations as a blank separated list of + * UTF-8 strings. Inside each string, the first character is the exception + * the rest is the translation (which may be empty). You can use double + * quotes for translations which should include white space. The double-quote + * can't be an exception character, deal with it... + */ +void unac_set_except_translations(const char *spectrans); +#endif /* RECOLL_DATADIR */ + /* * Return unac version number. */ diff --git a/website/doc.html b/website/doc.html index 633d3a25..4d06d750 100644 --- a/website/doc.html +++ b/website/doc.html @@ -12,7 +12,7 @@ - + @@ -34,6 +34,8 @@
  • English, HTML
    • +
  • + 中文，HTML
  • English, PDF
  • Italian (rather old)
    • diff --git a/website/features.html b/website/features.html index 62ccf2ed..33ba555a 100644 --- a/website/features.html +++ b/website/features.html @@ -87,11 +87,11 @@

    Document types

    Recoll can index many document types (along with their - compressed versions). Some types are handled internally (no - external application needed). Other types need a separate - application to be installed to extract the text. Types that - only need very common utilities (awk/sed/groff etc.) are - listed in the native section.

    + compressed versions). Some types are handled internally (no + external application needed). Other types need a separate + application to be installed to extract the text. Types that + only need very common utilities (awk/sed/groff/Python etc.) + are listed in the native section.

    File types indexed natively

    @@ -100,22 +100,25 @@
  • html.
    • -
  • maildir and mailbox (Mozilla, Thunderbird and Evolution mail ok).
    • +
  • maildir and + mailbox ( + Mozilla, + Thunderbird and + Evolutionmail ok). +
    • -
  • gaim and purple log files.
    • +
  • gaim and + purple log files.
    • -
  • Lyx files (needs Lyx to be installed).
    • +
  • Lyx files (needs + Lyx to be installed).
  • Scribus files.
    • -
  • Man pages (need groff).
    • +
  • Man pages (needs + groff).
    • + +
  • Dia diagrams.

File types indexed with external helpers

@@ -124,9 +127,10 @@ command in addition to the applications specifically listed.

The XML ones
-

The following types need xsltproc from the libxslt package. - Quite a few also need unzip:

+ +

The following types need + xsltproc from the libxslt package. + Quite a few also need unzip:

  • Abiword files.
    • @@ -148,6 +152,9 @@
    Other formats
    +

    The following need miscellaneous helper programs to decode + the internal formats.

    +
    • pdf with the pdftotext command, which can be installed @@ -166,7 +173,8 @@ "http://vitus.wagner.pp.ru/software/catdoc/">catdoc utilities.
    • CHM (Microsoft help) files - with Python, pychm + with Python, + pychm and chmlib.
    • GNU info files diff --git a/website/filters/filters.html b/website/filters/filters.html index 7e822a01..753d4b1e 100644 --- a/website/filters/filters.html +++ b/website/filters/filters.html @@ -75,6 +75,11 @@ should upgrade.

      +

      DIA files

      +

      rcldia is a new filter + for Dia files, + contributed by Stefan Friedel.

      +

      CHM files

      rclchm. The previous version of the filter mishandled files which had encoded internal URLs (not diff --git a/website/index.html.en b/website/index.html.en index 78083018..b69eb47f 100644 --- a/website/index.html.en +++ b/website/index.html.en @@ -7,7 +7,7 @@ - + @@ -69,20 +69,19 @@

      News:

        -
      • 2012-03-27: The Unity Lens can now handle results for embedded - documents, using the Recoll GUI as a proxy to extract and - display them.
        • +
      • 2012-04-07: we now have a Chinese user manual: Recoll现在有中文手册咯： + + Recoll中文手册，HTML
        • -
      • 2012-03-26: Recoll gets a Ubuntu Unity Lens. If you are running +
      • 2012-03-27: Recoll gets a Ubuntu Unity Lens. If you are running an Ubuntu release where this makes sense, you can install the recoll-lens package from the - - Recoll PPA.
        - Please note: for now, the Lens will only return - results which are normal files, not embedded documents. - Ie: no mbox messages, no attachments, no archive - members, etc. For now, you'll have to use the full Recoll - GUI to find these.
        • + + Recoll PPA. The Lens uses the Recoll GUI as a proxy to + extract and display embedded documents, which native utility + can't reach directly. And of course you still need to run + the GUI (or the command line recollindex) to get indexing + going !
      • 2012-03-24: Release 1.17 is out, see the Release notes.
        • diff --git a/website/pics/00REMAKEALB.sh b/website/pics/00REMAKEALB.sh new file mode 100644 index 00000000..b90c0ea0 --- /dev/null +++ b/website/pics/00REMAKEALB.sh @@ -0,0 +1,2 @@ +#!/bin/sh +onlylist=1 photalb . . diff --git a/website/pics/index.html b/website/pics/index.html index 2e9a69d3..3eab3b14 100644 --- a/website/pics/index.html +++ b/website/pics/index.html @@ -1,9 +1,30 @@ + - Photos + Recoll screenshots + + + + + + + + + + + + +
        + +

        Recoll Screenshots

        +
      • Back to Recoll home
        • + @@ -19,4 +40,5 @@ -
        + + diff --git a/website/pics/index.html.head b/website/pics/index.html.head new file mode 100644 index 00000000..32eed3ce --- /dev/null +++ b/website/pics/index.html.head @@ -0,0 +1,27 @@ + + + + + Recoll screenshots + + + + + + + + + + + + + + +
        + +

        Recoll Screenshots

        +
      • Back to Recoll home
        • + diff --git a/website/pics/recoll-HTML_search_results.html b/website/pics/recoll-HTML_search_results.html index c190076b..3985edb3 100644 --- a/website/pics/recoll-HTML_search_results.html +++ b/website/pics/recoll-HTML_search_results.html @@ -4,9 +4,9 @@ Photo -

        Prev Up +

        Prev Up Next      - Raw Image

        + Image

        A customized result list, thanks to Michael Croes. The html code follows, it should be pasted into the Preferences->Query Configuration->Result paragraph format string entry. diff --git a/website/pics/recoll0.html b/website/pics/recoll0.html index 2cead8b3..c5208800 100644 --- a/website/pics/recoll0.html +++ b/website/pics/recoll0.html @@ -4,9 +4,9 @@ Photo -

        Prev Up +

        Prev Up Next      - Raw Image

        + Image

        Search results.

        diff --git a/website/pics/recoll1.html b/website/pics/recoll1.html index 8be2a21b..882ae903 100644 --- a/website/pics/recoll1.html +++ b/website/pics/recoll1.html @@ -4,9 +4,9 @@ Photo -

        Prev Up +

        Prev Up Next      - Raw Image

        + Image

        A result list with a preview window open.

        diff --git a/website/pics/recoll2-thumb.png b/website/pics/recoll2-thumb.png index d3d85851..4f4cf021 100644 Binary files a/website/pics/recoll2-thumb.png and b/website/pics/recoll2-thumb.png differ diff --git a/website/pics/recoll2.html b/website/pics/recoll2.html index bc6b8d8d..d410d3ab 100644 --- a/website/pics/recoll2.html +++ b/website/pics/recoll2.html @@ -4,10 +4,10 @@ Photo -

        Prev Up +

        Prev Up Next      - Raw Image

        -

        The advanced search dialog.

        + Image

        +

        The two tabs in the advanced search dialog.

        diff --git a/website/pics/recoll2.png b/website/pics/recoll2.png index 995b66ff..2fa3a339 100644 Binary files a/website/pics/recoll2.png and b/website/pics/recoll2.png differ diff --git a/website/pics/recoll2.txt b/website/pics/recoll2.txt index a70ce9bd..4e503cb3 100644 --- a/website/pics/recoll2.txt +++ b/website/pics/recoll2.txt @@ -1 +1 @@ -The advanced search dialog. +The two tabs in the advanced search dialog. diff --git a/website/pics/recoll3.html b/website/pics/recoll3.html index ea9ff5d5..74d8ba5f 100644 --- a/website/pics/recoll3.html +++ b/website/pics/recoll3.html @@ -4,9 +4,9 @@ Photo -

        Prev Up +

        Prev Up Next      - Raw Image

        + Image

        A result list from which the native application (firefox) was started by clicking the Edit link.

        diff --git a/website/pics/recoll4.html b/website/pics/recoll4.html index 3bd0c1d8..505f8444 100644 --- a/website/pics/recoll4.html +++ b/website/pics/recoll4.html @@ -4,9 +4,9 @@ Photo -

        Prev Up +

        Prev Up Next      - Raw Image

        + Image

        The document history window looks a little like a result list I'm afraid...

        diff --git a/website/pics/recoll5.html b/website/pics/recoll5.html index d0c4c81f..479ccf87 100644 --- a/website/pics/recoll5.html +++ b/website/pics/recoll5.html @@ -4,9 +4,9 @@ Photo -

        Prev Up +

        Prev Up Next      - Raw Image

        + Image

        The term explorer tool in phonetic mode.

        diff --git a/website/pics/recoll_chinese.html b/website/pics/recoll_chinese.html index 133dacf4..bbe5b132 100644 --- a/website/pics/recoll_chinese.html +++ b/website/pics/recoll_chinese.html @@ -4,9 +4,9 @@ Photo -

        Prev Up +

        Prev Up Next      - Raw Image

        + Image

        Recoll searching Chinese text. Chinese text search is based on n-grams and relatively rough, but still useful (I am told).

        diff --git a/website/pics/result-table.html b/website/pics/result-table.html index 08167805..262d20d4 100644 --- a/website/pics/result-table.html +++ b/website/pics/result-table.html @@ -4,9 +4,9 @@ Photo -

        Prev Up +

        Prev Up Next      - Raw Image

        + Image