- Wikitty-commits - mailman.cloud.codelutin.com

r1416 - trunk/src/site/rst/user
by bpoussin＠users.nuiton.org 03 Feb '12

03 Feb '12

Author: bpoussin Date: 2012-02-03 19:31:47 +0100 (Fri, 03 Feb 2012) New Revision: 1416 Url: http://nuiton.org/repositories/revision/wikitty/1416 Log: update ~ et !~ documentation (eqIgnoreCaseAndAccent et neIgnoreCaseAndAccent) Modified: trunk/src/site/rst/user/query.rst Modified: trunk/src/site/rst/user/query.rst =================================================================== --- trunk/src/site/rst/user/query.rst 2012-02-03 18:29:27 UTC (rev 1415) +++ trunk/src/site/rst/user/query.rst 2012-02-03 18:31:47 UTC (rev 1416) @@ -123,6 +123,32 @@ WikittyQuery q = WikittyQueryParser.parse("User.lastname!=*ssin"); WikittyQuery q = WikittyQueryParser.parse("User.lastname!=*ssin"); +Egalité sans casse ni accent: eqIgnoreCaseAndAccent ++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Vérifie si un champ est égal a une valeur, si le champs est un +champs texte, il est possible d'ajouter des "*" en début et en fin de la +chaîne de recherche pour faire l'équivalent d'un startsWith ou endsWith. La +recherche est faite sans tenir compte de la casse ni des accents:: + + WikittyQuery q = new WikittyQueryMaker().eqIgnoreCaseAndAccent(User.ELEMENT_FIELD_USER_LASTNAME, "Poussin").end(); + WikittyQuery q = new WikittyQueryMaker().eqIgnoreCaseAndAccent(User.ELEMENT_FIELD_USER_LASTNAME, "*SSIN").end(); + + WikittyQuery q = WikittyQueryParser.parse("User.lastname~Poussin"); + WikittyQuery q = WikittyQueryParser.parse("User.lastname~*SSIN"); + +Non égalité sans casse ni accent: neIgnoreCaseAndAccent ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Vérifie si un champ n'est pas égal a une valeur. De la +même façon que le eqIgnoreCaseAndAccent, il est possible d'ajouter des '*'.:: + + WikittyQuery q = new WikittyQueryMaker().ne(User.ELEMENT_FIELD_USER_LASTNAME, "Poussin").end(); + WikittyQuery q = new WikittyQueryMaker().ne(User.ELEMENT_FIELD_USER_LASTNAME, "*SSIN").end(); + + WikittyQuery q = WikittyQueryParser.parse("User.lastname!~Poussin"); + WikittyQuery q = WikittyQueryParser.parse("User.lastname!~*SSIN"); + Contient un : containsOne +++++++++++++++++++++++++++

1 0

r1415 - trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query
by bpoussin＠users.nuiton.org 03 Feb '12

03 Feb '12

Author: bpoussin Date: 2012-02-03 19:29:27 +0100 (Fri, 03 Feb 2012) New Revision: 1415 Url: http://nuiton.org/repositories/revision/wikitty/1415 Log: Anomalie #1928: Query limit set to MAX (or -1) won't work remplacement de -1 par Integer.MAX_VALUE Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQuery.java Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQuery.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQuery.java 2012-02-03 18:27:46 UTC (rev 1414) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQuery.java 2012-02-03 18:29:27 UTC (rev 1415) @@ -60,7 +60,7 @@ private static final long serialVersionUID = 1L; /** Use to not limit result. */ - static final public int MAX = -1; + static final public int MAX = Integer.MAX_VALUE; /** query name */ protected String name;

1 0

r1414 - trunk/wikitty-api/src/test/java/org/nuiton/wikitty
by bpoussin＠users.nuiton.org 03 Feb '12

03 Feb '12

Author: bpoussin Date: 2012-02-03 19:27:46 +0100 (Fri, 03 Feb 2012) New Revision: 1414 Url: http://nuiton.org/repositories/revision/wikitty/1414 Log: suppression de @Ignore sur certain test et correction du code ou du test Modified: trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientAbstractTest.java trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java Modified: trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientAbstractTest.java =================================================================== --- trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientAbstractTest.java 2012-02-03 18:26:20 UTC (rev 1413) +++ trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientAbstractTest.java 2012-02-03 18:27:46 UTC (rev 1414) @@ -145,12 +145,14 @@ /** Manual extension name : media. */ protected static final String MEDIA_EXTENSION_NAME = "Media"; /** Manual extension media. */ - protected static final WikittyExtension MEDIA_EXTENSION = ExtensionFactory.create(MEDIA_EXTENSION_NAME, "6.0") + protected static final WikittyExtension MEDIA_EXTENSION = + ExtensionFactory.create(MEDIA_EXTENSION_NAME, "6.0") .addField("type", WikittyTypes.STRING) .extension(); protected static final String MOVIE_EXTENSION_NAME = "Movies"; /** Manual extension movies. */ - protected static final WikittyExtension MOVIE_EXTENSION = ExtensionFactory.create(MOVIE_EXTENSION_NAME, "2.0", MEDIA_EXTENSION_NAME) + protected static final WikittyExtension MOVIE_EXTENSION = + ExtensionFactory.create(MOVIE_EXTENSION_NAME, "2.0", MEDIA_EXTENSION_NAME) .addField("name", WikittyTypes.STRING) .addField("authors", WikittyTypes.STRING).maxOccur(Integer.MAX_VALUE) .addField("category", WikittyTypes.WIKITTY) @@ -234,7 +236,7 @@ edgarMovie.addExtension(MOVIE_EXTENSION); edgarMovie.setField(MOVIE_EXTENSION_NAME, "name", "J. Edgar"); edgarMovie.addToField(MOVIE_EXTENSION_NAME, "authors", "Eastwood"); - edgarMovie.setField("Movies", "date", "25/12/2011"); + edgarMovie.setField(MOVIE_EXTENSION_NAME, "date", "25/12/2011"); Wikitty dnrMovie = new WikittyImpl(); dnrMovie.addExtension(MEDIA_EXTENSION); Modified: trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java =================================================================== --- trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java 2012-02-03 18:26:20 UTC (rev 1413) +++ trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java 2012-02-03 18:27:46 UTC (rev 1414) @@ -106,10 +106,10 @@ w = wikittyClient.store(w); w = wikittyClient.restore(id); Assert.assertTrue(w.hasExtension(MOVIE_EXTENSION_NAME)); - Assert.assertTrue(w.hasField("Movies", "name")); - Assert.assertTrue(w.hasField("Movies", "authors")); - Assert.assertTrue(w.hasField("Movies", "category")); - Assert.assertTrue(w.hasField("Movies", "date")); + Assert.assertTrue(w.hasField(MOVIE_EXTENSION_NAME, "name")); + Assert.assertTrue(w.hasField(MOVIE_EXTENSION_NAME, "authors")); + Assert.assertTrue(w.hasField(MOVIE_EXTENSION_NAME, "category")); + Assert.assertTrue(w.hasField(MOVIE_EXTENSION_NAME, "date")); } @Test @@ -118,9 +118,9 @@ List<String> extIds = wikittyClient.getAllExtensionIds(); log.debug("extIds: " + extIds); Assert.assertEquals(8, extIds.size()); - Assert.assertTrue(extIds.contains("Movies[2.0]")); - WikittyExtension ext = wikittyClient.restoreExtension("Movies[2.0]"); - Assert.assertEquals("Movies", ext.getName()); + Assert.assertTrue(extIds.contains(MOVIE_EXTENSION_NAME+"[2.0]")); + WikittyExtension ext = wikittyClient.restoreExtension(MOVIE_EXTENSION_NAME+"[2.0]"); + Assert.assertEquals(MOVIE_EXTENSION_NAME, ext.getName()); Assert.assertEquals("2.0", ext.getVersion()); Assert.assertEquals(MOVIE_EXTENSION, ext); } @@ -167,7 +167,7 @@ Assert.assertEquals(1, extIds.size()); WikittyExtension ext = wikittyClient.restoreExtension(extIds.get(0)); - Assert.assertEquals("Movies", ext.getName()); + Assert.assertEquals(MOVIE_EXTENSION_NAME, ext.getName()); Assert.assertEquals("2.0", ext.getVersion()); Assert.assertEquals(MOVIE_EXTENSION, ext); } @@ -615,11 +615,7 @@ Assert.assertNotNull(restoredWikitty); } - /** - * FIXME echatellier 20120131 fix test (fails with solr analyzer) - */ @Test - @Ignore public void testStorageBinaryField() { String extName = "BinaryExt"; byte[] bytes = "Coucou le monde".getBytes(); @@ -639,7 +635,7 @@ Wikitty restoredWikitty = wikittyClient.restore(w.getId()); Assert.assertNotNull(restoredWikitty); Assert.assertEquals("LeBin", restoredWikitty.getFieldAsString(extName, "name")); - Assert.assertEquals(bytes, restoredWikitty.getFieldAsBytes(extName, "content")); + Assert.assertTrue(Arrays.equals(bytes, restoredWikitty.getFieldAsBytes(extName, "content"))); } @Test @@ -728,7 +724,7 @@ // The Dark Knight Rises // Die hard 4 assumeTrueSearchEngineCanRunTest(); // ? in regex - WikittyQuery query = new WikittyQueryMaker().eq("Movies.name", "*ar?*").end(); + WikittyQuery query = new WikittyQueryMaker().eq(MOVIE_EXTENSION_NAME + ".name", "*ar?*").end(); WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(2, results.getAll().size()); } @@ -1001,13 +997,13 @@ @Test public void testQueryMakerNeq() { // FIXME echatellier 20120125 should work - assumeTrueSearchEngineCanRunTest(); +// assumeTrueSearchEngineCanRunTest(); // 22 wikitty in init // only one with price == 42 // test strict equals WikittyQuery query = new WikittyQueryMaker().ne(Product.FQ_FIELD_PRODUCT_PRICE, 42).end(); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(21, results.getTotalResult()); Assert.assertEquals(21, results.getAll().size()); } @@ -1127,7 +1123,7 @@ importBooks(); // 13 importé = 35 // -1 avec cet id WikittyQuery query = new WikittyQueryMaker().idne("db9dc782-e650-4fd4-83ac-3c1c5c136cde").end(); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(34, results.getTotalResult()); } @@ -1137,33 +1133,33 @@ importBooks(); // 13 importé = 35 // -1 avec cet id WikittyQuery query = WikittyQueryParser.parse("id!=db9dc782-e650-4fd4-83ac-3c1c5c136cde"); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(34, results.getTotalResult()); } @Test public void testQueryMakerUnlike() throws IOException { // FIXME echatellier 20120125 should work - assumeTrueSearchEngineCanRunTest(); +// assumeTrueSearchEngineCanRunTest(); // + 22 in init db importBooks(); // 13 importé = 35 // -6 HP WikittyQuery query = new WikittyQueryMaker().unlike(Product.ELEMENT_FIELD_PRODUCT_NAME, "*Potter*").end(); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(29, results.getTotalResult()); // -6 HP = 11 } @Test public void testQueryParserUnlike() throws IOException { // FIXME echatellier 20120125 should work - assumeTrueSearchEngineCanRunTest(); +// assumeTrueSearchEngineCanRunTest(); // + 22 in init db importBooks(); // 13 importé = 35 // -6 HP WikittyQuery query = WikittyQueryParser.parse("Product.name UNLIKE *Potter*"); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(29, results.getTotalResult()); // -6 HP = 11 } @@ -1202,52 +1198,52 @@ @Test public void testQueryMakerNotsw() throws IOException { // FIXME echatellier 20120125 should work - assumeTrueSearchEngineCanRunTest(); +// assumeTrueSearchEngineCanRunTest(); // + 22 in init db importBooks(); // 13 importé = 35 // -6 HP WikittyQuery query = new WikittyQueryMaker().notsw(Product.ELEMENT_FIELD_PRODUCT_NAME, "Harry").end(); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(29, results.getTotalResult()); } @Test public void testQueryParserNotsw() throws IOException { // FIXME echatellier 20120125 should work - assumeTrueSearchEngineCanRunTest(); +// assumeTrueSearchEngineCanRunTest(); // + 22 in init db importBooks(); // 13 importé = 35 // -6 HP WikittyQuery query = WikittyQueryParser.parse("Product.name!=Harry*"); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(29, results.getTotalResult()); } @Test public void testQueryMakerNotew() throws IOException { // FIXME echatellier 20120125 should work - assumeTrueSearchEngineCanRunTest(); +// assumeTrueSearchEngineCanRunTest(); // + 22 in init db importBooks(); // 13 importé = 35 // -1 sorciers WikittyQuery query = new WikittyQueryMaker().notew(Product.ELEMENT_FIELD_PRODUCT_NAME, "sorciers").end(); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(34, results.getTotalResult()); } @Test public void testQueryParserNotew() throws IOException { // FIXME echatellier 20120125 should work - assumeTrueSearchEngineCanRunTest(); +// assumeTrueSearchEngineCanRunTest(); // + 22 in init db importBooks(); // 13 importé = 35 // -1 sorciers WikittyQuery query = WikittyQueryParser.parse("Product.name!=*sorciers"); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(34, results.getTotalResult()); } @@ -1318,7 +1314,7 @@ public void testQueryMakerTrue() { // 22 in init db WikittyQuery query = new WikittyQueryMaker().rTrue().end(); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(22, results.getTotalResult()); } @@ -1326,7 +1322,7 @@ public void testQueryParseTrue() { // 22 in init db WikittyQuery query = WikittyQueryParser.parse("TRUE"); - WikittyQueryResult<Product> results = wikittyClient.findAllByQuery(Product.class, query); + WikittyQueryResult<String> results = wikittyClient.findAllByQuery(query); Assert.assertEquals(22, results.getTotalResult()); } @@ -1419,7 +1415,7 @@ @Test public void testSearchFacetSingleField() throws Exception { - assumeTrueSearchEngineCanRunTest(); // facets +// assumeTrueSearchEngineCanRunTest(); // facets importBooks(); @@ -1437,11 +1433,7 @@ Assert.assertEquals(1, result.getTopic(Product.ELEMENT_FIELD_PRODUCT_CATEGORY).get(3).getCount()); } - /** - * FIXME echatellier 20120131 fix test (fails with solr analyzer) - */ @Test - @Ignore public void testQueryMarkerWilcardEquals() { assumeTrueSearchEngineCanRunTest(); // wildcard @@ -1449,26 +1441,22 @@ WikittyQueryResult<Product> results1 = wikittyClient.findAllByQuery(Product.class, query1); Assert.assertEquals(1, results1.getTotalResult()); - WikittyQuery query2 = new WikittyQueryMaker().eq("*" + WikittyUtil.FQ_FIELD_NAME_SEPARATOR + Product.FIELD_PRODUCT_NAME, "Lanfeust").end(); - WikittyQueryResult<Product> results2 = wikittyClient.findAllByQuery(Product.class, query2); - Assert.assertEquals(1, results2.getTotalResult()); + // ???13 + WikittyQuery query4 = new WikittyQueryMaker().eq("*.name." + WikittyTypes.STRING, "Lanfeust").end(); + WikittyQueryResult<Product> results4 = wikittyClient.findAllByQuery(Product.class, query4); + Assert.assertEquals(1, results4.getTotalResult()); // ??? WikittyQuery query3 = new WikittyQueryMaker().eq("Product.name." + WikittyTypes.STRING, "Lanfeust").end(); WikittyQueryResult<Product> results3 = wikittyClient.findAllByQuery(Product.class, query3); Assert.assertEquals(1, results3.getTotalResult()); - // ???13 - WikittyQuery query4 = new WikittyQueryMaker().eq("*.name." + WikittyTypes.STRING, "Lanfeust").end(); - WikittyQueryResult<Product> results4 = wikittyClient.findAllByQuery(Product.class, query4); - Assert.assertEquals(1, results4.getTotalResult()); + WikittyQuery query2 = new WikittyQueryMaker().eq("*" + WikittyUtil.FQ_FIELD_NAME_SEPARATOR + Product.FIELD_PRODUCT_NAME, "Lanfeust").end(); + WikittyQueryResult<Product> results2 = wikittyClient.findAllByQuery(Product.class, query2); + Assert.assertEquals(1, results2.getTotalResult()); } - /** - * FIXME echatellier 20120131 fix test (fails with solr analyzer) - */ @Test - @Ignore public void testQueryParserWilcardEquals() { assumeTrueSearchEngineCanRunTest(); // wildcard @@ -1538,12 +1526,15 @@ */ @Test public void testSearchWithSort1() { - assumeTrueSearchEngineCanRunTest(); // sorting +// assumeTrueSearchEngineCanRunTest(); // sorting + // FIXME poussin 20120202 comment savoir que les objets sont bien trie ? + // Il y a toute sorte d'extension avec *.name et on ne recupere que + // les ID. WikittyQuery query1 = WikittyQueryParser.parse("*.name=*"); query1.setSortAscending(new ElementField("*.name")); - WikittyQueryResult<Product> results1 = wikittyClient.findAllByQuery(Product.class, query1); - Assert.assertEquals(15, results1.getTotalResult()); + WikittyQueryResult<String> results1 = wikittyClient.findAllByQuery(query1); + Assert.assertEquals(22, results1.getTotalResult()); } /**

1 0

r1413 - in trunk/wikitty-solr/src/main: java/org/nuiton/wikitty/storage/solr resources
by bpoussin＠users.nuiton.org 03 Feb '12

03 Feb '12

Author: bpoussin Date: 2012-02-03 19:26:20 +0100 (Fri, 03 Feb 2012) New Revision: 1413 Url: http://nuiton.org/repositories/revision/wikitty/1413 Log: Anomalie #1941: If entity use two extension with same field name, *.fieldname faild Anomalie #1942: Search with *.fieldname faild if this entity is in WikittyTreeNode Evolution #1943: Field *.fieldname must be sortable - deplacement de la creation des champs speciaux #ft.all, #all, *_sortable pour les mettre au plus tard (juste avant le commit vers solr) de cette facon on simule le comportement que devrait avoir solr s'il etait capable de faire des copies de champs en utilisant des regexp Modified: trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/SolrResource.java trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittySearchEngineSolr.java trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittySolrConstant.java trunk/wikitty-solr/src/main/resources/schema.xml Modified: trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/SolrResource.java =================================================================== --- trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/SolrResource.java 2012-02-03 18:08:40 UTC (rev 1412) +++ trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/SolrResource.java 2012-02-03 18:26:20 UTC (rev 1413) @@ -42,6 +42,8 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import org.apache.commons.lang3.StringUtils; +import org.nuiton.wikitty.WikittyUtil; /** * Use to plug solr indexation in JTA transaction. @@ -131,6 +133,7 @@ synchronized (this) { Collection<SolrInputDocument> docs = getAddedDocs().values(); if (!docs.isEmpty()) { + addAllSolrfield(docs); solrServer.add(docs); } List<String> ids = getDeletedDocs(); @@ -147,6 +150,95 @@ } } + /** + * On fait ici le travail de SolR car il ne sait pas faire des copies de + * champs avec des regexp evaluee. + * Ce que l'on voudrait: + * <li> "ExtName.FieldName_s" avec "*\.(*)_*" devient #all.$1 et #ft.all.$1 + * + * @param docs + */ + protected void addAllSolrfield(Collection<SolrInputDocument> docs) { + for (SolrInputDocument doc : docs) { + Collection<String> fieldnames = new ArrayList<String>(doc.getFieldNames()); + // on boucle une premiere fois pour supprimer les anciennes valeurs + // on ne peut pas le faire juste avant le add, car au final plusieurs + // champs pourrait devoir etre stocker dans le meme champs (si deux + // extension on un champs portant le meme nom. ex: Titi.name et Toto.name) + for (String solrFqFieldName : fieldnames) { + if (!solrFqFieldName.startsWith(WikittySolrConstant.SOLR_WIKITTY_PREFIX)) { + String fqfieldName = StringUtils.substringBeforeLast( + solrFqFieldName, WikittySolrConstant.SOLR_WIKITTY_SUFFIX); + + // #all.<fieldname> + // permet de faire des recherches inter extension sur un champs ayant + // le meme nom. ex:Person.name et User.name + // Quoi qu'il arrive pour le #all on utilise du multivalue + String solrAllFieldName = + WikittySolrConstant.SOLR_ALL_EXTENSIONS + + WikittyUtil.FQ_FIELD_NAME_SEPARATOR + + WikittyUtil.getFieldNameFromFQFieldName(solrFqFieldName); + + // idem mais un champs sur plusieurs extension peut avoir des types + // different, on ajoute donc un champs pour la recherche fulltext + String solrFulltextAllFieldName = + WikittySolrConstant.SOLR_FULLTEXT_ALL_EXTENSIONS + + WikittyUtil.FQ_FIELD_NAME_SEPARATOR + + WikittyUtil.getFieldNameFromFQFieldName(fqfieldName); + + // sortable solr field name for this field ex: myExt.myField_s_sortable + String solrFqFieldNameSortable = + solrFqFieldName + WikittySolrConstant.SUFFIX_SORTABLE; + + doc.remove(solrAllFieldName); // #all.myField_s + doc.remove(solrFulltextAllFieldName); // #fulltext.all.myField + doc.remove(solrFqFieldNameSortable); // myExt.myField_s_sortable + } + } + + for (String solrFqFieldName : fieldnames) { + if (!solrFqFieldName.startsWith(WikittySolrConstant.SOLR_WIKITTY_PREFIX) + && !solrFqFieldName.endsWith(WikittySolrConstant.SUFFIX_SORTABLE)) { + Object fieldValue = doc.getFieldValue(solrFqFieldName); + String fqfieldName = StringUtils.substringBeforeLast( + solrFqFieldName, WikittySolrConstant.SOLR_WIKITTY_SUFFIX); + + // #all.<fieldname> + // permet de faire des recherches inter extension sur un champs ayant + // le meme nom. ex:Person.name et User.name + // Quoi qu'il arrive pour le #all on utilise du multivalue + String solrAllFieldName = WikittySolrConstant.SOLR_ALL_EXTENSIONS + + WikittyUtil.FQ_FIELD_NAME_SEPARATOR + + WikittyUtil.getFieldNameFromFQFieldName(solrFqFieldName); + + // idem mais un champs sur plusieurs extension peut avoir des types + // different, on ajoute donc un champs pour la recherche fulltext + String solrFulltextAllFieldName = WikittySolrConstant.SOLR_FULLTEXT_ALL_EXTENSIONS + + WikittyUtil.FQ_FIELD_NAME_SEPARATOR + + WikittyUtil.getFieldNameFromFQFieldName(fqfieldName); + + // sortable solr field name for this field ex: myExt.myField_s_sortable + String solrFqFieldNameSortable = + solrFqFieldName + WikittySolrConstant.SUFFIX_SORTABLE; + + doc.addField(solrAllFieldName, fieldValue); + doc.addField(solrFulltextAllFieldName, fieldValue); + + // add sortable for field #all. + Object oneFieldValue = SolrUtil.getOneValue(fieldValue); + doc.setField(solrAllFieldName + WikittySolrConstant.SUFFIX_SORTABLE, + oneFieldValue); + // add sortable for field #ft.all. + doc.setField(solrFulltextAllFieldName + WikittySolrConstant.SUFFIX_SORTABLE, + oneFieldValue); + // add sortable for real field + doc.setField(solrFqFieldNameSortable, oneFieldValue); + } + } + } + + } + @Override public int rollback() { clear(); Modified: trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java =================================================================== --- trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java 2012-02-03 18:08:40 UTC (rev 1412) +++ trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java 2012-02-03 18:26:20 UTC (rev 1413) @@ -96,6 +96,47 @@ this.elementModifier = elementModifier; } + /** + * Impossible de trouver le bon analyser Solr pour la recherche fulltext. + * Solr meme si on lui dit de mettre en minuscule et d'enlever les accents + * le fait pour l'indexation, mais pas la recherche :(. On le fait donc pour + * lui ici tant que le bon analyser n'est pas trouve + * + * Apres recherche il semble que ce soit normal :( + * sur la page + * http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#Analyzers + * on peut lire + * <pre> + * On wildcard and fuzzy searches, no text analysis is performed on the search word. + * </pre> + * Donc dès qu'on met des '*' les transformations ne sont plus faites :( + * + * plus d'explication: + * <li> http://java.dzone.com/articles/what%E2%80%99s-lowercasing-wildcard + * <li> http://wiki.apache.org/solr/MultitermQueryAnalysis + * + * @param element2solr + * @param value + * @return + */ + private String fixSolrBug(String element2solr, String value) { + String result = value; + if (element2solr.equals(WikittySolrConstant.SOLR_FULLTEXT) || + element2solr.startsWith(WikittySolrConstant.SOLR_FULLTEXT_ALL_EXTENSIONS) || + element2solr.endsWith(WikittySolrConstant.SUFFIX_STRING_FULLTEXT)|| + element2solr.endsWith(WikittySolrConstant.SUFFIX_STRING_LOWERCASE)) { // is string + // TODO poussin 20120107 impossible de comprendre pourquoi il faut force + // la chaine en lower case, ca devrait etre le role de solr/lucene via + // la description du schema de mettre la chaine dans la bonne forme + // pour la recheche en fonction du type du champs souhaite (ici fulltext) + // tant que ca marche pas on force a la main + // idem pour les accents :( + result = StringUtils.stripAccents(result); + result = result.toLowerCase(); + } + return result; + } + private String element2solr(Element element) { String result = elementModifier.convertToSolr(tx, element); return result; @@ -253,8 +294,14 @@ @Override public boolean visitEnter(Between o) { - solrQuery += element2solr(o.getElement()) - + ":[" + evalConditionValue(o.getMin()) + " TO " + evalConditionValue(o.getMax()) + "]"; + String element2solr = element2solr(o.getElement()); + String min = evalConditionValue(o.getMin()); + String max = evalConditionValue(o.getMax()); + + min = fixSolrBug(element2solr, min); + max = fixSolrBug(element2solr, max); + + solrQuery += element2solr + ":[" + min + " TO " + max + "]"; return false; } @@ -271,14 +318,17 @@ @Override public boolean visitEnter(ContainsAll o) { - solrQuery += element2solr(o.getElement()) - + ":("; + String element2solr = element2solr(o.getElement()); + + solrQuery += element2solr + ":("; + String sep = ""; for(String e : evalConditionValueAsList(o.getValues())) { - solrQuery += sep + e; + solrQuery += sep + fixSolrBug(element2solr, e); sep = " AND "; } solrQuery += ")"; + return false; } @@ -295,11 +345,12 @@ @Override public boolean visitEnter(ContainsOne o) { - solrQuery += element2solr(o.getElement()) - + ":("; + String element2solr = element2solr(o.getElement()); + + solrQuery += element2solr + ":("; String sep = ""; for(String e : evalConditionValueAsList(o.getValues())) { - solrQuery += sep + e; + solrQuery += sep + fixSolrBug(element2solr, e); sep = " OR "; } solrQuery += ")"; @@ -320,14 +371,17 @@ @Override public boolean visitEnter(Equals o) { String element2solr = element2solr(o.getElement()); + String value = evalConditionValue(o.getValue()); + if (element2solr.endsWith(WikittySolrConstant.SUFFIX_STRING)) { // is string if (o.isIgnoreCaseAndAccent()) { element2solr += WikittySolrConstant.SUFFIX_STRING_LOWERCASE; } } - solrQuery += element2solr - + ":" + evalConditionValue(o.getValue()); + value = fixSolrBug(element2solr, value); + + solrQuery += element2solr + ":" + value; return false; } @@ -340,14 +394,18 @@ @Override public boolean visitEnter(NotEquals o) { String element2solr = element2solr(o.getElement()); + String value = evalConditionValue(o.getValue()); + if (element2solr.endsWith(WikittySolrConstant.SUFFIX_STRING)) { // is string if (o.isIgnoreCaseAndAccent()) { element2solr += WikittySolrConstant.SUFFIX_STRING_LOWERCASE; } } + + value= fixSolrBug(element2solr, value); - solrQuery += "-" + element2solr - + ":" + evalConditionValue(o.getValue()); + solrQuery += "-" + element2solr + ":" + value; + return false; } @@ -358,8 +416,12 @@ @Override public boolean visitEnter(Greater o) { - solrQuery += element2solr(o.getElement()) - + ":{" + evalConditionValue(o.getValue()) + " TO *}"; + String element2solr = element2solr(o.getElement()); + String value = evalConditionValue(o.getValue()); + + value = fixSolrBug(element2solr, value); + + solrQuery += element2solr + ":{" + value + " TO *}"; return false; } @@ -370,8 +432,12 @@ @Override public boolean visitEnter(GreaterOrEquals o) { - solrQuery += element2solr(o.getElement()) - + ":[" + evalConditionValue(o.getValue()) + " TO *]"; + String element2solr = element2solr(o.getElement()); + String value = evalConditionValue(o.getValue()); + + value = fixSolrBug(element2solr, value); + + solrQuery += element2solr + ":[" + value + " TO *]"; return false; } @@ -383,15 +449,9 @@ @Override public boolean visitEnter(Keyword o) { String value = evalConditionValue(o.getValue()); - // TODO poussin 20120107 impossible de comprendre pourquoi il faut force - // la chaine en lower case, ca devrait etre le role de solr/lucene via - // la description du schema de mettre la chaine dans la bonne forme - // pour la recheche en fonction du type du champs souhaite (ici fulltext) - // tant que ca marche pas on force a la main - value = value.toLowerCase(); - // idem pour les accents :( - value = StringUtils.stripAccents(value); + value = fixSolrBug(WikittySolrConstant.SOLR_FULLTEXT, value); + // pour keyword on ajout automatiquement les *, sinon il faut faire un like value = "*" + value + "*"; solrQuery += value; @@ -405,8 +465,12 @@ @Override public boolean visitEnter(Less o) { - solrQuery += element2solr(o.getElement()) - + ":{* TO " + evalConditionValue(o.getValue()) + "}"; + String element2solr = element2solr(o.getElement()); + String value = evalConditionValue(o.getValue()); + + value = fixSolrBug(element2solr, value); + + solrQuery += element2solr + ":{* TO " + value + "}"; return false; } @@ -417,8 +481,12 @@ @Override public boolean visitEnter(LessOrEquals o) { - solrQuery += element2solr(o.getElement()) - + ":[* TO " + evalConditionValue(o.getValue()) + "]"; + String element2solr = element2solr(o.getElement()); + String value = evalConditionValue(o.getValue()); + + value = fixSolrBug(element2solr, value); + + solrQuery += element2solr + ":[* TO " + value + "]"; return false; } @@ -430,20 +498,13 @@ @Override public boolean visitEnter(Like o) { String element2solr = element2solr(o.getElement()); + String value = evalConditionValue(o.getValue()); + if (element2solr.endsWith(WikittySolrConstant.SUFFIX_STRING)) { // is string - // FIXME poussin 20111230 check la config solr et peut etre utiliser le lowercase ? element2solr += WikittySolrConstant.SUFFIX_STRING_FULLTEXT; } - String value = evalConditionValue(o.getValue()); - // TODO poussin 20120107 impossible de comprendre pourquoi il faut force - // la chaine en lower case, ca devrait etre le role de solr/lucene via - // la description du schema de mettre la chaine dans la bonne forme - // pour la recheche en fonction du type du champs souhaite (ici fulltext) - // tant que ca marche pas on force a la main - value = value.toLowerCase(); - // idem pour les accents :( - value = StringUtils.stripAccents(value); + value = fixSolrBug(element2solr, value); solrQuery += element2solr + ":" + value; return false; @@ -457,20 +518,13 @@ @Override public boolean visitEnter(Unlike o) { String element2solr = element2solr(o.getElement()); + String value = evalConditionValue(o.getValue()); + if (element2solr.endsWith(WikittySolrConstant.SUFFIX_STRING)) { // is string - // FIXME poussin 20111230 check la config solr et peut etre utiliser le lowercase ? element2solr += WikittySolrConstant.SUFFIX_STRING_FULLTEXT; } - String value = evalConditionValue(o.getValue()); - // TODO poussin 20120107 impossible de comprendre pourquoi il faut force - // la chaine en lower case, ca devrait etre le role de solr/lucene via - // la description du schema de mettre la chaine dans la bonne forme - // pour la recheche en fonction du type du champs souhaite (ici fulltext) - // tant que ca marche pas on force a la main - value = value.toLowerCase(); - // idem pour les accents :( - value = StringUtils.stripAccents(value); + value = fixSolrBug(element2solr, value); solrQuery += "-" + element2solr + ":" + value; return false; Modified: trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittySearchEngineSolr.java =================================================================== --- trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittySearchEngineSolr.java 2012-02-03 18:08:40 UTC (rev 1412) +++ trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittySearchEngineSolr.java 2012-02-03 18:26:20 UTC (rev 1413) @@ -939,48 +939,17 @@ } else { String solrFqFieldName; -// FIXME REMOVE IT if search on multivalued work with new hack (specific sortable field -// if (collection) { -// solrFqFieldName = SolrUtil.getSolrCollectionFieldName(fqfieldName, type); -// } else { // add suffix like _s for string type ex: myExt.myField_s solrFqFieldName = SolrUtil.getSolrFieldName(fqfieldName, type); -// } - // #all.<fieldname> - // permet de faire des recherches inter extension sur un champs ayant - // le meme nom. ex:Person.name et User.name - // Quoi qu'il arrive pour le #all on utilise du multivalue -// FIXME REMOVE IT if search on multivalued work with new hack (specific sortable field -// String solrAllFqFieldName = SolrUtil.getSolrCollectionFieldName(fqfieldName, type); - String solrAllFieldName = SOLR_ALL_EXTENSIONS - + WikittyUtil.FQ_FIELD_NAME_SEPARATOR - + WikittyUtil.getFieldNameFromFQFieldName(solrFqFieldName); - - // idem mais un champs sur plusieurs extension peut avoir des types - // different, on ajoute donc un champs pour la recherche fulltext - String solrFulltextAllFieldName = SOLR_FULLTEXT_ALL_EXTENSIONS - + WikittyUtil.FQ_FIELD_NAME_SEPARATOR - + WikittyUtil.getFieldNameFromFQFieldName(fqfieldName); - String solrNullFieldFqFieldName = SOLR_NULL_FIELD + fqfieldName; - // sortable solr field name for this field ex: myExt.myField_s_sortable - String solrFqFieldNameSortable = solrFqFieldName + SUFFIX_SORTABLE; - doc.remove(solrFqFieldName); // myExt.myField_s doc.remove(solrNullFieldFqFieldName); // #null_field-myExt.myField - doc.remove(solrAllFieldName); // #all.myField_s - doc.remove(solrFulltextAllFieldName); // #fulltext.all.myField - doc.remove(solrFqFieldNameSortable); // myExt.myField_s_sortable String solrNullFieldFqFieldNameValue = "true"; if(fieldValue != null) { doc.addField(solrFqFieldName, fieldValue); - doc.addField(solrAllFieldName, fieldValue); - doc.addField(solrFulltextAllFieldName, fieldValue); - Object oneFieldValue = SolrUtil.getOneValue(fieldValue); - doc.addField(solrFqFieldNameSortable, oneFieldValue); solrNullFieldFqFieldNameValue = "false"; if (log.isTraceEnabled()) { log.trace(String.format("index field '%s' with value '%s'", @@ -1042,7 +1011,7 @@ SolrInputDocument doc = new SolrInputDocument(); addToIndexDocument(doc, w, SOLR_ID); addToIndexDocument(doc, w, SOLR_EXTENSIONS); - // iter sur tous les champs et pas seulement sont qui ont une valeur + // iter sur tous les champs et pas seulement ceux qui ont une valeur // pour pouvoir les indexer comme des champs a null for (String fqfieldName : w.getAllFieldNames()) { addToIndexDocument(doc, w, fqfieldName); Modified: trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittySolrConstant.java =================================================================== --- trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittySolrConstant.java 2012-02-03 18:08:40 UTC (rev 1412) +++ trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittySolrConstant.java 2012-02-03 18:26:20 UTC (rev 1413) @@ -80,27 +80,28 @@ used for facetisation. Le champs est cree dynamiquement par solr via schema.xml */ static final public String TREENODE_ATTACHED_ALL = TREENODE_PREFIX + "attached-all"; - static final public String SUFFIX_BINARY = "_bi"; - static final public String SUFFIX_BOOLEAN = "_b"; - static final public String SUFFIX_NUMERIC = "_d"; - static final public String SUFFIX_DATE = "_dt"; - static final public String SUFFIX_STRING = "_s"; - static final public String SUFFIX_WIKITTY = "_w"; + static final public String SOLR_WIKITTY_SUFFIX = "_"; + static final public String SUFFIX_BINARY = SOLR_WIKITTY_SUFFIX + "bi"; + static final public String SUFFIX_BOOLEAN = SOLR_WIKITTY_SUFFIX + "b"; + static final public String SUFFIX_NUMERIC = SOLR_WIKITTY_SUFFIX + "d"; + static final public String SUFFIX_DATE = SOLR_WIKITTY_SUFFIX + "dt"; + static final public String SUFFIX_STRING = SOLR_WIKITTY_SUFFIX + "s"; + static final public String SUFFIX_WIKITTY = SOLR_WIKITTY_SUFFIX + "w"; - static final public String SUFFIX_SORTABLE = "_sortable"; + static final public String SUFFIX_SORTABLE = SOLR_WIKITTY_SUFFIX + "sortable"; // FIXME REMOVE IT if search on multivalued work with new hack (specific sortable field -// static final public String SUFFIX_BINARY_MULTIVALUED = "_bim"; -// static final public String SUFFIX_BOOLEAN_MULTIVALUED = "_bm"; -// static final public String SUFFIX_NUMERIC_MULTIVALUED = "_dm"; -// static final public String SUFFIX_DATE_MULTIVALUED = "_dtm"; -// static final public String SUFFIX_STRING_MULTIVALUED = "_sm"; -// static final public String SUFFIX_WIKITTY_MULTIVALUED = "_wm"; +// static final public String SUFFIX_BINARY_MULTIVALUED = SOLR_WIKITTY_SUFFIX + "bim"; +// static final public String SUFFIX_BOOLEAN_MULTIVALUED = SOLR_WIKITTY_SUFFIX + "bm"; +// static final public String SUFFIX_NUMERIC_MULTIVALUED = SOLR_WIKITTY_SUFFIX + "dm"; +// static final public String SUFFIX_DATE_MULTIVALUED = SOLR_WIKITTY_SUFFIX + "dtm"; +// static final public String SUFFIX_STRING_MULTIVALUED = SOLR_WIKITTY_SUFFIX + "sm"; +// static final public String SUFFIX_WIKITTY_MULTIVALUED = SOLR_WIKITTY_SUFFIX + "wm"; - static final public String SUFFIX_STRING_LOWERCASE = "_c"; - static final public String SUFFIX_STRING_FULLTEXT = "_t"; + static final public String SUFFIX_STRING_LOWERCASE = SOLR_WIKITTY_SUFFIX + "c"; + static final public String SUFFIX_STRING_FULLTEXT = SOLR_WIKITTY_SUFFIX + "t"; // FIXME REMOVE IT if search on multivalued work with new hack (specific sortable field -// public static final String SUFFIX_STRING_FULLTEXT_LOWERCASE = "_cm"; -// public static final String SUFFIX_STRING_FULLTEXT_MULTIVALUED = "_tm"; +// public static final String SUFFIX_STRING_FULLTEXT_LOWERCASE = SOLR_WIKITTY_SUFFIX + "cm"; +// public static final String SUFFIX_STRING_FULLTEXT_MULTIVALUED = SOLR_WIKITTY_SUFFIX + "tm"; } Modified: trunk/wikitty-solr/src/main/resources/schema.xml =================================================================== --- trunk/wikitty-solr/src/main/resources/schema.xml 2012-02-03 18:08:40 UTC (rev 1412) +++ trunk/wikitty-solr/src/main/resources/schema.xml 2012-02-03 18:26:20 UTC (rev 1413) @@ -74,14 +74,19 @@ Duplicate tokens at the same position (which may result from Stemmed Synonyms or WordDelim parts) are removed. --> +  <fieldType name="text" class="solr.TextField" positionIncrementGap="100"> <analyzer> +  + <tokenizer class="solr.WhitespaceTokenizerFactory"/> +  <filter class="solr.ASCIIFoldingFilterFactory"/>  -  - <tokenizer class="solr.WhitespaceTokenizerFactory"/>  <filter class="solr.PatternReplaceFilterFactory" pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/> @@ -95,10 +100,16 @@    - <filter class="solr.WordDelimiterFilterFactory" splitOnCaseChange="1" - splitOnNumerics="1" generateWordParts="1" - generateNumberParts="1" catenateWords="1" catenateNumbers="1" - catenateAll="1" preserveOriginal="1" protected="protwords.txt"/> + <filter class="solr.WordDelimiterFilterFactory" + splitOnCaseChange="1" + splitOnNumerics="1" + generateWordParts="1" + generateNumberParts="1" + catenateWords="1" + catenateNumbers="1" + catenateAll="1" + preserveOriginal="1" + protected="protwords.txt"/>  <filter class="solr.StopFilterFactory" ignoreCase="1" words="stopwords_fr.txt" enablePositionIncrements="true"/> @@ -139,13 +150,17 @@  - <dynamicField name="*_sortable" type="string" indexed="true" stored="false" multiValued="false"/> + <dynamicField name="*_sortable" type="string" indexed="true" stored="true" multiValued="false"/> <dynamicField name="#tree.attached.*" type="string" indexed="true" stored="true" multiValued="true"/> <dynamicField name="#null_field-*" type="boolean" indexed="true" stored="true" multiValued="false"/> +   <dynamicField name="#ft.all.*" type="text" indexed="true" stored="false" multiValued="true"/> +  +  <dynamicField name="*_s_c" type="string_lc" indexed="true" stored="false" multiValued="true"/>

1 0

r1412 - in trunk/wikitty-api/src/main/java/org/nuiton/wikitty: . query storage
by bpoussin＠users.nuiton.org 03 Feb '12

03 Feb '12

Author: bpoussin Date: 2012-02-03 19:08:40 +0100 (Fri, 03 Feb 2012) New Revision: 1412 Url: http://nuiton.org/repositories/revision/wikitty/1412 Log: Evolution #1944: Enhance in memory implementation Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/WikittyUtil.java trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/FacetTopicCountComparator.java trunk/wikitty-api/src/main/java/org/nuiton/wikitty/storage/WikittySearchEngineInMemory.java Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/WikittyUtil.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/WikittyUtil.java 2012-02-03 18:00:37 UTC (rev 1411) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/WikittyUtil.java 2012-02-03 18:08:40 UTC (rev 1412) @@ -1009,8 +1009,12 @@ * TODO poussin 20101208 redondant code with {@link WikittyExtension#extractFieldName(java.lang.String)} */ public static String getFieldNameFromFQFieldName(String fqFieldName) { - String[] fqFieldElements = fqFieldName.split(FQ_FIELD_NAME_SEPARATOR_REGEX); - return fqFieldElements[1]; + try { + String[] fqFieldElements = fqFieldName.split(FQ_FIELD_NAME_SEPARATOR_REGEX); + return fqFieldElements[1]; + } catch(ArrayIndexOutOfBoundsException eee) { + throw new WikittyException("Field is not fully qualified field:" + fqFieldName, eee); + } } /** given names of extension and field, return a fully qualified field name */ Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/FacetTopicCountComparator.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/FacetTopicCountComparator.java 2012-02-03 18:00:37 UTC (rev 1411) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/FacetTopicCountComparator.java 2012-02-03 18:08:40 UTC (rev 1412) @@ -55,7 +55,7 @@ public int compare(FacetTopic o1, FacetTopic o2) { int thisVal = o1.getCount(); int anotherVal = o2.getCount(); - // code recupere de Integer.compareTo - return (thisVal<anotherVal ? -1 : (thisVal==anotherVal ? 0 : 1)); + // code recupere de Integer.compareTo (mais inverse pour les 1/-1) + return (thisVal<anotherVal ? 1 : (thisVal==anotherVal ? 0 : -1)); } } Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/storage/WikittySearchEngineInMemory.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/storage/WikittySearchEngineInMemory.java 2012-02-03 18:00:37 UTC (rev 1411) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/storage/WikittySearchEngineInMemory.java 2012-02-03 18:08:40 UTC (rev 1412) @@ -191,8 +191,10 @@ for (org.nuiton.wikitty.entities.Element e : query.getFacetField()) { String fqf = e.getValue(); Object value = w.getFqField(fqf); - topic.get(fqf).add(value); - result = true; + if (value != null) { + topic.get(fqf).add(value); + result = true; + } } // create facet query @@ -203,6 +205,7 @@ facetName = q.getCondition().toString(); } topic.get(facetName).add(facetName); + result = true; } } @@ -380,35 +383,6 @@ } }; - static private Predicate NotEqualsPredicate = new Predicate() { - public boolean check(FieldType type, Collection values, Collection expected) { - boolean result = true; - if (values != null && expected.size() > 0) { - // gestion des type STRING differement car il peut y avoir des '*' - if (type != null && type.getType() == WikittyTypes.STRING) { - Iterator i = expected.iterator(); - String exp = String.valueOf(i.next()); - for (Object fieldValue : values) { - String val = String.valueOf(fieldValue); - result = !matchString(val, exp, false); - if (!result) { - // si une des valeurs correspond, on retourne true - break; - } - } - } else { - expected = CollectionUtils.subtract(expected, values); - // si lorsqu'on retire tous les elements en commun avec la valeur - // du champs il ne reste plus rien, c'est que le equals est - // vrai et donc le not equals est faux - result = !expected.isEmpty(); - } - } - return result; - - } - }; - static private Predicate EqualsIgnoreCaseAndAccentPredicate = new Predicate() { public boolean check(FieldType type, Collection values, Collection expected) { boolean result = false; @@ -438,35 +412,6 @@ } }; - static private Predicate NotEqualsIgnoreCaseAndAccentPredicate = new Predicate() { - public boolean check(FieldType type, Collection values, Collection expected) { - boolean result = true; - if (values != null && expected.size() > 0) { - // gestion des type STRING differement car il peut y avoir des '*' - if (type != null && type.getType() == WikittyTypes.STRING) { - Iterator i = expected.iterator(); - String exp = String.valueOf(i.next()); - for (Object fieldValue : values) { - String val = String.valueOf(fieldValue); - result = !matchString(val, exp, true); - if (!result) { - // si une des valeurs correspond, on retourne true - break; - } - } - } else { - expected = CollectionUtils.subtract(expected, values); - // si lorsqu'on retire tous les elements en commun avec la valeur - // du champs il ne reste plus rien, c'est que le equals est - // vrai et donc le not equals est faux - result = !expected.isEmpty(); - } - } - return result; - - } - }; - static private Predicate GreaterPredicate = new Predicate() { public boolean check(FieldType type, Collection values, Collection expected) { boolean result = false; @@ -607,27 +552,6 @@ } }; - static private Predicate UnlikePredicate = new Predicate() { - public boolean check(FieldType type, Collection values, Collection expected) { - boolean result = false; - - if (values != null && expected.size() > 0) { - // si on a pas une valeur, c'est pas bon :( - Iterator i = expected.iterator(); - String exp = String.valueOf(i.next()); - for (Object fieldValue : values) { - String val = String.valueOf(fieldValue); - result = !matchString(val, exp, true); - if (result) { - // si une des valeurs correspond, on retourne true - break; - } - } - } - return result; - } - }; - static private Predicate NullPredicate = new Predicate() { public boolean check(FieldType type, Collection values, Collection expected) { boolean result = values == null || values.contains(null); @@ -654,7 +578,7 @@ * @return true if sub match s */ static private boolean matchString(String s, String sub, boolean ignoreCaseAndAccent) { - boolean result = false; + boolean result; if (sub.startsWith("*") && sub.endsWith("*")) { sub = StringUtils.substring(sub, 1, -1); if (ignoreCaseAndAccent) { @@ -757,17 +681,17 @@ * * @return une nouvelle collection avec les elements dans le bon type */ - protected Collection convert(FieldType type, Object o) { - Collection result; + protected Collection<Object> convert(FieldType type, Object o) { + Collection<Object> result; try { if (o instanceof Collection) { // order of collection must be maintained, for that use LinkedHashSet if (type == null) { // if type is null don't change type (in case of id or extension // or when we know object is already in right type) - result = new LinkedHashSet((Collection)o); + result = new LinkedHashSet<Object>((Collection<Object>)o); } else { - result = new LinkedHashSet(); + result = new LinkedHashSet<Object>(); for (Object v : (Collection)o) { result.add(type.getContainedValidObject(v)); } @@ -812,7 +736,7 @@ if (contained != null) { // si on a reussi a convertir dans le bon type on fait la verif Collection values; - values = (Collection)fieldValues.get(fqf); + values = fieldValues.get(fqf); result = predicate.check(type, values, contained); if (result) { @@ -1094,10 +1018,10 @@ boolean result = false; if (o.isIgnoreCaseAndAccent()) { - result = check(NotEqualsIgnoreCaseAndAccentPredicate, o.getElement(), + result = !check(EqualsIgnoreCaseAndAccentPredicate, o.getElement(), evalConditionValue(o.getValue())); } else { - result = check(NotEqualsPredicate, o.getElement(), + result = !check(EqualsPredicate, o.getElement(), evalConditionValue(o.getValue())); } @@ -1225,7 +1149,7 @@ public boolean visitEnter(Unlike o) { boolean result = false; - result = check(UnlikePredicate, o.getElement(), + result = !check(LikePredicate, o.getElement(), evalConditionValue(o.getValue())); evalStack.push(result);

1 0

r1411 - trunk/wikitty-api/src/main/java/org/nuiton/wikitty
by bpoussin＠users.nuiton.org 03 Feb '12

03 Feb '12

Author: bpoussin Date: 2012-02-03 19:00:37 +0100 (Fri, 03 Feb 2012) New Revision: 1411 Url: http://nuiton.org/repositories/revision/wikitty/1411 Log: Anomalie #1940: findByXXX with entity class in argument don't add filter condition on extension of this entity Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/WikittyClient.java Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/WikittyClient.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/WikittyClient.java 2012-02-01 20:45:50 UTC (rev 1410) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/WikittyClient.java 2012-02-03 18:00:37 UTC (rev 1411) @@ -589,7 +589,7 @@ List<WikittyQueryResult<E>> result = null; List<WikittyQuery> serviceQueries; if (queries != null) { - if (clazz.isAssignableFrom(BusinessEntity.class)) { + if (BusinessEntity.class.isAssignableFrom(clazz)) { // on demande un business entity donc on modifie // pas les criteres pour ajouter les contraintes sur les // extensions

1 0

r1410 - trunk/wikitty-solr/src/main/resources
by echatellier＠users.nuiton.org 01 Feb '12

01 Feb '12

Author: echatellier Date: 2012-02-01 21:45:50 +0100 (Wed, 01 Feb 2012) New Revision: 1410 Url: http://nuiton.org/repositories/revision/wikitty/1410 Log: update schema version Modified: trunk/wikitty-solr/src/main/resources/schema.xml Modified: trunk/wikitty-solr/src/main/resources/schema.xml =================================================================== --- trunk/wikitty-solr/src/main/resources/schema.xml 2012-02-01 19:19:16 UTC (rev 1409) +++ trunk/wikitty-solr/src/main/resources/schema.xml 2012-02-01 20:45:50 UTC (rev 1410) @@ -31,10 +31,10 @@ http://wiki.apache.org/solr/SchemaXml --> -<schema name="wikitty" version="1.1"> +<schema name="wikitty" version="1.4"> <types> -  +  <fieldtype name="binary" stored="false" indexed="false" class="solr.StrField" />  @@ -54,12 +54,10 @@  <fieldType name="string" class="solr.StrField" - compressThreshold="1000" sortMissingLast="true" omitNorms="true"/>  <fieldType name="string_lc" class="solr.TextField" - compressThreshold="1000" sortMissingLast="true"> <analyzer>  <tokenizer class="solr.StandardTokenizerFactory"/> @@ -77,7 +75,6 @@ WordDelim parts) are removed. --> <fieldType name="text" class="solr.TextField" - compressThreshold="1000" positionIncrementGap="100"> <analyzer> <filter class="solr.ASCIIFoldingFilterFactory"/> @@ -93,7 +90,6 @@ <filter class="solr.LengthFilterFactory" min="1" max="100"/>  - <filter class="solr.LowerCaseFilterFactory"/>

1 0

r1409 - in trunk: wikitty-api/src/test/java/org/nuiton/wikitty wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr wikitty-solr/src/main/resources
by bpoussin＠users.nuiton.org 01 Feb '12

01 Feb '12

Author: bpoussin Date: 2012-02-01 20:19:16 +0100 (Wed, 01 Feb 2012) New Revision: 1409 Url: http://nuiton.org/repositories/revision/wikitty/1409 Log: - Dans schema.xml suppression de la difference d'analyser entre indexer et query - pour les recherches fulltext, vu que solr est pas capable de faire le travail, on le fait pour lui: suppression des accents et mise en lower case Modified: trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java trunk/wikitty-solr/src/main/resources/schema.xml Modified: trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java =================================================================== --- trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java 2012-02-01 18:49:01 UTC (rev 1408) +++ trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java 2012-02-01 19:19:16 UTC (rev 1409) @@ -1609,7 +1609,6 @@ * FIXME echatellier 20120131 fix test (fails with solr analyzer) */ @Test - @Ignore public void testSearchWithAccent() { WikittyGroupImpl group = new WikittyGroupImpl(); group.setName("coucou"); Modified: trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java =================================================================== --- trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java 2012-02-01 18:49:01 UTC (rev 1408) +++ trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java 2012-02-01 19:19:16 UTC (rev 1409) @@ -27,6 +27,7 @@ import java.util.ArrayList; import java.util.List; import org.apache.commons.lang.ClassUtils; +import org.apache.commons.lang3.StringUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.nuiton.wikitty.WikittyException; @@ -388,6 +389,8 @@ // pour la recheche en fonction du type du champs souhaite (ici fulltext) // tant que ca marche pas on force a la main value = value.toLowerCase(); + // idem pour les accents :( + value = StringUtils.stripAccents(value); // pour keyword on ajout automatiquement les *, sinon il faut faire un like value = "*" + value + "*"; @@ -439,6 +442,8 @@ // pour la recheche en fonction du type du champs souhaite (ici fulltext) // tant que ca marche pas on force a la main value = value.toLowerCase(); + // idem pour les accents :( + value = StringUtils.stripAccents(value); solrQuery += element2solr + ":" + value; return false; @@ -464,6 +469,8 @@ // pour la recheche en fonction du type du champs souhaite (ici fulltext) // tant que ca marche pas on force a la main value = value.toLowerCase(); + // idem pour les accents :( + value = StringUtils.stripAccents(value); solrQuery += "-" + element2solr + ":" + value; return false; Modified: trunk/wikitty-solr/src/main/resources/schema.xml =================================================================== --- trunk/wikitty-solr/src/main/resources/schema.xml 2012-02-01 18:49:01 UTC (rev 1408) +++ trunk/wikitty-solr/src/main/resources/schema.xml 2012-02-01 19:19:16 UTC (rev 1409) @@ -79,7 +79,7 @@ <fieldType name="text" class="solr.TextField" compressThreshold="1000" positionIncrementGap="100"> - <analyzer type="index"> + <analyzer> <filter class="solr.ASCIIFoldingFilterFactory"/>  @@ -115,43 +115,6 @@  <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> </analyzer> - - <analyzer type="query"> - <filter class="solr.ASCIIFoldingFilterFactory"/> - -  -  - <tokenizer class="solr.WhitespaceTokenizerFactory"/> -  - <filter class="solr.PatternReplaceFilterFactory" - pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/> -  - <filter class="solr.LengthFilterFactory" min="1" max="100"/> -  - <filter class="solr.LowerCaseFilterFactory"/> -  -  -  - <filter class="solr.WordDelimiterFilterFactory" splitOnCaseChange="1" - splitOnNumerics="1" stemEnglishPossessive="1" - generateWordParts="1" - generateNumberParts="1" catenateWords="1" catenateNumbers="1" - catenateAll="1" preserveOriginal="1" protected="protwords.txt"/> -  - <filter class="solr.StopFilterFactory" ignoreCase="1" - words="stopwords_fr.txt" enablePositionIncrements="true"/> -  - <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" - ignoreCase="true" expand="true"/> -  -  - -  - <filter class="solr.RemoveDuplicatesTokenFilterFactory"/> - </analyzer> </fieldType>

1 0

r1408 - in trunk: wikitty-api/src/main/java/org/nuiton/wikitty/query wikitty-api/src/main/java/org/nuiton/wikitty/query/conditions wikitty-api/src/main/java/org/nuiton/wikitty/storage wikitty-api/src/test/java/org/nuiton/wikitty wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr
by bpoussin＠users.nuiton.org 01 Feb '12

01 Feb '12

Author: bpoussin Date: 2012-02-01 19:49:01 +0100 (Wed, 01 Feb 2012) New Revision: 1408 Url: http://nuiton.org/repositories/revision/wikitty/1408 Log: Evolution #1932: (Not)Equals must be ignore case and ignore accent if needed Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryMaker.java trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryParser.java trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryVisitorCopy.java trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryVisitorToString.java trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/conditions/Equals.java trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/conditions/NotEquals.java trunk/wikitty-api/src/main/java/org/nuiton/wikitty/storage/WikittySearchEngineInMemory.java trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryMaker.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryMaker.java 2012-02-01 09:33:28 UTC (rev 1407) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryMaker.java 2012-02-01 18:49:01 UTC (rev 1408) @@ -456,6 +456,39 @@ } /** + * @see {@link Equals} + */ + public WikittyQueryMaker eqIgnoreCaseAndAccent(Element element) { + addCondition(new Equals(element, true)); + return this; + } + + /** + * Equals. + * + * Restrict search so that the field value equals the parameter. + * + * You might use patterns in your equality. + * + * @param element the field on which the search is made + * @param value the value the element must be equals to + * @return {@code this} + * @see {@link Equals} + */ + public WikittyQueryMaker eqIgnoreCaseAndAccent(String fqfield, Object value) { + return eqIgnoreCaseAndAccent(new ElementField(fqfield), value); + } + + /** + * @see {@link Equals} + */ + public WikittyQueryMaker eqIgnoreCaseAndAccent(Element element, Object value) { + ConditionValue s = convertToConditionValue(value); + addCondition(new Equals(element, s, true)); + return this; + } + + /** * Extension equals. * * Restrict search to wikitties that got the extension in parameter. @@ -537,6 +570,38 @@ } /** + * @see {@link NotEquals} + */ + public WikittyQueryMaker neIgnoreCaseAndAccent(Element element) { + addCondition(new NotEquals(element, true)); + return this; + } + + /** + * Not equals. + * + * Restrict search to elements that are not equals to the value given in + * parameter. + * + * @param fqfield the element on which the restriction is put + * @param value the value the element must not be equals to. + * @return {@code this} with the {@code neq} restriction added. + * @see {@link NotEquals} + */ + public WikittyQueryMaker neIgnoreCaseAndAccent(String fqfield, Object value) { + return neIgnoreCaseAndAccent(new ElementField(fqfield), value); + } + + /** + * @see {@link NotEquals} + */ + public WikittyQueryMaker neIgnoreCaseAndAccent(Element element, Object value) { + ConditionValue s = convertToConditionValue(value); + addCondition(new NotEquals(element, s, true)); + return this; + } + + /** * Extension not equals. * * Restrict search to wikitties that do not get the extension given in Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryParser.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryParser.java 2012-02-01 09:33:28 UTC (rev 1407) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryParser.java 2012-02-01 18:49:01 UTC (rev 1408) @@ -106,6 +106,7 @@ public static final String CURLY_BRACKET_CLOSE = "}"; public static final String CURLY_BRACKET_OPEN = "{"; public static final String EQUALS = "="; + public static final String EQUALS_IGNORE_CASE_AND_ACCENT = "~"; public static final String GREATER = ">"; public static final String GREATER_OR_EQUALS = ">="; public static final String LESS = "<"; @@ -113,6 +114,7 @@ public static final String LIKE = "LIKE"; public static final String NOT = "NOT"; public static final String NOT_EQUALS = "!="; + public static final String NOT_EQUALS_IGNORE_CASE_AND_ACCENT = "!~"; public static final String OR = "OR"; public static final String BRACKET_CLOSE = ")"; public static final String BRACKET_OPEN = "("; @@ -291,7 +293,8 @@ not(), isNull(), isNotNull(), select(), greatereq(), lesseq(), between(), containsAll(), containsOne(), - eq(), neq(), less(), greater(), like(), notlike(), + eq(), neq(), eqIgnoreCaseAndAccent(), neqIgnoreCaseAndAccent(), + less(), greater(), like(), notlike(), rTrue(), rFalse(), keyword() ); } @@ -337,6 +340,24 @@ push(new NotEquals(toElement(pop(1).toString()), (ConditionValue)pop()))); } + /** + * gere eq, startsWith, endsWith, isNull + * @return + */ + Rule eqIgnoreCaseAndAccent() { + return Sequence(field(), push(match()), space(), EQUALS_IGNORE_CASE_AND_ACCENT, space(), value(), + push(new Equals(toElement(pop(1).toString()), (ConditionValue)pop(), true))); + } + + /** + * gere eq, isNull + * @return + */ + Rule neqIgnoreCaseAndAccent() { + return Sequence(field(), push(match()), space(), NOT_EQUALS_IGNORE_CASE_AND_ACCENT, space(), value(), + push(new NotEquals(toElement(pop(1).toString()), (ConditionValue)pop(), true))); + } + Rule less() { return Sequence(field(), push(match()), space(), LESS, space(), value(), push(new Less(toElement(pop(1).toString()), (ConditionValue)pop()))); Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryVisitorCopy.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryVisitorCopy.java 2012-02-01 09:33:28 UTC (rev 1407) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryVisitorCopy.java 2012-02-01 18:49:01 UTC (rev 1408) @@ -226,7 +226,11 @@ @Override public boolean visitEnter(Equals o) { - getQueryMaker().eq(o.getElement()); + if (o.isIgnoreCaseAndAccent()) { + getQueryMaker().eqIgnoreCaseAndAccent(o.getElement()); + } else { + getQueryMaker().eq(o.getElement()); + } return true; } @@ -237,7 +241,12 @@ @Override public boolean visitEnter(NotEquals o) { - getQueryMaker().ne(o.getElement()); + if (o.isIgnoreCaseAndAccent()) { + getQueryMaker().neIgnoreCaseAndAccent(o.getElement()); + } else { + getQueryMaker().ne(o.getElement()); + } + return true; } Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryVisitorToString.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryVisitorToString.java 2012-02-01 09:33:28 UTC (rev 1407) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/WikittyQueryVisitorToString.java 2012-02-01 18:49:01 UTC (rev 1408) @@ -202,8 +202,12 @@ @Override public boolean visitEnter(Equals o) { - text += o.getElement().getValue() - + WikittyQueryParser.EQUALS; + text += o.getElement().getValue(); + if (o.isIgnoreCaseAndAccent()) { + text += WikittyQueryParser.EQUALS_IGNORE_CASE_AND_ACCENT; + } else { + text += WikittyQueryParser.EQUALS; + } return true; } @@ -302,8 +306,12 @@ @Override public boolean visitEnter(NotEquals o) { - text += o.getElement().getValue() - + WikittyQueryParser.NOT_EQUALS; + text += o.getElement().getValue(); + if (o.isIgnoreCaseAndAccent()) { + text += WikittyQueryParser.NOT_EQUALS_IGNORE_CASE_AND_ACCENT; + } else { + text += WikittyQueryParser.NOT_EQUALS; + } return true; } Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/conditions/Equals.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/conditions/Equals.java 2012-02-01 09:33:28 UTC (rev 1407) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/conditions/Equals.java 2012-02-01 18:49:01 UTC (rev 1408) @@ -2,8 +2,8 @@ * #%L * Wikitty :: api * - * $Id: Equals.java 1136 2011-08-12 14:24:03Z tchemit $ - * $HeadURL: http://svn.nuiton.org/svn/wikitty/trunk/wikitty-api/src/main/java/org.nuito… $ + * $Id$ + * $HeadURL$ * %% * Copyright (C) 2009 - 2010 CodeLutin, Benjamin Poussin * %% @@ -38,6 +38,10 @@ * * <li> {@link WikittyQueryMaker}.eq("myext.myfield", "*jour") match field "bonjour" but not "BONJOUR" * + * <p> + * You can specify ignoreCaseAndAccent to check equality. This mode + * ignore case and accent. example: "çéçù" == "CecU" in this mode. + * * @author poussin * @version $Revision$ * @since 3.3 @@ -50,6 +54,8 @@ // serialVersionUID is used for serialization. private static final long serialVersionUID = 1L; + protected boolean ignoreCaseAndAccent = false; + public Equals(Element element) { super(element); } @@ -61,4 +67,24 @@ public Equals(Element element, ConditionValue value) { super(element, value); } + + public Equals(Element element, boolean ignoreCaseAndAccent) { + super(element); + this.ignoreCaseAndAccent = ignoreCaseAndAccent; + } + + public Equals(Element element, String value, boolean ignoreCaseAndAccent) { + super(element, value); + this.ignoreCaseAndAccent = ignoreCaseAndAccent; + } + + public Equals(Element element, ConditionValue value, boolean ignoreCaseAndAccent) { + super(element, value); + this.ignoreCaseAndAccent = ignoreCaseAndAccent; + } + + public boolean isIgnoreCaseAndAccent() { + return ignoreCaseAndAccent; + } + } Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/conditions/NotEquals.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/conditions/NotEquals.java 2012-02-01 09:33:28 UTC (rev 1407) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/query/conditions/NotEquals.java 2012-02-01 18:49:01 UTC (rev 1408) @@ -39,6 +39,12 @@ * * <li> {@link WikittyQueryMaker}.ne("myext.myfield", "*jour") not match field "bonjour" but match "BONJOUR" * + * <p> + * You can specify ignoreCaseAndAccent to check equality. This mode + * ignore case and accent. + * + * <li> {@link WikittyQueryMaker}.neIgnoreCaseAndAccent("myext.myfield", "*jour") not match field "bonjour" and not match "BONJOUR" + * * @author poussin * @version $Revision$ * @since 3.3 @@ -51,6 +57,8 @@ // serialVersionUID is used for serialization. private static final long serialVersionUID = 1L; + protected boolean ignoreCaseAndAccent = false; + public NotEquals(Element element) { super(element); } @@ -63,4 +71,23 @@ super(element, value); } + public NotEquals(Element element, boolean ignoreCaseAndAccent) { + super(element); + this.ignoreCaseAndAccent = ignoreCaseAndAccent; + } + + public NotEquals(Element element, String value, boolean ignoreCaseAndAccent) { + super(element, value); + this.ignoreCaseAndAccent = ignoreCaseAndAccent; + } + + public NotEquals(Element element, ConditionValue value, boolean ignoreCaseAndAccent) { + super(element, value); + this.ignoreCaseAndAccent = ignoreCaseAndAccent; + } + + public boolean isIgnoreCaseAndAccent() { + return ignoreCaseAndAccent; + } + } \ No newline at end of file Modified: trunk/wikitty-api/src/main/java/org/nuiton/wikitty/storage/WikittySearchEngineInMemory.java =================================================================== --- trunk/wikitty-api/src/main/java/org/nuiton/wikitty/storage/WikittySearchEngineInMemory.java 2012-02-01 09:33:28 UTC (rev 1407) +++ trunk/wikitty-api/src/main/java/org/nuiton/wikitty/storage/WikittySearchEngineInMemory.java 2012-02-01 18:49:01 UTC (rev 1408) @@ -44,7 +44,7 @@ import org.apache.commons.collections.bag.HashBag; import org.apache.commons.collections.map.LazyMap; import org.apache.commons.lang.ClassUtils; -import org.apache.commons.lang.StringUtils; +import org.apache.commons.lang3.StringUtils; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.nuiton.wikitty.WikittyException; @@ -409,6 +409,64 @@ } }; + static private Predicate EqualsIgnoreCaseAndAccentPredicate = new Predicate() { + public boolean check(FieldType type, Collection values, Collection expected) { + boolean result = false; + if (values != null && expected.size() > 0) { + // gestion des type STRING differement car il peut y avoir des '*' + if (type != null && type.getType() == WikittyTypes.STRING) { + Iterator i = expected.iterator(); + String exp = String.valueOf(i.next()); + for (Object fieldValue : values) { + String val = String.valueOf(fieldValue); + result = matchString(val, exp, true); + if (result) { + // si une des valeurs correspond, on retourne true + break; + } + } + } else { + expected = CollectionUtils.subtract(expected, values); + // si lorsqu'on retire tous les elements en commun avec la valeur + // du champs il ne reste plus rien, c'est que le equals est + // vrai + result = expected.isEmpty(); + } + } + return result; + + } + }; + + static private Predicate NotEqualsIgnoreCaseAndAccentPredicate = new Predicate() { + public boolean check(FieldType type, Collection values, Collection expected) { + boolean result = true; + if (values != null && expected.size() > 0) { + // gestion des type STRING differement car il peut y avoir des '*' + if (type != null && type.getType() == WikittyTypes.STRING) { + Iterator i = expected.iterator(); + String exp = String.valueOf(i.next()); + for (Object fieldValue : values) { + String val = String.valueOf(fieldValue); + result = !matchString(val, exp, true); + if (!result) { + // si une des valeurs correspond, on retourne true + break; + } + } + } else { + expected = CollectionUtils.subtract(expected, values); + // si lorsqu'on retire tous les elements en commun avec la valeur + // du champs il ne reste plus rien, c'est que le equals est + // vrai et donc le not equals est faux + result = !expected.isEmpty(); + } + } + return result; + + } + }; + static private Predicate GreaterPredicate = new Predicate() { public boolean check(FieldType type, Collection values, Collection expected) { boolean result = false; @@ -515,10 +573,9 @@ String exp = String.valueOf(i.next()); for (Object fieldValue : values) { String val = String.valueOf(fieldValue); - // TODO poussin 20111228 verifie que le containsIgnoreCase - // supprime les accents, si ce n'est pas le cas, il faut - // trouver comment faire - result = StringUtils.containsIgnoreCase(val, exp); + result = StringUtils.containsIgnoreCase( + StringUtils.stripAccents(val), + StringUtils.stripAccents(exp)); if (result) { // si une des valeurs correspond, on retourne true break; @@ -593,39 +650,43 @@ * @param s string * @param sub string to match with 's' parameter. This string can start * or and with star '*' - * @param ignoreCase if true match is done in ingore case mode + * @param ignoreCaseAndAccent if true match is done in ingore case mode * @return true if sub match s */ - static private boolean matchString(String s, String sub, boolean ignoreCase) { - // TODO poussin 20111228 verifie que le containsIgnoreCase - // supprime les accents, si ce n'est pas le cas, il faut - // trouver comment faire - + static private boolean matchString(String s, String sub, boolean ignoreCaseAndAccent) { boolean result = false; if (sub.startsWith("*") && sub.endsWith("*")) { sub = StringUtils.substring(sub, 1, -1); - if (ignoreCase) { - result = StringUtils.containsIgnoreCase(s, sub); + if (ignoreCaseAndAccent) { + result = StringUtils.containsIgnoreCase( + StringUtils.stripAccents(s), + StringUtils.stripAccents(sub)); } else { result = StringUtils.contains(s, sub); } } else if (sub.startsWith("*")) { sub = StringUtils.substring(sub, 1); - if (ignoreCase) { - result = StringUtils.endsWithIgnoreCase(s, sub); + if (ignoreCaseAndAccent) { + result = StringUtils.endsWithIgnoreCase( + StringUtils.stripAccents(s), + StringUtils.stripAccents(sub)); } else { result = StringUtils.endsWith(s, sub); } } else if (sub.endsWith("*")) { sub = StringUtils.substring(sub, 0, -1); - if (ignoreCase) { - result = StringUtils.startsWithIgnoreCase(s, sub); + if (ignoreCaseAndAccent) { + result = StringUtils.startsWithIgnoreCase( + StringUtils.stripAccents(s), + StringUtils.stripAccents(sub)); } else { result = StringUtils.startsWith(s, sub); } } else { - if (ignoreCase) { - result = StringUtils.equalsIgnoreCase(s, sub); + if (ignoreCaseAndAccent) { + result = StringUtils.equalsIgnoreCase( + StringUtils.stripAccents(s), + StringUtils.stripAccents(sub)); } else { result = StringUtils.equals(s, sub); } @@ -1011,8 +1072,13 @@ public boolean visitEnter(Equals o) { boolean result = false; - result = check(EqualsPredicate, o.getElement(), - evalConditionValue(o.getValue())); + if (o.isIgnoreCaseAndAccent()) { + result = check(EqualsIgnoreCaseAndAccentPredicate, o.getElement(), + evalConditionValue(o.getValue())); + } else { + result = check(EqualsPredicate, o.getElement(), + evalConditionValue(o.getValue())); + } evalStack.push(result); return false; @@ -1027,8 +1093,13 @@ public boolean visitEnter(NotEquals o) { boolean result = false; - result = check(NotEqualsPredicate, o.getElement(), - evalConditionValue(o.getValue())); + if (o.isIgnoreCaseAndAccent()) { + result = check(NotEqualsIgnoreCaseAndAccentPredicate, o.getElement(), + evalConditionValue(o.getValue())); + } else { + result = check(NotEqualsPredicate, o.getElement(), + evalConditionValue(o.getValue())); + } evalStack.push(result); return false; Modified: trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java =================================================================== --- trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java 2012-02-01 09:33:28 UTC (rev 1407) +++ trunk/wikitty-api/src/test/java/org/nuiton/wikitty/WikittyClientTest.java 2012-02-01 18:49:01 UTC (rev 1408) @@ -250,7 +250,7 @@ w = wikittyClient.store(w); Assert.fail("not null contraint don't work on String"); } catch (WikittyException eee) { - eee.printStackTrace(); +// eee.printStackTrace(); // ok id must not be null } @@ -259,7 +259,7 @@ w = wikittyClient.store(w); Assert.fail("not null contraint don't work in Collection"); } catch (WikittyException eee) { - eee.printStackTrace(); +// eee.printStackTrace(); // ok id must not be null } @@ -1458,7 +1458,7 @@ WikittyQueryResult<Product> results3 = wikittyClient.findAllByQuery(Product.class, query3); Assert.assertEquals(1, results3.getTotalResult()); - // ??? + // ???13 WikittyQuery query4 = new WikittyQueryMaker().eq("*.name." + WikittyTypes.STRING, "Lanfeust").end(); WikittyQueryResult<Product> results4 = wikittyClient.findAllByQuery(Product.class, query4); Assert.assertEquals(1, results4.getTotalResult()); @@ -1639,7 +1639,8 @@ { // avec accent specifiquement sur le champs name mais sans accent pour la recherche // specifique à solr - WikittyQuery query = new WikittyQueryMaker().eq(WikittyGroup.FQ_FIELD_WIKITTYGROUP_NAME + "_c_t", "ceca").end(); + WikittyQuery query = new WikittyQueryMaker().eqIgnoreCaseAndAccent( + WikittyGroup.FQ_FIELD_WIKITTYGROUP_NAME, "ceca").end(); WikittyQueryResult<String> result = wikittyClient.findAllByQuery(query); Assert.assertEquals(1, result.size()); } @@ -1647,7 +1648,8 @@ // avec accent specifiquement sur le champs name // mais sans accent pour la recherche // et et majuscule - WikittyQuery query = new WikittyQueryMaker().eq(WikittyGroup.FQ_FIELD_WIKITTYGROUP_NAME + "_c_t", "CECA").end(); + WikittyQuery query = new WikittyQueryMaker().eqIgnoreCaseAndAccent( + WikittyGroup.FQ_FIELD_WIKITTYGROUP_NAME, "CECA").end(); WikittyQueryResult<String> result = wikittyClient.findAllByQuery(query); Assert.assertEquals(1, result.size()); } @@ -1655,7 +1657,8 @@ // avec accent specifiquement sur le champs name // mais sans accent pour la recherche // et et majuscule - WikittyQuery query = new WikittyQueryMaker().like(WikittyGroup.FQ_FIELD_WIKITTYGROUP_NAME, "ceca").end(); + WikittyQuery query = new WikittyQueryMaker().like( + WikittyGroup.FQ_FIELD_WIKITTYGROUP_NAME, "ceca").end(); WikittyQueryResult<String> result = wikittyClient.findAllByQuery(query); Assert.assertEquals(1, result.size()); } Modified: trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java =================================================================== --- trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java 2012-02-01 09:33:28 UTC (rev 1407) +++ trunk/wikitty-solr/src/main/java/org/nuiton/wikitty/storage/solr/WikittyQueryVisitorToSolr.java 2012-02-01 18:49:01 UTC (rev 1408) @@ -318,8 +318,16 @@ @Override public boolean visitEnter(Equals o) { - solrQuery += element2solr(o.getElement()) + String element2solr = element2solr(o.getElement()); + if (element2solr.endsWith(WikittySolrConstant.SUFFIX_STRING)) { // is string + if (o.isIgnoreCaseAndAccent()) { + element2solr += WikittySolrConstant.SUFFIX_STRING_LOWERCASE; + } + } + + solrQuery += element2solr + ":" + evalConditionValue(o.getValue()); + return false; } @@ -330,7 +338,14 @@ @Override public boolean visitEnter(NotEquals o) { - solrQuery += "-" + element2solr(o.getElement()) + String element2solr = element2solr(o.getElement()); + if (element2solr.endsWith(WikittySolrConstant.SUFFIX_STRING)) { // is string + if (o.isIgnoreCaseAndAccent()) { + element2solr += WikittySolrConstant.SUFFIX_STRING_LOWERCASE; + } + } + + solrQuery += "-" + element2solr + ":" + evalConditionValue(o.getValue()); return false; }

1 0

r1407 - in trunk: . wikitty-api wikitty-dto wikitty-generators wikitty-hessian-client wikitty-hessian-server wikitty-jdbc wikitty-perf-test wikitty-solr wikitty-struts
by maven-release＠users.nuiton.org 01 Feb '12

01 Feb '12

Author: maven-release Date: 2012-02-01 10:33:28 +0100 (Wed, 01 Feb 2012) New Revision: 1407 Url: http://nuiton.org/repositories/revision/wikitty/1407 Log: [maven-release-plugin] prepare for next development iteration Modified: trunk/pom.xml trunk/wikitty-api/pom.xml trunk/wikitty-dto/pom.xml trunk/wikitty-generators/pom.xml trunk/wikitty-hessian-client/pom.xml trunk/wikitty-hessian-server/pom.xml trunk/wikitty-jdbc/pom.xml trunk/wikitty-perf-test/pom.xml trunk/wikitty-solr/pom.xml trunk/wikitty-struts/pom.xml Modified: trunk/pom.xml =================================================================== --- trunk/pom.xml 2012-02-01 09:33:26 UTC (rev 1406) +++ trunk/pom.xml 2012-02-01 09:33:28 UTC (rev 1407) @@ -15,7 +15,7 @@ <groupId>org.nuiton</groupId> <artifactId>wikitty</artifactId> - <version>3.4</version> + <version>3.5-SNAPSHOT</version> <modules> @@ -795,10 +795,10 @@  <scm> - <connection>scm:svn:http://svn.nuiton.org/svn/wikitty/tags/wikitty-3.4</connection> - <developerConnection>scm:svn:http://svn.nuiton.org/svn/wikitty/tags/wikitty-3.4 + <connection>scm:svn:http://svn.nuiton.org/svn/wikitty/trunk</connection> + <developerConnection>scm:svn:http://svn.nuiton.org/svn/wikitty/trunk </developerConnection> - <url>http://www.nuiton.org/repositories/browse/wikitty/tags/wikitty-3.4</url> + <url>http://www.nuiton.org/repositories/browse/wikitty/trunk</url> </scm> <distributionManagement> Modified: trunk/wikitty-api/pom.xml =================================================================== --- trunk/wikitty-api/pom.xml 2012-02-01 09:33:26 UTC (rev 1406) +++ trunk/wikitty-api/pom.xml 2012-02-01 09:33:28 UTC (rev 1407) @@ -6,7 +6,7 @@ <parent> <groupId>org.nuiton</groupId> <artifactId>wikitty</artifactId> - <version>3.4</version> + <version>3.5-SNAPSHOT</version> </parent>  Modified: trunk/wikitty-dto/pom.xml =================================================================== --- trunk/wikitty-dto/pom.xml 2012-02-01 09:33:26 UTC (rev 1406) +++ trunk/wikitty-dto/pom.xml 2012-02-01 09:33:28 UTC (rev 1407) @@ -6,7 +6,7 @@ <parent> <groupId>org.nuiton</groupId> <artifactId>wikitty</artifactId> - <version>3.4</version> + <version>3.5-SNAPSHOT</version> </parent>  Modified: trunk/wikitty-generators/pom.xml =================================================================== --- trunk/wikitty-generators/pom.xml 2012-02-01 09:33:26 UTC (rev 1406) +++ trunk/wikitty-generators/pom.xml 2012-02-01 09:33:28 UTC (rev 1407) @@ -6,7 +6,7 @@ <parent> <groupId>org.nuiton</groupId> <artifactId>wikitty</artifactId> - <version>3.4</version> + <version>3.5-SNAPSHOT</version> </parent>  Modified: trunk/wikitty-hessian-client/pom.xml =================================================================== --- trunk/wikitty-hessian-client/pom.xml 2012-02-01 09:33:26 UTC (rev 1406) +++ trunk/wikitty-hessian-client/pom.xml 2012-02-01 09:33:28 UTC (rev 1407) @@ -6,7 +6,7 @@ <parent> <groupId>org.nuiton</groupId> <artifactId>wikitty</artifactId> - <version>3.4</version> + <version>3.5-SNAPSHOT</version> </parent>  Modified: trunk/wikitty-hessian-server/pom.xml =================================================================== --- trunk/wikitty-hessian-server/pom.xml 2012-02-01 09:33:26 UTC (rev 1406) +++ trunk/wikitty-hessian-server/pom.xml 2012-02-01 09:33:28 UTC (rev 1407) @@ -6,7 +6,7 @@ <parent> <groupId>org.nuiton</groupId> <artifactId>wikitty</artifactId> - <version>3.4</version> + <version>3.5-SNAPSHOT</version> </parent>  Modified: trunk/wikitty-jdbc/pom.xml =================================================================== --- trunk/wikitty-jdbc/pom.xml 2012-02-01 09:33:26 UTC (rev 1406) +++ trunk/wikitty-jdbc/pom.xml 2012-02-01 09:33:28 UTC (rev 1407) @@ -6,7 +6,7 @@ <parent> <groupId>org.nuiton</groupId> <artifactId>wikitty</artifactId> - <version>3.4</version> + <version>3.5-SNAPSHOT</version> </parent> <groupId>org.nuiton.wikitty</groupId> Modified: trunk/wikitty-perf-test/pom.xml =================================================================== --- trunk/wikitty-perf-test/pom.xml 2012-02-01 09:33:26 UTC (rev 1406) +++ trunk/wikitty-perf-test/pom.xml 2012-02-01 09:33:28 UTC (rev 1407) @@ -6,7 +6,7 @@ <parent> <groupId>org.nuiton</groupId> <artifactId>wikitty</artifactId> - <version>3.4</version> + <version>3.5-SNAPSHOT</version> </parent> <groupId>org.nuiton.wikitty</groupId> Modified: trunk/wikitty-solr/pom.xml =================================================================== --- trunk/wikitty-solr/pom.xml 2012-02-01 09:33:26 UTC (rev 1406) +++ trunk/wikitty-solr/pom.xml 2012-02-01 09:33:28 UTC (rev 1407) @@ -6,7 +6,7 @@ <parent> <groupId>org.nuiton</groupId> <artifactId>wikitty</artifactId> - <version>3.4</version> + <version>3.5-SNAPSHOT</version> </parent>  Modified: trunk/wikitty-struts/pom.xml =================================================================== --- trunk/wikitty-struts/pom.xml 2012-02-01 09:33:26 UTC (rev 1406) +++ trunk/wikitty-struts/pom.xml 2012-02-01 09:33:28 UTC (rev 1407) @@ -6,7 +6,7 @@ <parent> <groupId>org.nuiton</groupId> <artifactId>wikitty</artifactId> - <version>3.4</version> + <version>3.5-SNAPSHOT</version> </parent>

1 0