Word boundary is not

I ran into an even worse problem when searching text for words like .NET, C++, C#, and C. You would think that computer programmers would know better than to name a language something that is hard to write regular expressions for.

Anyway, this is what I found out (summarized mostly from http://www.regular-expressions.info, which is a great site): In most flavors of regex, characters that are matched by the short-hand character class w are the characters that are treated as word characters by word boundaries. Java is an exception. Java supports Unicode for b but not for w. (I’m sure there was a good reason for it at the time).

The w stands for «word character». It always matches the ASCII characters [A-Za-z0-9_]. Notice the inclusion of the underscore and digits (but not dash!). In most flavors that support Unicode, w includes many characters from other scripts. There is a lot of inconsistency about which characters are actually included. Letters and digits from alphabetic scripts and ideographs are generally included. Connector punctuation other than the underscore and numeric symbols that aren’t digits may or may not be included. XML Schema and XPath even include all symbols in w. But Java, JavaScript, and PCRE match only ASCII characters with w.

Which is why Java-based regex searches for C++, C# or .NET (even when you remember to escape the period and pluses) are screwed by the b.

Note: I’m not sure what to do about mistakes in text, like when someone doesn’t put a space after a period at the end of a sentence. I allowed for it, but I’m not sure that it’s necessarily the right thing to do.

Anyway, in Java, if you’re searching text for the those weird-named languages, you need to replace the b with before and after whitespace and punctuation designators. For example:

public static String grep(String regexp, String multiLineStringToSearch) {
    String result = "";
    String[] lines = multiLineStringToSearch.split("\n");
    Pattern pattern = Pattern.compile(regexp);
    for (String line : lines) {
        Matcher matcher = pattern.matcher(line);
        if (matcher.find()) {
            result = result + "n" + line;
        }
    }
    return result.trim();
}

Then in your test or main function:

    String beforeWord = "(\s|\.|\,|\!|\?|\(|\)|\'|\"|^)";   
    String afterWord =  "(\s|\.|\,|\!|\?|\(|\)|\'|\"|$)";
    text = "Programming in C, (C++) C#, Java, and .NET.";
    System.out.println("text="+text);
    // Here is where Java word boundaries do not work correctly on "cutesy" computer language names.  
    System.out.println("Bad word boundary can't find because of Java: grep with word boundary for .NET="+ grep("\b\.NET\b", text));
    System.out.println("Should find: grep exactly for .NET="+ grep(beforeWord+"\.NET"+afterWord, text));
    System.out.println("Bad word boundary can't find because of Java: grep with word boundary for C#="+ grep("\bC#\b", text));
    System.out.println("Should find: grep exactly for C#="+ grep("C#"+afterWord, text));
    System.out.println("Bad word boundary can't find because of Java:grep with word boundary for C++="+ grep("\bC\+\+\b", text));
    System.out.println("Should find: grep exactly for C++="+ grep(beforeWord+"C\+\+"+afterWord, text));

    System.out.println("Should find: grep with word boundary for Java="+ grep("\bJava\b", text));
    System.out.println("Should find: grep for case-insensitive java="+ grep("?i)\bjava\b", text));
    System.out.println("Should find: grep with word boundary for C="+ grep("\bC\b", text));  // Works Ok for this example, but see below
    // Because of the stupid too-short cutsey name, searches find stuff it shouldn't.
    text = "Worked on C&O (Chesapeake and Ohio) Canal when I was younger; more recently developed in Lisp.";
    System.out.println("text="+text);
    System.out.println("Bad word boundary because of C name: grep with word boundary for C="+ grep("\bC\b", text));
    System.out.println("Should be blank: grep exactly for C="+ grep(beforeWord+"C"+afterWord, text));
    // Make sure the first and last cases work OK.

    text = "C is a language that should have been named differently.";
    System.out.println("text="+text);
    System.out.println("grep exactly for C="+ grep(beforeWord+"C"+afterWord, text));

    text = "One language that should have been named differently is C";
    System.out.println("text="+text);
    System.out.println("grep exactly for C="+ grep(beforeWord+"C"+afterWord, text));

    //Make sure we don't get false positives
    text = "The letter 'c' can be hard as in Cat, or soft as in Cindy. Computer languages should not require disambiguation (e.g. Ruby, Python vs. Fortran, Hadoop)";
    System.out.println("text="+text);
    System.out.println("Should be blank: grep exactly for C="+ grep(beforeWord+"C"+afterWord, text));

P.S. My thanks to http://regexpal.com/ without whom the regex world would be very miserable!

The metacharacter b is an anchor like the caret and the dollar sign. It matches at a position that is called a “word boundary”. This match is zero-length.

There are three different positions that qualify as word boundaries:

• Before the first character in the string, if the first character is a word character.
• After the last character in the string, if the last character is a word character.
• Between two characters in the string, where one is a word character and the other is not a word character.

Simply put: b allows you to perform a “whole words only” search using a regular expression in the form of bwordb. A “word character” is a character that can be used to form words. All characters that are not “word characters” are “non-word characters”.

Exactly which characters are word characters depends on the regex flavor you’re working with. In most flavors, characters that are matched by the short-hand character class w are the characters that are treated as word characters by word boundaries. Java is an exception. Java supports Unicode for b but not for w.

Most flavors, except the ones discussed below, have only one metacharacter that matches both before a word and after a word. This is because any position between characters can never be both at the start and at the end of a word. Using only one operator makes things easier for you.

Since digits are considered to be word characters, b4b can be used to match a 4 that is not part of a larger number. This regex does not match 44 sheets of a4. So saying “b matches before and after an alphanumeric sequence” is more exact than saying “before and after a word”.

B is the negated version of b. B matches at every position where b does not. Effectively, B matches at any position between two word characters as well as at any position between two non-word characters.

Looking Inside The Regex Engine

Let’s see what happens when we apply the regex bisb to the string This island is beautiful. The engine starts with the first token b at the first character T. Since this token is zero-length, the position before the character is inspected. b matches here, because the T is a word character and the character before it is the void before the start of the string. The engine continues with the next token: the literal i. The engine does not advance to the next character in the string, because the previous regex token was zero-length. i does not match T, so the engine retries the first token at the next character position.

b cannot match at the position between the T and the h. It cannot match between the h and the i either, and neither between the i and the s.

The next character in the string is a space. b matches here because the space is not a word character, and the preceding character is. Again, the engine continues with the i which does not match with the space.

Advancing a character and restarting with the first regex token, b matches between the space and the second i in the string. Continuing, the regex engine finds that i matches i and s matches s. Now, the engine tries to match the second b at the position before the l. This fails because this position is between two word characters. The engine reverts to the start of the regex and advances one character to the s in island. Again, the b fails to match and continues to do so until the second space is reached. It matches there, but matching the i fails.

But b matches at the position before the third i in the string. The engine continues, and finds that i matches i and s matches s. The last token in the regex, b, also matches at the position before the third space in the string because the space is not a word character, and the character before it is.

The engine has successfully matched the word is in our string, skipping the two earlier occurrences of the characters i and s. If we had used the regular expression is, it would have matched the is in This.

Tcl Word Boundaries

Word boundaries, as described above, are supported by most regular expression flavors. Notable exceptions are the POSIX and XML Schema flavors, which don’t support word boundaries at all. Tcl uses a different syntax.

In Tcl, b matches a backspace character, just like x08 in most regex flavors (including Tcl’s). B matches a single backslash character in Tcl, just like \ in all other regex flavors (and Tcl too).

Tcl uses the letter “y” instead of the letter “b” to match word boundaries. y matches at any word boundary position, while Y matches at any position that is not a word boundary. These Tcl regex tokens match exactly the same as b and B in Perl-style regex flavors. They don’t discriminate between the start and the end of a word.

Tcl has two more word boundary tokens that do discriminate between the start and end of a word. m matches only at the start of a word. That is, it matches at any position that has a non-word character to the left of it, and a word character to the right of it. It also matches at the start of the string if the first character in the string is a word character. M matches only at the end of a word. It matches at any position that has a word character to the left of it, and a non-word character to the right of it. It also matches at the end of the string if the last character in the string is a word character.

The only regex engine that supports Tcl-style word boundaries (besides Tcl itself) is the JGsoft engine. In PowerGREP and EditPad Pro, b and B are Perl-style word boundaries, while y, Y, m and M are Tcl-style word boundaries.

In most situations, the lack of m and M tokens is not a problem. ywordy finds “whole words only” occurrences of “word” just like mwordM would. Mwordm could never match anywhere, since M never matches at a position followed by a word character, and m never at a position preceded by one. If your regular expression needs to match characters before or after y, you can easily specify in the regex whether these characters should be word characters or non-word characters. If you want to match any word, yw+y gives the same result as m.+M. Using w instead of the dot automatically restricts the first y to the start of a word, and the second y to the end of a word. Note that y.+y would not work. This regex matches each word, and also each sequence of non-word characters between the words in your subject string. That said, if your flavor supports m and M, the regex engine could apply mw+M slightly faster than yw+y, depending on its internal optimizations.

If your regex flavor supports lookahead and lookbehind, you can use (?<!w)(?=w) to emulate Tcl’s m and (?<=w)(?!w) to emulate M. Though quite a bit more verbose, these lookaround constructs match exactly the same as Tcl’s word boundaries.

If your flavor has lookahead but not lookbehind, and also has Perl-style word boundaries, you can use b(?=w) to emulate Tcl’s m and b(?!w) to emulate M. b matches at the start or end of a word, and the lookahead checks if the next character is part of a word or not. If it is we’re at the start of a word. Otherwise, we’re at the end of a word.

GNU Word Boundaries

The GNU extensions to POSIX regular expressions add support for the b and B word boundaries, as described above. GNU also uses its own syntax for start-of-word and end-of-word boundaries. < matches at the start of a word, like Tcl’s m. > matches at the end of a word, like Tcl’s M.

Boost also treats < and > as word boundaries when using the ECMAScript, extended, egrep, or awk grammar.

POSIX Word Boundaries

The POSIX standard defines [[:<:]] as a start-of-word boundary, and [[:>:]] as an end-of-word boundary. Though the syntax is borrowed from POSIX bracket expressions, these tokens are word boundaries that have nothing to do with and cannot be used inside character classes. Tcl and GNU also support POSIX word boundaries. PCRE supports POSIX word boundaries starting with version 8.34. Boost supports them in all its grammars.

The question relies on a number of unidentified assumptions about word boundaries, which are not totally alien but also are not obvious or obviously right. The main problem I see is the premise that there is this one thing, word boundary, that solves myriad problems.

The notion of there being a single «phonological tree» seems to be historically based on importing notions of structure from syntax (we wanted phonology to be more like syntax), but the properties of tree-like representations as used in syllable and foot structure are not the same as those employed in syntactic representations (prosodic structure is not seriously recursive in the way that syntactic trees are; phonological «trees» flout the single-mother convention). Attempting to align phonological grouping with morphosyntactic grouping just leads to tears, though that is not obvious if you consider just English. The problem is that combining a VC root with a VC prefix and a VC suffix typically leads to phonological V.C+V.C+VC, i.e. syllable boundaries seriously misaligned with morpheme boundaries.

In English and in contrast to other languages such as Arabic, there is not much evidence for resyllabification between words, so prosodic and syntactic constituency are not generally at odds. At the level of affixation, we do have mismatches involving V-initial suffixes (invite [ɪn.ˈvajʔ], invitee [ɪn.vaj.ˈtʰi]), but not at the phrasal level in e.g. «invite Igor». In asking about word boundaries in «the big house», «motorcycle» or «What are you going to do?», you have to have a theory of entities (are there both word and syllable boundaries? Are there also morpheme boundaries?), and what those entities do for you. Are there necessary or sufficient criteria for diagnosing «.», «+» or «#»?

The reason for positing word boundaries is usually syntactic: «the» is a word, it occupies a certain syntactic position, same with «big». We might claim that «motorcycle» has an internal word boundary because «motor» and «cycle» are words, and neither can reasonably be called a prefix or suffix. Phonologically speaking, there is nothing about «motorcycle» that demands a word boundary.

Certain concatenations that can be lumped together under the rubric «contraction», for example «going to» → «gonna», «will not» → «won’t», «got you» → «gotcha», also «Harry’s», behave phonologically more like affixational structures, even though they are syntactically more like word combinations. Just positing a readjustment of boundaries (removing the «#») does not solve all of the problems, especially in negative inflections (my analytic prejudice is now revealed).

The final complication in analyzing the aforementioned concatenations is that boundaries are also invoked to account for some facts of speech speech rhythm. The two syllables of «lighthouse» have a fixed rhythmic organization (prominence on the first syllable), but the phrase «light house» has variable rhythm (depends on whether you’re shopping for a light house vs a heavy house; or is the discussion about a house that is light vs. a hose that is light). Again, attempting to reduce these speech rhythm properties to nothing more than differences in word boundaries has proven to be futile. Once you introduce some other mechanism for encoding rhythmic distinctions, manipulations of word boundaries becomes unnecessary – we can just posit that word boundaries are there if and only if we syntactically concatenate two words. You still have to have an account of whether «won’t» is two syntactic words (as opposed to two syntactically-mandated functions manifested within a single word).

In other words, manipulating word boundaries has not proven to be a useful method of analysis.