Slides from Southeast LinuxFest 2011 talk

Regular Expression Practice UPDATE: Here is a video of the talk and the link:

Lazy day links

The talk at SELF 2011 is done - Whew! I'll get around to posting the slides soon, but for now, I just wanna be lazy and post a link or two.

"Crucial Concepts Behind Advanced Regular Expressions" offers an interesting mix of concepts, ranging from everyday usage (greedy/non-greedy, and word boundaries) to the esoteric (atomic groups, callbacks, and recursion). I don't agree that the concepts are all crucial, but I thought it was a good read.

Humor: Parsing HTML with Regex - funny!!

Regex for Bash-style Variables, Concluded

The last regex handled double-quoted strings, but not the variable name and and equals sign that precede them. I could add to the regex to specifically match the name and equals sign, but I know my earlier regex which handled single quotes also matched the variable name and equals sign. Ultimately, I want one regex which matches all forms of Bash variable setting, so I'll combine the two.

I've been careful up till now to appropriately quote my regex for use at the command line. Now that both single and double quotes will be present, I'll switch to using a file for my script.

$ cat bashvars
#!/usr/bin/perl -n

print "$&\n" if m/("(\\"|[^"])*"|'[^']*'|[^#\n])*/
$ cat in
FOO=42 # answer to the question
BAR='easter bunny #2' # hippity hoppity
BAZ="\"DON'T PANIC\" in large, friendly letters"
$ ./bashvars in
FOO=42
BAR='easter bunny #2'
BAZ="\"DON'T PANIC\" in large, friendly letters"

If the input is limited to just lines that set variables, the above script works, but if the input is, say, a whole Bash script, it quickly becomes apparent that more than just variables are matched. I will (finally) add to the regex to insist that it match a variable name and equals sign. I'll also add a semicolon to the most generic character class to cover those times when a variable setting is followed by code on the same line.

$ cat bashvars2
#!/usr/bin/perl -n

$v = qr/("(\\"|[^"])*"|'[^']*'|[^#;\n])*/;
$kvp = qr/^\s*([_a-zA-Z]\w*=$v)/;
print "$1\n" if $_ =~ $kvp;

Handling Double Quotes in Regex for Bash-Style Variables

Expanding on Sunday's post, I want to add double quote handling to the regex for Bash-style variables.  This can be done similarly to the single quote handling, with a couple of wrinkles.  A double-quoted string may contain escaped quotes, which do not terminate the string, but instead cause literal double quote characters to be included in the string.  The escape character is the backslash. A double-quoted string may also contain single quotes, which are interpreted literally.

$ BAZ="\"DON'T PANIC\" in large, friendly letters"
echo $BAZ
"DON'T PANIC" in large, friendly letters

The double quote analog of the single quote regex looks like "[^"]*", but that won't handle the escaped quotes. A regex of \\" will, but only once. To combine these regex, I use the same approach as before, moving the * from the first regex to the combined regex, and weighting the regex to prefer \\". The combined regex is "(\\"|[^"])*".

$ cat in
BAZ="\"DON'T PANIC\" in large, friendly letters"
$ perl -ne 'print "$&\n" if m/"(\\"|[^"])*"/' in
"\"DON'T PANIC\" in large, friendly letters"

Notice, this doesn't match the key portion of the variable line. I'll address that in the next post.

A regular expression to check for prime numbers

In ruby:
http://www.noulakaz.net/weblog/2007/03/18/a-regular-expression-to-check-for-prime-numbers/

and the original, in perl:
http://montreal.pm.org/tech/neil_kandalgaonkar.shtml

I cringe a bit when I think about performance... but hey - it's cool!

Regex for Bash-style Variables

Returning to the regex from Friday's post, the first thing to understand is context.  What should it match and what should it not match?  Here, I want to match key-value pairs.  These kvp's are written using Bash (or rather Bourne shell) syntax, for example:

FOO=42
BAR='easter bunny'

Trivially, .* will match these lines, or any line for that matter, so what should our regex not match?  I'll start by excluding comments at the end of the line.

FOO=42 # answer to the question
BAR='easter bunny' # hippity hoppity

A regex of [^#]* would do the trick, except that a # character might appear within a string.

FOO=42 # answer to the question
BAR='easter bunny #2' # hippity hoppity

A regex of '[^']*' works for the quoted string, but not for the text leading up to the quote and not for the line without quotes.  These two regex can be combined, but the results aren't quite right.

$ cat in
FOO=42 # answer to the question
BAR='easter bunny #2' # hippity hoppity
$ perl -ne 'print "$&\n" if '"m/[^#]*|'[^']*'/" in
FOO=42
BAR='easter bunny
$ perl -ne 'print "$&\n" if '"m/'[^']*'|[^#]*/" in
FOO=42
BAR='easter bunny

The matches are actually the same as for the first regex alone, regardless of the ordering.  The [^#]* regex is partially consuming the quote, rather than the second regex consuming the whole quote.

My favorite solution to this problem is to weight the combined regex in favor of the quote-matching portion.  I do this by removing the * from [^#]*.  The combined regex then matches either a whole quoted string or a single non-# character.  I then put a * on the combined regex, so it repeatedly consumes quotes or single non-# characters.

$ perl -ne 'print "$&\n" if '"m/('[^']*'|[^#])*/" in
FOO=42
BAR='easter bunny #2'

Now, the order of the subexpressions is important.  Next time, I'll expand this to handle double-quoted strings.

Courtesy of xkcd