I think Perl 6's Rules are a natural and important extension to the language. Perl has always been a tool of choice for text processing; but people tend to [ab]use regular expressions even where a real parser would be much more appropriate because their language -- Perl, Python, Ruby, C#, pretty much any commercially popular language these days -- gives you regular expressions on a platter, and their presence in the core language or standard library sort of enshrines regexes as a tool of choice.
Look at it this way: pretty much any "Programming ______" book will at least touch on that language's regexp implementation. How many such books currently cover implementing basic parsers for non-regular grammars?
So yeah, if Perl 6 can bring parsers to the mainstream by providing easy, visible tools to implement them, and even simply by making parsers something you learn about in the course of learning Perl, that'll be a huge win for us all.
"Programming ______" book will at least touch on that language's regexp implementation. How many such books currently cover implementing basic parsers for non-regular grammars?
A good point, but I think the books and languages didn't have much choice until recently. There really is a big jump in complexity from using a few regexes in a program, and going full-on into a parser using lex/yacc (or your language's port, like javacc).
I think it's only been very recently that there's been a growth in parsing paradigms that are more amenable to inclusion as libraries with ease-of-use at least vaguely comparable to regexes. Things like combinator-style parser libraries (Parsec and similar libraries in other languages), and OMeta (http://tinlizzie.org/ometa/).
When you actually get down to writing it, a top-down parser is a fairly simple beast. A programmer doesn't really need many extra tools to do it. You just create a bunch of functions which peak-at symbols, consume symbols and call each other.
The art is in having an exactly nailed-down specification concerning how these function are going to work together before you even begin. And that art comes from first analyzing and manipulate your language's grammar. That's a process that's distinct from the final code that you write and it requires knowing a little bit about abstract languages. (Your functions are pieces of the language but not the language pieces one would intuitively imagine. The parts of a language are great examples of "programming objects" which are not "object oriented" objects).
What's hard about understanding an abstract language is that it isn't a command or feature that within a language but is ... a language in itself. Creating an abstract language involves moving to a higher logical level, just using a programming language involves moving to a higher logical level than using an application. It's not that it's impossibly hard but it requires a student to interrupt any rote learning they're doing and think. It's a great thing but it's also a road block to a significant percentage of students.
Given this situation, I'm not sure how any language feature can help. Sure, you could plunk a whole parser generator into the code but I'd say it's harder to understand the combination of language specification and generated-code that results from such a thing.
I'd like to know how the Perl 6 people expect to deal with this challenge.
I agree that it would be nice to move people from complex regular expressions to reasonably simple parsers.
In the real world you seldomly have an exactly nailed-down specification of anything. If you're lucky enough to do something as basic as parsing URIs or JSON, you have, but normally not.
Writing a recursive descending parser isn't hard, as you said, but it's again something that needs testing, you need put some effort into it to get good error messages from it - and why would you repeat such a task? Also you often need to write a lexer -- another burden of which Perl 6 grammars free you.
Regarding your thoughts about abstract languages: Perl 6 provides tools to make experimenting with them rather easy, which helps learning about them.
I've made the experience that a reasonably good programmer can write some 20 lines of Perl code, and it does what he wants. That doesn't imply that he can write a 20 lines grammar that does what he wants, without doing some debugging and corrections first.
Writing grammars is a skill that just has to be learned, line any other skill too. Providing tools that make it fun and rewarding to play with parsing stuff is the best we can do.
Look at it this way: pretty much any "Programming ______" book will at least touch on that language's regexp implementation. How many such books currently cover implementing basic parsers for non-regular grammars?
So yeah, if Perl 6 can bring parsers to the mainstream by providing easy, visible tools to implement them, and even simply by making parsers something you learn about in the course of learning Perl, that'll be a huge win for us all.